-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 22:12:52 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg106-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:13:08 (1763349188) === oleg106-client.virtnet: executing check_config_client /mnt/lustre oleg106-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg106-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa03498997800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa03498997800.idle_timeout=debug disable quota as required oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:13:41 (1763349221) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8193] == sanity-flr test 0a: lfs mirror create with -N option == 22:13:43 (1763349223) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:13:59 (1763349239) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:14:02 (1763349242) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:14:05 (1763349245) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:14:27 (1763349267) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:14:30 (1763349270) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 22:14:33 (1763349273) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa03498997800.stats=clear osc.lustre-OST0001-osc-ffffa03498997800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:14:40 (1763349280) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:14:48 (1763349288) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:14:54 (1763349294) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:14:57 (1763349297) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:15:05 (1763349305) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:15:11 (1763349311) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:15:18 (1763349318) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:15:24 (1763349324) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:15:32 (1763349332) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:15:38 (1763349338) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099458 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104477 s, 10.0 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:15:44 (1763349344) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.315235 s, 36.6 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:15:51 (1763349351) Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (56s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:16:47 (1763349407) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.30982 s, 20.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:17:25 (1763349445) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (68s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:18:34 (1763349514) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.94749 s, 17.8 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 22:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.922s user 0m0.003s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.962s user 0m0.002s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.972s user 0m0.002s sys 0m0.183s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa03498997800.ost_server_uuid in FULL state after 0 sec PASS 33b (76s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:19:49 (1763349589) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:19:52 (1763349592) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:19:55 (1763349595) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:19:58 (1763349598) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 22:20:04 (1763349604) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.486864 s, 25.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:20:14 (1763349614) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290877 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246291 s, 42.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:20:21 (1763349621) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:20:29 (1763349629) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000977373 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000982591 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108414 s, 945 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000831354 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 22:20:37 (1763349637) create test files with size 15757698 .. 2b124d1485a6447bd5799c366caf35a4 - 626021df18153a2fa7a436ad9caba755 - 18b06217b84adac5f8b5c069215907a0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 22:20:56 (1763349656) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (10s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:21:06 (1763349666) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:21:12 (1763349672) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628578 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316227 s, 33.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 22:21:19 (1763349679) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0661539 s, 31.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.287646 s, 14.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120182 s, 17.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 22:21:31 (1763349691) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:21:34 (1763349694) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:21:36 (1763349696) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042276 s, 24.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 22:21:42 (1763349702) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.242894 s, 13.0 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:21:49 (1763349709) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:21:57 (1763349717) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.298175 s, 35.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:22:07 (1763349727) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa0348304d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.211682 s, 49.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa034868ec000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:22:18 (1763349738) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:22:48 (1763349768) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:23:18 (1763349798) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:23:21 (1763349801) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209724 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.110298 s, 28.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x76cd2687 (stale) Mirror 2: 0x76cd2687 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 0c3536e9a3ebd2fac7b99e178427e554 - mirror 1 checksum: 2361177cd2797cfa90659bf87219fe3d - mirror 2 checksum: 0c3536e9a3ebd2fac7b99e178427e554 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 22:23:25 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 22:23:25 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0c3536e9a3ebd2fac7b99e178427e554 - mirror 1 checksum: 2361177cd2797cfa90659bf87219fe3d - mirror 2 checksum: 2361177cd2797cfa90659bf87219fe3d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 22:23:25 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 22:23:25 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:23:31 (1763349811) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307021 s, 34.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:23:37 (1763349817) Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 22:23:47 (1763349827) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 22:24:10 (1763349850) targets are mounted 22:24:10 (1763349850) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471269 s, 22.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:24:26 (1763349866) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.04414 s, 464 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162721 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:24:36 (1763349876) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0577278 s, 355 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00943661 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:24:45 (1763349885) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00680968 s, 2.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:24:52 (1763349892) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0781077 s, 262 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:24:59 (1763349899) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00917975 s, 446 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.048769 s, 84.0 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:25:06 (1763349906) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:25:26 (1763349926) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:25:53 (1763349953) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:26:15 (1763349975) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:26:38 (1763349998) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:26:44 (1763350004) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55878 mirror split pid 55879 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7635: 55878 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55879 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 22:27:50 (1763350070) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 71 skipping zero size read skipping zero length punch hole skipping zero length zero range 000004 1763350071.895950 write 0x03d64d thru 0x03ffff (0x29b3 bytes) 000005 1763350071.897280 mapread 0x01f536 thru 0x02494a (0x5415 bytes) truncating to largest ever: 0x28f30 000006 1763350071.924067 trunc from 0x040000 to 0x028f2f (0x170d0 bytes) 000007 1763350071.942183 mapread 0x0169b4 thru 0x01e913 (0x7f60 bytes) 000008 1763350071.953714 trunc from 0x028f30 to 0x003db5 (0x2517a bytes) 000009 1763350071.977609 trunc from 0x001e79 to 0x003db5 (0x1f3d bytes) 000010 1763350071.983294 read 0x000cdb thru 0x003348 (0x266e bytes) 000011 1763350071.990899 punch from 0x003835 to 0x003db5 (0x581 bytes) 000012 1763350071.998177 trunc from 0x003d6b to 0x003db5 (0x4b bytes) 000013 1763350072.002709 trunc from 0x003db6 to 0x001976 (0x243f bytes) 000014 1763350072.019841 write 0x020d5c thru 0x02f400 (0xe6a5 bytes) 000015 1763350072.022032 write 0x009807 thru 0x00f816 (0x6010 bytes) 000016 1763350072.032021 write 0x0224a5 thru 0x023143 (0xc9f bytes) 000017 1763350072.034688 read 0x015160 thru 0x01fadf (0xa980 bytes) fallocating to largest ever: 0x2b76c 000019 1763350072.044214 fallocate 0x025a41 thru 0x02b76b (0x5d2b bytes) 000020 1763350072.049185 fallocate 0x01e3db thru 0x0275a0 (0x91c6 bytes) 000022 1763350072.058069 trunc from 0x02932a to 0x0294e0 (0x1b7 bytes) 000023 1763350072.086358 read 0x013ce7 thru 0x017956 (0x3c70 bytes) 000025 1763350072.092601 mapread 0x00746e thru 0x00a9e3 (0x3576 bytes) 000026 1763350072.096159 write 0x002bb3 thru 0x011f0b (0xf359 bytes) 000027 1763350072.097968 mapwrite 0x034d4c thru 0x03d9d3 (0x8c88 bytes) 000028 1763350072.180500 mapread 0x03b625 thru 0x03d9d3 (0x23af bytes) fallocating to largest ever: 0x3240c 000029 1763350072.184276 fallocate 0x027e25 thru 0x03240b (0xa5e7 bytes) fallocating to largest ever: 0x3338b 000030 1763350072.193806 fallocate 0x02ac1a thru 0x03338a (0x8771 bytes) 000031 1763350072.202595 punch from 0x000ebf to 0x00739f (0x64e1 bytes) 000032 1763350072.215115 MIRROR_EXTEND to 2 mirrors 000033 1763350072.840214 MIRROR_RESYNC 2 mirrors 000034 1763350073.158938 punch from 0x034fc9 to 0x03a58a (0x55c2 bytes) zero_range to largest ever: 0x3c74e 000035 1763350073.231347 trunc from 0x0364d2 to 0x03c74d (0x627c bytes) 000036 1763350073.237536 fallocate 0x00cf1a thru 0x013c2b (0x6d12 bytes) 000037 1763350073.242562 mapwrite 0x03bee0 thru 0x03ffff (0x4120 bytes) 000038 1763350073.353441 read 0x006934 thru 0x014fdd (0xe6aa bytes) 000040 1763350073.371942 trunc from 0x021fc4 to 0x0260d7 (0x4114 bytes) 000042 1763350073.383891 MIRROR_RESYNC 2 mirrors 000043 1763350074.156186 fallocate 0x006ab3 thru 0x014924 (0xde72 bytes) 000044 1763350074.211105 punch from 0x00090f to 0x004de2 (0x44d4 bytes) 000045 1763350074.220426 mapwrite 0x00411f thru 0x008100 (0x3fe2 bytes) 000046 1763350074.265449 trunc from 0x00688c to 0x015f78 (0xf6ed bytes) 000047 1763350074.271894 write 0x027a34 thru 0x02f069 (0x7636 bytes) 000048 1763350074.284915 read 0x011a98 thru 0x01af2f (0x9498 bytes) 000049 1763350074.295133 read 0x018a42 thru 0x018ef2 (0x4b1 bytes) 000050 1763350074.296174 mapread 0x026f9d thru 0x035564 (0xe5c8 bytes) 000051 1763350074.306913 read 0x01e88b thru 0x02d510 (0xec86 bytes) 000052 1763350074.307512 trunc from 0x040000 to 0x00f2e1 (0x30d1e bytes) 000053 1763350074.356596 mapread 0x003d3f thru 0x00f09b (0xb35d bytes) 000054 1763350074.358480 trunc from 0x00f2e2 to 0x02bfc9 (0x1cce8 bytes) fallocating to largest ever: 0x3d1c6 000055 1763350074.380120 fallocate 0x0323be thru 0x03d1c5 (0xae08 bytes) 000056 1763350074.387248 mapwrite 0x028494 thru 0x033ddc (0xb949 bytes) 000057 1763350074.420908 mapwrite 0x03d4ec thru 0x03ffff (0x2b14 bytes) 000058 1763350074.483871 mapread 0x037448 thru 0x039a47 (0x2600 bytes) 000059 1763350074.484876 mapwrite 0x0316a5 thru 0x03eb16 (0xd472 bytes) 000060 1763350074.502501 write 0x017a8d thru 0x01aa7a (0x2fee bytes) 000061 1763350074.503609 punch from 0x02dc8b to 0x032dc9 (0x513f bytes) 000062 1763350074.512043 mapread 0x030fc1 thru 0x031239 (0x279 bytes) 000064 1763350074.525216 write 0x020e09 thru 0x02b676 (0xa86e bytes) 000065 1763350074.527004 write 0x0186bb thru 0x01ead4 (0x641a bytes) 000066 1763350074.530701 mapwrite 0x016933 thru 0x01bab1 (0x517f bytes) 000067 1763350074.556974 mapread 0x00ef95 thru 0x01c676 (0xd6e2 bytes) 000068 1763350074.558306 trunc from 0x040000 to 0x01965e (0x269a1 bytes) fallocating to largest ever: 0x3de05 000069 1763350074.577262 fallocate 0x032e13 thru 0x03de04 (0xaff2 bytes) 000070 1763350074.585936 mapread 0x0192c6 thru 0x01965e (0x399 bytes) 000071 1763350074.596538 MIRROR_RESYNC 2 mirrors 000072 1763350075.183347 read 0x00a7d4 thru 0x01795b (0xd188 bytes) 000073 1763350075.196265 trunc from 0x0109d4 to 0x01965e (0x8c8b bytes) 000074 1763350075.227603 punch from 0x015134 to 0x01965e (0x452b bytes) 000075 1763350075.234186 read 0x015528 thru 0x017aa4 (0x257d bytes) 000077 1763350075.245796 write 0x00351d thru 0x005d90 (0x2874 bytes) 000078 1763350075.246769 punch from 0x00b1a4 to 0x0143a9 (0x9206 bytes) 000079 1763350075.256047 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1965f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1965f): Mirror 2: 0xeafffd52 (stale) CRC-32 checksum value for chunk [0, 0x1965f): Mirror 1: 0x86f410a3 Mirror 3: 0x86f410a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1965f: skipped 000080 1763350076.014164 write 0x005316 thru 0x011870 (0xc55b bytes) fallocating to largest ever: 0x40000 000081 1763350076.056232 fallocate 0x03c8c9 thru 0x03ffff (0x3737 bytes) 000082 1763350076.068601 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa69042f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa69042f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000083 1763350076.426199 trunc from 0x040000 to 0x02b4fa (0x14b05 bytes) 000084 1763350076.481192 trunc from 0x02b4fb to 0x00439c (0x2715e bytes) 000085 1763350076.498163 read 0x003e39 thru 0x00439c (0x564 bytes) 000087 1763350076.506442 mapread 0x003706 thru 0x00439c (0xc97 bytes) 000088 1763350076.507601 trunc from 0x000b7d to 0x00439c (0x3820 bytes) 000089 1763350076.523261 trunc from 0x00439d to 0x034236 (0x2fe9a bytes) 000090 1763350076.549530 punch from 0x007f1e to 0x010c97 (0x8d7a bytes) 000091 1763350076.557393 write 0x002fc5 thru 0x00736d (0x43a9 bytes) 000092 1763350076.577832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000093 1763350076.771917 trunc from 0x023bed to 0x0273be (0x37d2 bytes) 000094 1763350076.776702 fallocate 0x00a95e thru 0x012509 (0x7bac bytes) 000096 1763350076.784907 mapwrite 0x037be7 thru 0x03ffff (0x8419 bytes) 000097 1763350076.852791 trunc from 0x032bab to 0x03a2a8 (0x76fe bytes) 000098 1763350076.858299 mapwrite 0x00580a thru 0x012713 (0xcf0a bytes) 000099 1763350076.908995 mapwrite 0x02abca thru 0x02fa32 (0x4e69 bytes) 000100 1763350076.936359 mapread 0x0075bf thru 0x013035 (0xba77 bytes) 000104 1763350076.949814 MIRROR_EXTEND to 2 mirrors 000105 1763350077.753276 trunc from 0x040000 to 0x020c8c (0x1f373 bytes) 000106 1763350077.838994 write 0x03436a thru 0x03739f (0x3036 bytes) 000108 1763350077.841582 write 0x007ad9 thru 0x01786b (0xfd93 bytes) 000109 1763350077.858292 write 0x010095 thru 0x012195 (0x2101 bytes) 000110 1763350077.859210 write 0x03722c thru 0x03ffff (0x8dd4 bytes) 000111 1763350077.860509 trunc from 0x03a853 to 0x03b0cd (0x87b bytes) 000112 1763350077.900020 mapwrite 0x02ca00 thru 0x038862 (0xbe63 bytes) 000113 1763350077.949438 trunc from 0x008bb7 to 0x00a4fc (0x1946 bytes) 000114 1763350077.955714 write 0x037488 thru 0x038590 (0x1109 bytes) 000115 1763350077.957015 mapwrite 0x0009dc thru 0x00c82d (0xbe52 bytes) 000116 1763350077.996554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000118 1763350078.150114 mapwrite 0x01277e thru 0x01af55 (0x87d8 bytes) 000119 1763350078.210209 write 0x0080f2 thru 0x00abd3 (0x2ae2 bytes) 000120 1763350078.228975 trunc from 0x040000 to 0x011c4c (0x2e3b3 bytes) 000121 1763350078.264486 read 0x00c4a6 thru 0x00d966 (0x14c1 bytes) 000122 1763350078.265409 write 0x0015e7 thru 0x00f135 (0xdb4f bytes) 000124 1763350078.274468 mapwrite 0x00b3d7 thru 0x011904 (0x652e bytes) 000125 1763350078.323713 fallocate 0x02d491 thru 0x02e164 (0xcd4 bytes) 000126 1763350078.330032 punch from 0x028b1d to 0x02e164 (0x5648 bytes) 000127 1763350078.338769 mapread 0x004e5b thru 0x00c82c (0x79d2 bytes) 000128 1763350078.339775 punch from 0x01cbd5 to 0x01e8b8 (0x1ce4 bytes) 000129 1763350078.347145 mapread 0x00b4a1 thru 0x017355 (0xbeb5 bytes) 000130 1763350078.357175 read 0x028f36 thru 0x02e164 (0x522f bytes) 000131 1763350078.371901 read 0x0035ec thru 0x00c9e3 (0x93f8 bytes) 000132 1763350078.372505 write 0x038102 thru 0x03ffff (0x7efe bytes) 000133 1763350078.373845 write 0x005ba1 thru 0x00d016 (0x7476 bytes) 000135 1763350078.376184 trunc from 0x00117f to 0x0027ce (0x1650 bytes) 000136 1763350078.380006 write 0x0323e0 thru 0x032b49 (0x76a bytes) 000137 1763350078.387797 mapwrite 0x00561a thru 0x0152bf (0xfca6 bytes) 000139 1763350078.413284 trunc from 0x022a76 to 0x031e36 (0xf3c1 bytes) 000140 1763350078.421253 read 0x0398b1 thru 0x03ffff (0x674f bytes) 000141 1763350078.422759 fallocate 0x01d2f0 thru 0x025b30 (0x8841 bytes) 000142 1763350078.433091 fallocate 0x000584 thru 0x000a64 (0x4e1 bytes) 000143 1763350078.438447 punch from 0x017c29 to 0x01ae9d (0x3275 bytes) 000144 1763350078.446972 read 0x02a245 thru 0x03333c (0x90f8 bytes) 000145 1763350078.457840 read 0x02dedd thru 0x030548 (0x266c bytes) 000146 1763350078.458373 punch from 0x01dc14 to 0x02120b (0x35f8 bytes) 000147 1763350078.464182 mapread 0x026cd8 thru 0x02ec13 (0x7f3c bytes) 000148 1763350078.465858 read 0x01186e thru 0x0191b1 (0x7944 bytes) 000149 1763350078.467569 read 0x01a5e4 thru 0x01f46c (0x4e89 bytes) 000151 1763350078.477357 trunc from 0x036e8e to 0x03ffff (0x9172 bytes) 000152 1763350078.482075 mapread 0x020d5f thru 0x02c0b5 (0xb357 bytes) 000153 1763350078.483099 read 0x004150 thru 0x008b37 (0x49e8 bytes) 000155 1763350078.484065 write 0x0071d9 thru 0x00ac6b (0x3a93 bytes) 000156 1763350078.485642 write 0x010e0f thru 0x019b7e (0x8d70 bytes) 000157 1763350078.488881 trunc from 0x01ac67 to 0x02a84a (0xfbe4 bytes) 000159 1763350078.499046 punch from 0x002aea to 0x00ac4e (0x8165 bytes) 000160 1763350078.521393 trunc from 0x0327b7 to 0x03ffff (0xd849 bytes) 000161 1763350078.531852 mapwrite 0x02969e thru 0x031565 (0x7ec8 bytes) 000162 1763350078.594175 write 0x019fca thru 0x01b37b (0x13b2 bytes) 000163 1763350078.595276 punch from 0x0020d3 to 0x00dbc7 (0xbaf5 bytes) 000164 1763350078.603111 trunc from 0x0354b1 to 0x0398e8 (0x4438 bytes) 000166 1763350078.611971 trunc from 0x02b6be to 0x038b1f (0xd462 bytes) 000167 1763350078.620537 fallocate 0x02d379 thru 0x03639a (0x9022 bytes) 000168 1763350078.651090 mapwrite 0x03eee1 thru 0x03ffff (0x111f bytes) 000169 1763350078.694531 punch from 0x036d3f to 0x03ffff (0x92c1 bytes) 000171 1763350078.716875 write 0x01de84 thru 0x023ec0 (0x603d bytes) 000173 1763350078.723015 punch from 0x0022b4 to 0x0054d0 (0x321d bytes) 000174 1763350078.731506 punch from 0x03f226 to 0x03ffff (0xdda bytes) 000175 1763350078.742371 trunc from 0x035bd7 to 0x03b771 (0x5b9b bytes) 000176 1763350078.750135 write 0x0192ea thru 0x023fb7 (0xacce bytes) 000177 1763350078.752482 mapwrite 0x0006a1 thru 0x00aa91 (0xa3f1 bytes) 000178 1763350078.833399 mapread 0x0372b7 thru 0x03ffff (0x8d49 bytes) 000179 1763350078.884827 write 0x000065 thru 0x0015d9 (0x1575 bytes) 000180 1763350078.886572 fallocate 0x01ed77 thru 0x0254b6 (0x6740 bytes) 000181 1763350078.898464 mapwrite 0x017d76 thru 0x024b4f (0xcdda bytes) 000182 1763350078.923588 fallocate 0x0169a3 thru 0x01bf97 (0x55f5 bytes) 000183 1763350078.941542 trunc from 0x040000 to 0x02c82d (0x137d2 bytes) 000185 1763350078.980365 mapread 0x021e1e thru 0x023834 (0x1a17 bytes) 000186 1763350078.981263 mapwrite 0x02d494 thru 0x0348e3 (0x7450 bytes) 000187 1763350079.078411 mapread 0x0249bd thru 0x029d63 (0x53a7 bytes) 000188 1763350079.079399 trunc from 0x0093e3 to 0x01610c (0xcd2a bytes) 000189 1763350079.085928 mapread 0x018fd6 thru 0x0234df (0xa50a bytes) 000190 1763350079.087596 trunc from 0x01afc2 to 0x021553 (0x6592 bytes) 000191 1763350079.098229 write 0x01b420 thru 0x029b35 (0xe716 bytes) 000193 1763350079.105835 fallocate 0x03b80f thru 0x03ffff (0x47f1 bytes) 000194 1763350079.112815 read 0x008e93 thru 0x011d1f (0x8e8d bytes) 000195 1763350079.127325 MIRROR_EXTEND to 2 mirrors 000196 1763350079.983224 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000198 1763350080.130176 trunc from 0x01a2f9 to 0x01fe14 (0x5b1c bytes) 000199 1763350080.156964 punch from 0x034351 to 0x03fd12 (0xb9c2 bytes) 000200 1763350080.167941 fallocate 0x03bcc7 thru 0x03ffff (0x4339 bytes) 000201 1763350080.174188 trunc from 0x0241f7 to 0x0341b7 (0xffc1 bytes) 000202 1763350080.181034 read 0x017915 thru 0x0211c3 (0x98af bytes) 000203 1763350080.191959 fallocate 0x00d45b thru 0x01ba8a (0xe630 bytes) 000204 1763350080.200050 write 0x02d611 thru 0x035efe (0x88ee bytes) 000205 1763350080.221965 mapread 0x03f6bc thru 0x03ffff (0x944 bytes) 000206 1763350080.236997 read 0x0130dc thru 0x017f5d (0x4e82 bytes) 000208 1763350080.246902 punch from 0x0269d5 to 0x02da66 (0x7092 bytes) 000209 1763350080.269104 write 0x01761a thru 0x01aaa7 (0x348e bytes) 000210 1763350080.270058 trunc from 0x040000 to 0x00c2b0 (0x33d4f bytes) 000211 1763350080.317217 punch from 0x008bc8 to 0x00c2b0 (0x36e9 bytes) 000212 1763350080.322760 fallocate 0x02d433 thru 0x0331ed (0x5dbb bytes) 000214 1763350080.327450 trunc from 0x005bfb to 0x00c2b0 (0x66b6 bytes) 000216 1763350080.333027 read 0x0005de thru 0x008154 (0x7b77 bytes) 000218 1763350080.347029 write 0x033a12 thru 0x03e1f6 (0xa7e5 bytes) 000219 1763350080.350235 write 0x01f1d9 thru 0x02cb65 (0xd98d bytes) 000220 1763350080.366248 fallocate 0x010540 thru 0x011c9b (0x175c bytes) 000221 1763350080.371277 fallocate 0x01959b thru 0x02651c (0xcf82 bytes) 000222 1763350080.379129 trunc from 0x03e1f7 to 0x010685 (0x2db71 bytes) 000224 1763350080.400978 write 0x039cae thru 0x03b372 (0x16c5 bytes) 000225 1763350080.402099 punch from 0x00e2f9 to 0x01a92f (0xc637 bytes) 000226 1763350080.408550 trunc from 0x03b373 to 0x012719 (0x28c59 bytes) 000227 1763350080.433597 mapread 0x008d55 thru 0x009f5c (0x1208 bytes) 000228 1763350080.434493 read 0x00728f thru 0x00cd41 (0x5ab3 bytes) 000229 1763350080.435075 write 0x03b9ef thru 0x03ec35 (0x3247 bytes) 000231 1763350080.439025 fallocate 0x039dce thru 0x03ffff (0x6232 bytes) 000232 1763350080.447378 punch from 0x02d56d to 0x03c0fc (0xeb90 bytes) 000233 1763350080.460420 write 0x02ab8d thru 0x02debf (0x3333 bytes) 000234 1763350080.475388 mapread 0x021c0e thru 0x02a93a (0x8d2d bytes) 000235 1763350080.476352 mapwrite 0x01032a thru 0x01e9f9 (0xe6d0 bytes) 000236 1763350080.499772 trunc from 0x024c50 to 0x026f47 (0x22f8 bytes) 000237 1763350080.508982 fallocate 0x03b4f6 thru 0x03ffff (0x4b0a bytes) 000238 1763350080.523411 MIRROR_EXTEND to 2 mirrors 000239 1763350081.120961 write 0x01eebd thru 0x02df25 (0xf069 bytes) 000240 1763350081.188214 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa1323f86 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe103b3f3 Mirror 3: 0xe103b3f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000241 1763350082.096775 mapwrite 0x03b184 thru 0x03ffff (0x4e7c bytes) 000242 1763350082.220489 trunc from 0x030082 to 0x035c32 (0x5bb1 bytes) 000243 1763350082.234779 mapwrite 0x0067b3 thru 0x01677f (0xffcd bytes) 000244 1763350082.277332 write 0x03b79e thru 0x03ffff (0x4862 bytes) 000245 1763350082.288995 MIRROR_RESYNC 3 mirrors 000246 1763350083.013566 mapread 0x026e71 thru 0x035916 (0xeaa6 bytes) 000247 1763350083.051081 write 0x026b54 thru 0x036745 (0xfbf2 bytes) 000248 1763350083.139178 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x218e0584 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x218e0584 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1763350083.429580 trunc from 0x040000 to 0x003c43 (0x3c3bc bytes) 000250 1763350083.473035 mapwrite 0x000c30 thru 0x00f220 (0xe5f1 bytes) 000251 1763350083.566412 fallocate 0x0210f2 thru 0x02b8f0 (0xa7ff bytes) 000252 1763350083.577380 mapread 0x00ea61 thru 0x00f220 (0x7c0 bytes) 000253 1763350083.578942 read 0x0047df thru 0x006279 (0x1a9b bytes) 000255 1763350083.583193 trunc from 0x00f221 to 0x02e7eb (0x1f5cb bytes) 000256 1763350083.609003 punch from 0x0256ac to 0x02cdcb (0x7720 bytes) 000257 1763350083.619289 trunc from 0x02e7ec to 0x00f343 (0x1f4a8 bytes) 000258 1763350083.641253 fallocate 0x00b0ee thru 0x010940 (0x5853 bytes) 000259 1763350083.652741 fallocate 0x02d2be thru 0x03924a (0xbf8d bytes) 000260 1763350083.662598 write 0x01c3a4 thru 0x02a736 (0xe393 bytes) 000261 1763350083.680987 mapwrite 0x004bf5 thru 0x010149 (0xb555 bytes) 000262 1763350083.712191 trunc from 0x03924b to 0x01532b (0x23f1f bytes) 000263 1763350083.730211 mapwrite 0x028855 thru 0x0362cc (0xda78 bytes) 000266 1763350083.786323 write 0x00c2a2 thru 0x012fe4 (0x6d43 bytes) 000267 1763350083.787446 trunc from 0x02aaac to 0x0362cc (0xb821 bytes) 000269 1763350083.794644 read 0x0301c5 thru 0x0362cc (0x6108 bytes) 000270 1763350083.811106 fallocate 0x030110 thru 0x032463 (0x2354 bytes) 000271 1763350083.819712 fallocate 0x020529 thru 0x021742 (0x121a bytes) 000272 1763350083.825968 write 0x02cbe2 thru 0x0303f9 (0x3818 bytes) 000273 1763350083.827011 trunc from 0x01b9a9 to 0x025412 (0x9a6a bytes) 000274 1763350083.832385 mapread 0x00f08f thru 0x01356e (0x44e0 bytes) 000275 1763350083.833441 trunc from 0x0362cd to 0x03f797 (0x94cb bytes) 000277 1763350083.860710 mapwrite 0x003492 thru 0x008665 (0x51d4 bytes) 000278 1763350083.891608 fallocate 0x02f3ea thru 0x03ea80 (0xf697 bytes) 000280 1763350083.911054 trunc from 0x00c8b5 to 0x010a12 (0x415e bytes) 000281 1763350083.919701 mapread 0x0226d2 thru 0x026fed (0x491c bytes) 000282 1763350083.946169 punch from 0x039274 to 0x03f797 (0x6524 bytes) 000283 1763350083.952075 read 0x00f6fc thru 0x012bba (0x34bf bytes) 000284 1763350083.967795 write 0x02ed56 thru 0x032037 (0x32e2 bytes) 000285 1763350083.969713 trunc from 0x001af0 to 0x010790 (0xeca1 bytes) 000286 1763350083.980828 write 0x025b8d thru 0x026d7b (0x11ef bytes) 000287 1763350083.982743 write 0x00f371 thru 0x011b4a (0x27da bytes) 000288 1763350083.993573 read 0x0376b5 thru 0x03f797 (0x80e3 bytes) 000289 1763350083.994301 fallocate 0x01e791 thru 0x023cc6 (0x5536 bytes) 000290 1763350084.002406 write 0x009730 thru 0x00de37 (0x4708 bytes) 000291 1763350084.016120 write 0x00ce51 thru 0x0100c1 (0x3271 bytes) 000292 1763350084.017754 write 0x01ad61 thru 0x023691 (0x8931 bytes) 000293 1763350084.020059 trunc from 0x00f1ef to 0x01ac39 (0xba4b bytes) 000294 1763350084.046438 read 0x01a464 thru 0x01c729 (0x22c6 bytes) 000295 1763350084.057638 mapread 0x03ca32 thru 0x03f797 (0x2d66 bytes) 000296 1763350084.059967 trunc from 0x03f798 to 0x00f75a (0x3003d bytes) 000297 1763350084.087581 fallocate 0x0213cf thru 0x02f452 (0xe084 bytes) 000298 1763350084.101052 trunc from 0x011e91 to 0x018875 (0x69e5 bytes) 000299 1763350084.118736 trunc from 0x02f453 to 0x026f8a (0x84c8 bytes) 000300 1763350084.152215 trunc from 0x0127d4 to 0x017cfd (0x552a bytes) 000301 1763350084.162485 trunc from 0x026f8b to 0x017ec4 (0xf0c6 bytes) 000302 1763350084.185134 fallocate 0x017a1d thru 0x02518a (0xd76e bytes) 000304 1763350084.196729 read 0x019b24 thru 0x02518a (0xb667 bytes) 000305 1763350084.220898 read 0x011421 thru 0x016c6c (0x584c bytes) 000306 1763350084.221628 trunc from 0x00e670 to 0x01b6ac (0xd03d bytes) 000307 1763350084.227362 punch from 0x00822a to 0x016dc3 (0xeb9a bytes) 000308 1763350084.235385 mapread 0x020151 thru 0x02518a (0x503a bytes) 000309 1763350084.237298 mapread 0x000091 thru 0x009256 (0x91c6 bytes) 000310 1763350084.250041 fallocate 0x02ba34 thru 0x030a31 (0x4ffe bytes) 000311 1763350084.256103 write 0x00a9e1 thru 0x0157dc (0xadfc bytes) 000312 1763350084.257464 trunc from 0x030a32 to 0x031e9a (0x1469 bytes) 000313 1763350084.283276 mapwrite 0x02faee thru 0x03cfd3 (0xd4e6 bytes) 000314 1763350084.343181 punch from 0x0033ce to 0x00f019 (0xbc4c bytes) 000315 1763350084.354509 trunc from 0x018f62 to 0x01996f (0xa0e bytes) 000316 1763350084.362225 trunc from 0x03cfd4 to 0x0312e1 (0xbcf2 bytes) 000318 1763350084.388160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000320 1763350084.539703 write 0x002236 thru 0x00742b (0x51f6 bytes) 000321 1763350084.571129 mapread 0x00da20 thru 0x011241 (0x3822 bytes) 000323 1763350084.583331 fallocate 0x02cac7 thru 0x0384c9 (0xba03 bytes) 000324 1763350084.590398 trunc from 0x012989 to 0x0140a7 (0x171f bytes) 000325 1763350084.595377 mapread 0x005a7e thru 0x01180c (0xbd8f bytes) 000326 1763350084.596471 write 0x004b65 thru 0x00e775 (0x9c11 bytes) 000327 1763350084.598127 mapread 0x023d01 thru 0x032f28 (0xf228 bytes) 000329 1763350084.614381 fallocate 0x035994 thru 0x03ffff (0xa66c bytes) 000330 1763350084.620407 mapread 0x027d4d thru 0x0333be (0xb672 bytes) 000331 1763350084.621453 fallocate 0x00c3fa thru 0x0164f2 (0xa0f9 bytes) 000332 1763350084.626089 fallocate 0x0176ca thru 0x01de95 (0x67cc bytes) 000333 1763350084.634634 mapwrite 0x0274ee thru 0x03542f (0xdf42 bytes) 000334 1763350084.669187 trunc from 0x040000 to 0x03b8cb (0x4734 bytes) 000336 1763350084.692625 fallocate 0x0382d2 thru 0x03ffff (0x7d2e bytes) 000337 1763350084.699107 punch from 0x0134d6 to 0x01a509 (0x7034 bytes) 000339 1763350084.710715 trunc from 0x00619d to 0x0085d1 (0x2435 bytes) 000341 1763350084.721911 MIRROR_EXTEND to 2 mirrors 000342 1763350085.596873 trunc from 0x03b8cc to 0x02c398 (0xf533 bytes) 000343 1763350085.640251 mapwrite 0x01d278 thru 0x022fb0 (0x5d39 bytes) 000345 1763350085.688322 write 0x0253e7 thru 0x02978e (0x43a8 bytes) 000346 1763350085.694996 punch from 0x020411 to 0x02c398 (0xbf88 bytes) 000347 1763350085.708916 read 0x026f0a thru 0x02c398 (0x548f bytes) 000348 1763350085.730226 mapread 0x0161ac thru 0x01fb39 (0x998e bytes) 000349 1763350085.740801 write 0x004498 thru 0x00cd42 (0x88ab bytes) 000350 1763350085.750033 trunc from 0x024599 to 0x02c398 (0x7e00 bytes) 000351 1763350085.758373 mapread 0x01c147 thru 0x0237e6 (0x76a0 bytes) 000352 1763350085.767959 write 0x0030f5 thru 0x005805 (0x2711 bytes) 000353 1763350085.769060 trunc from 0x02c399 to 0x0018f3 (0x2aaa5 bytes) 000354 1763350085.811795 trunc from 0x000313 to 0x0018f3 (0x15e1 bytes) 000355 1763350085.823505 punch from 0x00038d to 0x0018f3 (0x1567 bytes) 000356 1763350085.830723 trunc from 0x0018f4 to 0x00e897 (0xcfa4 bytes) 000357 1763350085.857104 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe898): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe898): Mirror 2: 0xc87ceb0c (stale) CRC-32 checksum value for chunk [0, 0xe898): Mirror 1: 0x14813168 Mirror 3: 0x14813168 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe898: skipped 000358 1763350086.731553 mapread 0x004f34 thru 0x00e897 (0x9964 bytes) 000359 1763350086.753359 write 0x010993 thru 0x014463 (0x3ad1 bytes) 000360 1763350086.795232 fallocate 0x03acc4 thru 0x03ffff (0x533c bytes) 000363 1763350086.814338 trunc from 0x008880 to 0x010747 (0x7ec8 bytes) 000365 1763350086.847912 mapread 0x008a47 thru 0x015586 (0xcb40 bytes) 000366 1763350086.872040 punch from 0x007150 to 0x009b8c (0x2a3d bytes) 000367 1763350086.876790 mapread 0x03ef38 thru 0x03ffff (0x10c8 bytes) 000369 1763350086.893873 MIRROR_RESYNC 3 mirrors 000370 1763350087.662545 mapread 0x00925e thru 0x00f4e7 (0x628a bytes) 000371 1763350087.685893 punch from 0x026a3a to 0x02e7a1 (0x7d68 bytes) 000372 1763350087.729918 mapread 0x00ebce thru 0x019fab (0xb3de bytes) 000373 1763350087.760552 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x256a6c92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x256a6c92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000374 1763350088.059723 mapread 0x035c3f thru 0x03b89c (0x5c5e bytes) 000375 1763350088.085453 write 0x032bea thru 0x03ffff (0xd416 bytes) 000376 1763350088.094320 write 0x03c934 thru 0x03ffff (0x36cc bytes) 000377 1763350088.095302 punch from 0x0345ec to 0x03cca1 (0x86b6 bytes) 000378 1763350088.118068 mapwrite 0x018609 thru 0x01ac36 (0x262e bytes) 000379 1763350088.151411 mapwrite 0x03578a thru 0x03f1e5 (0x9a5c bytes) 000380 1763350088.191385 read 0x03a95f thru 0x03ffff (0x56a1 bytes) 000381 1763350088.194998 trunc from 0x03618b to 0x03868e (0x2504 bytes) 000382 1763350088.206026 punch from 0x01d807 to 0x02be75 (0xe66f bytes) 000383 1763350088.213188 punch from 0x01eae9 to 0x02816c (0x9684 bytes) 000384 1763350088.222725 read 0x02289f thru 0x0288c4 (0x6026 bytes) 000385 1763350088.241159 mapwrite 0x004990 thru 0x0099fc (0x506d bytes) 000386 1763350088.256878 write 0x008f6e thru 0x00b30a (0x239d bytes) 000388 1763350088.259444 mapwrite 0x0115df thru 0x017526 (0x5f48 bytes) 000389 1763350088.285384 MIRROR_RESYNC 2 mirrors 000390 1763350088.992189 trunc from 0x040000 to 0x0301a9 (0xfe56 bytes) 000391 1763350089.064190 punch from 0x0061e4 to 0x00e067 (0x7e84 bytes) 000392 1763350089.071753 punch from 0x0295bb to 0x02aa81 (0x14c7 bytes) 000393 1763350089.081900 read 0x0235bf thru 0x02e77e (0xb1c0 bytes) 000394 1763350089.090175 read 0x004412 thru 0x00c120 (0x7d0f bytes) 000395 1763350089.099614 read 0x0136e6 thru 0x019250 (0x5b6b bytes) 000396 1763350089.100721 mapread 0x0147fc thru 0x01adea (0x65ef bytes) 000397 1763350089.102381 mapwrite 0x03e7fb thru 0x03ffff (0x1805 bytes) 000398 1763350089.169791 mapwrite 0x01cd12 thru 0x01d704 (0x9f3 bytes) 000399 1763350089.193755 mapread 0x01f0be thru 0x026a6d (0x79b0 bytes) 000400 1763350089.194610 fallocate 0x02c898 thru 0x0307d5 (0x3f3e bytes) 000401 1763350089.201390 trunc from 0x040000 to 0x008aae (0x37551 bytes) 000402 1763350089.226304 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8aaf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8aaf): Mirror 2: 0x39298649 (stale) CRC-32 checksum value for chunk [0, 0x8aaf): Mirror 1: 0x29dac2b9 Mirror 3: 0x29dac2b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8aaf: skipped 000403 1763350089.953595 trunc from 0x008aaf to 0x032728 (0x29c7a bytes) 000404 1763350090.041480 mapread 0x01e5b1 thru 0x02dffd (0xfa4d bytes) 000406 1763350090.079721 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32729): Mirror 3: 0x46dafcd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32729): Mirror 3: 0x46dafcd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32729: skipped 000407 1763350090.359390 read 0x030f40 thru 0x032728 (0x17e9 bytes) 000408 1763350090.366102 mapwrite 0x00ebef thru 0x015ac4 (0x6ed6 bytes) 000409 1763350090.412196 read 0x02a904 thru 0x0320b4 (0x77b1 bytes) 000410 1763350090.433399 trunc from 0x032729 to 0x026209 (0xc51f bytes) 000411 1763350090.458365 mapread 0x005862 thru 0x010d38 (0xb4d7 bytes) 000412 1763350090.459386 trunc from 0x02620a to 0x004650 (0x21bb9 bytes) 000413 1763350090.479871 write 0x01b85f thru 0x020d15 (0x54b7 bytes) 000414 1763350090.481154 read 0x016a96 thru 0x020d15 (0xa280 bytes) 000415 1763350090.501149 trunc from 0x020d16 to 0x012ebc (0xde59 bytes) 000416 1763350090.535633 punch from 0x00d4f1 to 0x012ebc (0x59cc bytes) 000417 1763350090.543496 read 0x0000c8 thru 0x00a96d (0xa8a6 bytes) 000418 1763350090.544255 mapwrite 0x030657 thru 0x0317b5 (0x115f bytes) 000419 1763350090.592436 fallocate 0x025f1b thru 0x02878b (0x2871 bytes) 000420 1763350090.598762 fallocate 0x02403c thru 0x02ea80 (0xaa45 bytes) 000421 1763350090.605425 fallocate 0x0196a5 thru 0x019f3f (0x89b bytes) 000423 1763350090.616922 fallocate 0x02d4c9 thru 0x03010a (0x2c42 bytes) 000424 1763350090.630258 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000425 1763350090.800512 fallocate 0x0250a6 thru 0x02a1ab (0x5106 bytes) 000426 1763350090.806257 trunc from 0x0159da to 0x022f0a (0xd531 bytes) 000428 1763350090.815911 mapwrite 0x00837b thru 0x0122dd (0x9f63 bytes) 000429 1763350090.869601 fallocate 0x028aef thru 0x02c8ef (0x3e01 bytes) 000430 1763350090.876269 write 0x000e12 thru 0x00670e (0x58fd bytes) 000431 1763350090.886831 read 0x00b636 thru 0x0127cd (0x7198 bytes) 000432 1763350090.887467 trunc from 0x00ae51 to 0x012890 (0x7a40 bytes) 000433 1763350090.898401 mapwrite 0x02fbcd thru 0x03f479 (0xf8ad bytes) 000434 1763350090.968323 mapread 0x00b39a thru 0x018f8e (0xdbf5 bytes) 000435 1763350090.995201 fallocate 0x019caa thru 0x01faef (0x5e46 bytes) 000436 1763350091.006228 punch from 0x00d1ce to 0x010765 (0x3598 bytes) 000438 1763350091.012991 mapwrite 0x01c1e2 thru 0x029f6e (0xdd8d bytes) 000439 1763350091.059172 mapread 0x01c410 thru 0x02ad55 (0xe946 bytes) 000440 1763350091.060193 mapwrite 0x00d548 thru 0x015168 (0x7c21 bytes) 000441 1763350091.086125 mapwrite 0x01e313 thru 0x028794 (0xa482 bytes) 000442 1763350091.104492 fallocate 0x00f459 thru 0x011dc0 (0x2968 bytes) 000443 1763350091.109880 write 0x02ad5c thru 0x034a52 (0x9cf7 bytes) 000445 1763350091.112089 mapwrite 0x035bba thru 0x03ffff (0xa446 bytes) 000446 1763350091.150891 mapwrite 0x03f63e thru 0x03ffff (0x9c2 bytes) 000448 1763350091.169500 write 0x026f20 thru 0x0315bf (0xa6a0 bytes) 000449 1763350091.173524 write 0x0369b6 thru 0x037efb (0x1546 bytes) 000451 1763350091.176543 trunc from 0x040000 to 0x0242a3 (0x1bd5c bytes) 000452 1763350091.197427 fallocate 0x02c479 thru 0x038b23 (0xc6ab bytes) 000454 1763350091.203985 mapread 0x0336c0 thru 0x038b23 (0x5464 bytes) 000456 1763350091.213775 trunc from 0x038b24 to 0x00f0e3 (0x29a40 bytes) 000457 1763350091.231454 fallocate 0x03b5e5 thru 0x03ffff (0x4a1b bytes) 000458 1763350091.236716 fallocate 0x03e7f9 thru 0x03ffff (0x1807 bytes) 000459 1763350091.241696 trunc from 0x040000 to 0x006b3b (0x394c4 bytes) 000460 1763350091.271228 fallocate 0x039f42 thru 0x03ffff (0x60be bytes) 000461 1763350091.279822 mapwrite 0x015764 thru 0x022ee5 (0xd782 bytes) 000462 1763350091.332093 trunc from 0x040000 to 0x03db94 (0x246b bytes) 000463 1763350091.359507 trunc from 0x03db95 to 0x03cb4c (0x1048 bytes) 000464 1763350091.397104 trunc from 0x039a89 to 0x03cb4c (0x30c4 bytes) 000466 1763350091.407644 trunc from 0x0373e4 to 0x03cb4c (0x5769 bytes) 000467 1763350091.415078 trunc from 0x03cb4d to 0x01f4e6 (0x1d666 bytes) 000468 1763350091.435586 mapwrite 0x036647 thru 0x039bb2 (0x356c bytes) 000469 1763350091.482877 trunc from 0x039bb3 to 0x032489 (0x7729 bytes) 000470 1763350091.501916 punch from 0x030d3e to 0x032489 (0x174c bytes) 000471 1763350091.507399 fallocate 0x030268 thru 0x03b2f6 (0xb08f bytes) 000472 1763350091.515859 mapread 0x012443 thru 0x0159c0 (0x357e bytes) 000474 1763350091.518172 read 0x00d3a6 thru 0x01306c (0x5cc7 bytes) 000475 1763350091.531219 mapread 0x02d6be thru 0x034833 (0x7176 bytes) 000476 1763350091.532220 fallocate 0x01c858 thru 0x02bf2b (0xf6d4 bytes) 000478 1763350091.538048 punch from 0x02bb3b to 0x035447 (0x990d bytes) 000480 1763350091.544942 read 0x0206bf thru 0x0271b0 (0x6af2 bytes) 000482 1763350091.546071 trunc from 0x03b2f7 to 0x0267cb (0x14b2b bytes) 000483 1763350091.570513 mapread 0x01febb thru 0x021179 (0x12bf bytes) 000484 1763350091.572134 read 0x008a5b thru 0x01515f (0xc705 bytes) 000485 1763350091.574958 trunc from 0x0267cc to 0x01a54c (0xc27f bytes) 000487 1763350091.604123 punch from 0x00426b to 0x00bf3e (0x7cd4 bytes) 000488 1763350091.613756 trunc from 0x012188 to 0x014cc4 (0x2b3d bytes) 000489 1763350091.619946 trunc from 0x01a54d to 0x006138 (0x14414 bytes) 000490 1763350091.637855 read 0x0015c1 thru 0x006138 (0x4b78 bytes) 000492 1763350091.647509 write 0x0253a8 thru 0x0302a1 (0xaefa bytes) 000493 1763350091.649292 write 0x031c06 thru 0x0359f4 (0x3def bytes) 000494 1763350091.651043 trunc from 0x0359f5 to 0x018ef7 (0x1cafd bytes) 000495 1763350091.680824 punch from 0x0000a9 to 0x00b76d (0xb6c5 bytes) 000496 1763350091.689504 write 0x00c600 thru 0x0100be (0x3abf bytes) 000497 1763350091.699580 mapwrite 0x02e169 thru 0x033c6b (0x5b03 bytes) 000498 1763350091.751330 MIRROR_EXTEND to 2 mirrors 000499 1763350092.409216 mapwrite 0x004b26 thru 0x00f0b9 (0xa594 bytes) 000501 1763350092.528457 punch from 0x001832 to 0x00ca00 (0xb1cf bytes) 000502 1763350092.542065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000503 1763350092.667148 mapwrite 0x00c64f thru 0x01830e (0xbcc0 bytes) 000506 1763350092.722194 read 0x005d84 thru 0x0145bf (0xe83c bytes) 000508 1763350092.729512 trunc from 0x033c6c to 0x0239e9 (0x10282 bytes) 000509 1763350092.774943 write 0x00d5f9 thru 0x0180ed (0xaaf5 bytes) 000510 1763350092.776220 fallocate 0x00317b thru 0x0067d3 (0x3659 bytes) 000511 1763350092.785778 punch from 0x001efe to 0x00f007 (0xd10a bytes) 000512 1763350092.810468 fallocate 0x000422 thru 0x006f9a (0x6b79 bytes) 000513 1763350092.817493 fallocate 0x020b84 thru 0x0233d1 (0x284e bytes) 000514 1763350092.823637 punch from 0x00fd35 to 0x016fb6 (0x7282 bytes) 000515 1763350092.828327 write 0x0348e6 thru 0x0372ee (0x2a09 bytes) 000516 1763350092.829338 trunc from 0x0286b9 to 0x0372ee (0xec36 bytes) 000518 1763350092.844475 write 0x0266d3 thru 0x02dad3 (0x7401 bytes) 000519 1763350092.860026 trunc from 0x0372ef to 0x012656 (0x24c98 bytes) 000520 1763350092.888169 trunc from 0x012657 to 0x01a300 (0x7caa bytes) 000522 1763350092.913048 fallocate 0x01c38e thru 0x023564 (0x71d7 bytes) 000524 1763350092.919483 punch from 0x00afc4 to 0x01241f (0x745c bytes) 000525 1763350092.929627 MIRROR_EXTEND to 2 mirrors 000526 1763350093.615089 mapread 0x001fd1 thru 0x00f759 (0xd789 bytes) 000528 1763350093.637678 punch from 0x015d14 to 0x016b81 (0xe6e bytes) 000529 1763350093.676790 trunc from 0x00bbe2 to 0x0195c4 (0xd9e3 bytes) 000530 1763350093.682009 fallocate 0x016905 thru 0x01f955 (0x9051 bytes) 000531 1763350093.688226 read 0x00361c thru 0x0114c4 (0xdea9 bytes) 000532 1763350093.703183 mapwrite 0x017787 thru 0x01e998 (0x7212 bytes) 000533 1763350093.767488 read 0x002cb7 thru 0x005e98 (0x31e2 bytes) 000534 1763350093.774448 fallocate 0x01568a thru 0x016574 (0xeeb bytes) 000535 1763350093.779386 mapread 0x012815 thru 0x01e998 (0xc184 bytes) 000539 1763350093.781789 read 0x00fd16 thru 0x016a9f (0x6d8a bytes) 000540 1763350093.782371 fallocate 0x00a24c thru 0x0152ce (0xb083 bytes) 000541 1763350093.791646 write 0x036065 thru 0x03fb3f (0x9adb bytes) 000542 1763350093.793378 fallocate 0x02381b thru 0x0310e6 (0xd8cc bytes) 000543 1763350093.803799 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fb40): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb40): Mirror 2: 0xfe88ebf7 (stale) CRC-32 checksum value for chunk [0, 0x3fb40): Mirror 1: 0xa98196c3 Mirror 3: 0xa98196c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb40: skipped 000544 1763350094.862077 fallocate 0x03b43e thru 0x03ffff (0x4bc2 bytes) 000545 1763350094.922515 punch from 0x02acbd to 0x037e91 (0xd1d5 bytes) 000546 1763350094.929854 punch from 0x02dabb to 0x03c7c6 (0xed0c bytes) 000547 1763350094.935011 read 0x01abf4 thru 0x028f90 (0xe39d bytes) 000549 1763350094.947513 mapread 0x01cd3b thru 0x0219ec (0x4cb2 bytes) 000550 1763350094.949447 fallocate 0x0221a3 thru 0x02481f (0x267d bytes) 000551 1763350094.953940 read 0x03cd16 thru 0x03ffff (0x32ea bytes) 000552 1763350094.965690 mapwrite 0x016176 thru 0x01f54c (0x93d7 bytes) 000553 1763350094.986918 fallocate 0x013203 thru 0x017897 (0x4695 bytes) 000554 1763350094.996944 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc7d0d8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc7d0d8b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000555 1763350095.374602 trunc from 0x040000 to 0x03b00a (0x4ff5 bytes) 000556 1763350095.423256 trunc from 0x03b00b to 0x004c40 (0x363ca bytes) 000557 1763350095.447897 trunc from 0x001ef5 to 0x003fdf (0x20eb bytes) 000558 1763350095.453327 trunc from 0x004c41 to 0x027a02 (0x22dc2 bytes) 000559 1763350095.471585 punch from 0x01d707 to 0x01da82 (0x37c bytes) 000560 1763350095.489580 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27a03): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a03): Mirror 2: 0xa1c7744c (stale) CRC-32 checksum value for chunk [0, 0x27a03): Mirror 1: 0xa1c7744c Mirror 3: 0xa1c7744c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a03: skipped 000561 1763350096.253220 mapread 0x00cc67 thru 0x00f7ed (0x2b87 bytes) 000562 1763350096.274224 trunc from 0x01d003 to 0x0216b0 (0x46ae bytes) 000563 1763350096.323280 write 0x022c50 thru 0x029747 (0x6af8 bytes) 000564 1763350096.344517 mapread 0x01c5b4 thru 0x02017b (0x3bc8 bytes) 000565 1763350096.345394 trunc from 0x029748 to 0x00aeff (0x1e848 bytes) 000566 1763350096.388832 mapread 0x002e8a thru 0x00a666 (0x77dd bytes) 000567 1763350096.399551 mapread 0x0014c6 thru 0x0089c5 (0x7500 bytes) 000568 1763350096.400459 mapwrite 0x038751 thru 0x03fede (0x778e bytes) 000569 1763350096.477321 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fedf): Mirror 3: 0x67ac5e79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fedf): Mirror 3: 0x67ac5e79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fedf: skipped 000570 1763350096.826383 trunc from 0x03fedf to 0x03d928 (0x25b6 bytes) 000572 1763350096.854852 mapread 0x012da0 thru 0x019932 (0x6b93 bytes) 000573 1763350096.873913 write 0x0284cf thru 0x034320 (0xbe52 bytes) 000574 1763350096.888692 mapwrite 0x0038ca thru 0x00d89c (0x9fd3 bytes) 000575 1763350096.916180 write 0x02ae9d thru 0x038e7f (0xdfe3 bytes) 000576 1763350096.924526 mapwrite 0x007d57 thru 0x00b6c1 (0x396b bytes) 000577 1763350096.964562 punch from 0x030134 to 0x031ebe (0x1d8b bytes) 000578 1763350096.973579 read 0x001cc0 thru 0x00d6da (0xba1b bytes) 000579 1763350096.978762 trunc from 0x02d163 to 0x03a642 (0xd4e0 bytes) 000580 1763350096.990739 write 0x02dd95 thru 0x031bf9 (0x3e65 bytes) 000581 1763350097.008211 fallocate 0x0213ad thru 0x027ef9 (0x6b4d bytes) 000582 1763350097.017639 fallocate 0x03139f thru 0x03bb3f (0xa7a1 bytes) 000583 1763350097.025539 punch from 0x0182bf to 0x0275c1 (0xf303 bytes) 000584 1763350097.035929 mapread 0x02fc2c thru 0x03d928 (0xdcfd bytes) 000585 1763350097.048513 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d929): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d929): Mirror 3: 0x13379ad (stale) CRC-32 checksum value for chunk [0, 0x3d929): Mirror 1: 0x2fa35712 Mirror 4: 0x2fa35712 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d929: skipped 000586 1763350098.019431 mapread 0x038e21 thru 0x03b75b (0x293b bytes) 000588 1763350098.049263 mapread 0x0266ce thru 0x033506 (0xce39 bytes) 000590 1763350098.085473 read 0x013271 thru 0x018fed (0x5d7d bytes) 000592 1763350098.112236 MIRROR_RESYNC 3 mirrors 000593 1763350099.177523 mapread 0x000bbb thru 0x007594 (0x69da bytes) 000594 1763350099.218536 MIRROR_EXTEND to 4 mirrors 000595 1763350099.992335 write 0x028d32 thru 0x0342f5 (0xb5c4 bytes) 000596 1763350100.042923 read 0x021217 thru 0x02c76d (0xb557 bytes) 000597 1763350100.050632 trunc from 0x03af20 to 0x03d928 (0x2a09 bytes) 000598 1763350100.057454 trunc from 0x03d929 to 0x0260a2 (0x17886 bytes) 000599 1763350100.078920 trunc from 0x0260a3 to 0x02506e (0x1034 bytes) 000600 1763350100.102844 fallocate 0x01dbac thru 0x028e78 (0xb2cd bytes) 000601 1763350100.108970 trunc from 0x004ae9 to 0x013b1f (0xf037 bytes) 000602 1763350100.118267 mapwrite 0x02e578 thru 0x0306a8 (0x2131 bytes) 000604 1763350100.192453 trunc from 0x0306a9 to 0x01622a (0x1a47e bytes) 000605 1763350100.215430 punch from 0x002282 to 0x00ed61 (0xcae0 bytes) 000606 1763350100.220838 read 0x00e507 thru 0x0114eb (0x2fe5 bytes) 000607 1763350100.231860 fallocate 0x024d47 thru 0x02c2e5 (0x759f bytes) 000608 1763350100.242611 trunc from 0x011f4d to 0x016dff (0x4eb3 bytes) 000609 1763350100.248134 write 0x039b93 thru 0x03ffff (0x646d bytes) 000610 1763350100.249483 read 0x039502 thru 0x03ffff (0x6afe bytes) 000611 1763350100.254595 MIRROR_RESYNC 4 mirrors 000612 1763350101.104613 trunc from 0x040000 to 0x021f95 (0x1e06a bytes) 000613 1763350101.185821 trunc from 0x008857 to 0x00ca67 (0x4211 bytes) 000614 1763350101.194522 mapwrite 0x027708 thru 0x02a097 (0x2990 bytes) 000615 1763350101.293819 trunc from 0x01852b to 0x01c253 (0x3d29 bytes) 000616 1763350101.302321 trunc from 0x02a098 to 0x034689 (0xa5f2 bytes) 000618 1763350101.320834 MIRROR_RESYNC 4 mirrors 000619 1763350102.380814 MIRROR_EXTEND to 5 mirrors 000620 1763350103.221411 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000621 1763350103.615615 read 0x02496d thru 0x026b0a (0x219e bytes) 000622 1763350103.625395 fallocate 0x01c719 thru 0x024821 (0x8109 bytes) 000623 1763350103.680471 trunc from 0x03468a to 0x01fc0c (0x14a7d bytes) 000624 1763350103.719993 mapread 0x01d3c7 thru 0x01fc0c (0x2846 bytes) 000625 1763350103.763953 fallocate 0x026a3a thru 0x034bd0 (0xe197 bytes) 000626 1763350103.771520 fallocate 0x0179d6 thru 0x01d0f2 (0x571d bytes) 000627 1763350103.780290 write 0x00f936 thru 0x01f319 (0xf9e4 bytes) 000628 1763350103.803087 mapwrite 0x035226 thru 0x036fab (0x1d86 bytes) 000629 1763350103.852198 mapwrite 0x03f8fb thru 0x03ffff (0x705 bytes) 000630 1763350103.903767 write 0x02d516 thru 0x031fcc (0x4ab7 bytes) 000631 1763350103.921126 fallocate 0x00b021 thru 0x0125f8 (0x75d8 bytes) 000632 1763350103.926569 trunc from 0x040000 to 0x02518e (0x1ae71 bytes) 000633 1763350103.955548 punch from 0x01ef45 to 0x02518e (0x624a bytes) 000634 1763350103.967149 punch from 0x01062e to 0x010dfe (0x7d1 bytes) 000635 1763350103.973167 trunc from 0x02518f to 0x01df2a (0x7264 bytes) 000636 1763350104.005348 punch from 0x00c7db to 0x0185da (0xbe00 bytes) 000637 1763350104.014932 fallocate 0x001b6a thru 0x00e396 (0xc82d bytes) 000638 1763350104.024626 mapread 0x00a488 thru 0x00fb22 (0x569b bytes) 000639 1763350104.040312 read 0x007b6c thru 0x016141 (0xe5d6 bytes) 000640 1763350104.051117 fallocate 0x02a48c thru 0x02db8d (0x3702 bytes) 000641 1763350104.057109 punch from 0x009658 to 0x0190f9 (0xfaa2 bytes) 000642 1763350104.064125 mapwrite 0x0052e4 thru 0x00d0a0 (0x7dbd bytes) 000643 1763350104.129865 punch from 0x02d142 to 0x02db8d (0xa4c bytes) 000644 1763350104.135475 mapwrite 0x0196cf thru 0x02636f (0xcca1 bytes) 000645 1763350104.157831 fallocate 0x0343f9 thru 0x03cf2c (0x8b34 bytes) 000646 1763350104.162246 punch from 0x00907e to 0x0153c1 (0xc344 bytes) 000647 1763350104.166582 punch from 0x020b6a to 0x029016 (0x84ad bytes) 000648 1763350104.173343 fallocate 0x0100b9 thru 0x01e494 (0xe3dc bytes) 000649 1763350104.179848 trunc from 0x02db8e to 0x01be17 (0x11d76 bytes) 000650 1763350104.197436 write 0x000e8e thru 0x00bdf5 (0xaf68 bytes) 000651 1763350104.206047 write 0x027978 thru 0x02f626 (0x7caf bytes) 000652 1763350104.207608 read 0x02c1cd thru 0x02eb8b (0x29bf bytes) 000654 1763350104.208695 punch from 0x01e334 to 0x02345e (0x512b bytes) 000655 1763350104.212622 mapwrite 0x037d88 thru 0x03e2f5 (0x656e bytes) 000656 1763350104.259746 trunc from 0x0320e0 to 0x032fb3 (0xed4 bytes) 000657 1763350104.267908 write 0x0285c1 thru 0x0296d6 (0x1116 bytes) 000658 1763350104.269971 trunc from 0x03e2f6 to 0x00774d (0x36ba8 bytes) 000659 1763350104.289573 fallocate 0x01e8a4 thru 0x02249b (0x3bf8 bytes) 000660 1763350104.301462 punch from 0x01efe8 to 0x02249b (0x34b4 bytes) 000661 1763350104.309030 trunc from 0x02249c to 0x036cff (0x14864 bytes) 000663 1763350104.325509 punch from 0x02bce5 to 0x036cff (0xb01b bytes) 000664 1763350104.335843 trunc from 0x036d00 to 0x004f64 (0x31d9b bytes) 000665 1763350104.354282 mapwrite 0x020ae2 thru 0x025e41 (0x5360 bytes) 000666 1763350104.420002 trunc from 0x025e42 to 0x02583c (0x605 bytes) 000667 1763350104.444132 write 0x02936a thru 0x02e28f (0x4f26 bytes) 000669 1763350104.446044 fallocate 0x03851a thru 0x03b713 (0x31fa bytes) 000670 1763350104.458438 read 0x0392ea thru 0x03b713 (0x242a bytes) 000671 1763350104.470068 punch from 0x014bef to 0x0187e8 (0x3bfa bytes) 000672 1763350104.475284 punch from 0x02a19a to 0x02e9d5 (0x483c bytes) 000673 1763350104.493131 trunc from 0x03a8bb to 0x03b713 (0xe59 bytes) 000674 1763350104.498346 punch from 0x01312b to 0x01eada (0xb9b0 bytes) 000675 1763350104.507105 trunc from 0x01ff23 to 0x020e9d (0xf7b bytes) 000676 1763350104.514795 trunc from 0x03b714 to 0x03a916 (0xdfd bytes) 000677 1763350104.533826 write 0x03647c thru 0x03ffff (0x9b84 bytes) 000678 1763350104.535276 mapread 0x022c47 thru 0x02c39f (0x9759 bytes) 000680 1763350104.559252 MIRROR_RESYNC 4 mirrors 000681 1763350106.027339 trunc from 0x022983 to 0x0274cd (0x4b4b bytes) 000682 1763350106.080500 fallocate 0x025577 thru 0x02dd75 (0x87ff bytes) 000683 1763350106.088379 mapwrite 0x029438 thru 0x0316dd (0x82a6 bytes) 000684 1763350106.133223 punch from 0x022604 to 0x0275c0 (0x4fbd bytes) 000685 1763350106.147126 MIRROR_RESYNC 4 mirrors 000686 1763350107.438091 read 0x03b490 thru 0x03ffff (0x4b70 bytes) 000687 1763350107.446982 mapwrite 0x01047c thru 0x017e01 (0x7986 bytes) 000688 1763350107.591236 fallocate 0x0000af thru 0x0055f4 (0x5546 bytes) 000689 1763350107.598029 read 0x032cfe thru 0x0381e3 (0x54e6 bytes) 000690 1763350107.611181 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe2644bf5 (stale) Mirror 4: 0xe2644bf5 (stale) Mirror 5: 0xe2644bf5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x512f219e Mirror 6: 0x512f219e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1763350108.705408 mapwrite 0x0280ef thru 0x035f46 (0xde58 bytes) 000693 1763350108.886629 fallocate 0x02ca24 thru 0x034a30 (0x800d bytes) 000694 1763350108.893443 trunc from 0x040000 to 0x0317c0 (0xe83f bytes) 000695 1763350108.948603 punch from 0x001c5e to 0x0059c5 (0x3d68 bytes) 000697 1763350108.957610 fallocate 0x008503 thru 0x013899 (0xb397 bytes) 000698 1763350108.964736 trunc from 0x028ad9 to 0x0317c0 (0x8ce8 bytes) 000699 1763350108.974141 read 0x0256b8 thru 0x02727d (0x1bc6 bytes) 000700 1763350108.976321 fallocate 0x00279e thru 0x00404c (0x18af bytes) 000701 1763350108.987296 read 0x015c1f thru 0x019f15 (0x42f7 bytes) 000702 1763350109.013383 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x317c1): Mirror 3: 0x22357bcc (stale) Mirror 4: 0x22357bcc (stale) Mirror 6: 0x195a1cc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317c1): Mirror 3: 0x22357bcc (stale) Mirror 4: 0x22357bcc (stale) Mirror 6: 0x195a1cc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317c1: skipped 000703 1763350109.360500 mapwrite 0x026772 thru 0x02e055 (0x78e4 bytes) 000704 1763350109.431297 punch from 0x03138e to 0x0317c0 (0x433 bytes) 000705 1763350109.439091 trunc from 0x0010cc to 0x0083c2 (0x72f7 bytes) 000706 1763350109.445820 trunc from 0x019f01 to 0x0239f6 (0x9af6 bytes) 000708 1763350109.454644 trunc from 0x0317c1 to 0x002d6a (0x2ea56 bytes) 000709 1763350109.501875 mapread 0x0014b6 thru 0x002d6a (0x18b5 bytes) 000710 1763350109.513012 read 0x001175 thru 0x002ab2 (0x193e bytes) 000711 1763350109.516344 punch from 0x0027f2 to 0x002d6a (0x579 bytes) 000712 1763350109.523465 mapwrite 0x00ab50 thru 0x00bc9a (0x114b bytes) 000714 1763350109.597849 punch from 0x00451e to 0x00bc9a (0x777d bytes) 000716 1763350109.606999 trunc from 0x001537 to 0x0098d1 (0x839b bytes) 000717 1763350109.619140 write 0x02d067 thru 0x03160d (0x45a7 bytes) 000720 1763350109.630289 MIRROR_RESYNC 4 mirrors 000721 1763350110.685862 trunc from 0x001d22 to 0x00394b (0x1c2a bytes) 000722 1763350110.716958 punch from 0x0196c0 to 0x019eb3 (0x7f4 bytes) 000723 1763350110.724195 punch from 0x007ab6 to 0x014292 (0xc7dd bytes) 000724 1763350110.730491 punch from 0x004824 to 0x00fee5 (0xb6c2 bytes) 000725 1763350110.735553 fallocate 0x01d256 thru 0x02acbe (0xda69 bytes) 000726 1763350110.741579 fallocate 0x03386b thru 0x03ee15 (0xb5ab bytes) 000727 1763350110.748232 fallocate 0x02aadf thru 0x02f21d (0x473f bytes) 000728 1763350110.753092 trunc from 0x0183b4 to 0x021c66 (0x98b3 bytes) 000730 1763350110.759530 mapwrite 0x00f8e7 thru 0x01ef2e (0xf648 bytes) 000731 1763350110.831038 mapread 0x00a903 thru 0x00e1c1 (0x38bf bytes) 000732 1763350110.832089 fallocate 0x028a89 thru 0x033a11 (0xaf89 bytes) 000733 1763350110.845471 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3160e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3160e): Mirror 3: 0x8026c5c5 (stale) Mirror 4: 0x8026c5c5 (stale) Mirror 6: 0x8026c5c5 (stale) CRC-32 checksum value for chunk [0, 0x3160e): Mirror 1: 0x5e6cb2ab Mirror 7: 0x5e6cb2ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3160e: skipped 000734 1763350111.720271 fallocate 0x02e4ac thru 0x03197f (0x34d4 bytes) 000735 1763350111.762778 write 0x038cd0 thru 0x03ffff (0x7330 bytes) 000736 1763350111.769873 mapread 0x00ae43 thru 0x00b105 (0x2c3 bytes) 000737 1763350111.781859 trunc from 0x029730 to 0x038dc7 (0xf698 bytes) 000738 1763350111.796031 punch from 0x025b0d to 0x0272f1 (0x17e5 bytes) 000740 1763350111.804066 trunc from 0x040000 to 0x038a90 (0x756f bytes) 000741 1763350111.826920 punch from 0x012e96 to 0x01dce7 (0xae52 bytes) 000743 1763350111.834454 trunc from 0x038a91 to 0x036be1 (0x1eaf bytes) 000744 1763350111.854086 trunc from 0x0212e0 to 0x02b08c (0x9dad bytes) 000745 1763350111.862894 write 0x01849a thru 0x025a6d (0xd5d4 bytes) 000746 1763350111.880907 fallocate 0x0116c4 thru 0x020916 (0xf253 bytes) 000747 1763350111.887210 trunc from 0x0118f9 to 0x017263 (0x596b bytes) 000749 1763350111.894047 trunc from 0x036be2 to 0x036af4 (0xed bytes) 000750 1763350111.912104 read 0x028710 thru 0x02a5d8 (0x1ec9 bytes) 000752 1763350111.927423 MIRROR_RESYNC 5 mirrors 000753 1763350113.359892 read 0x008da5 thru 0x011fc4 (0x9220 bytes) 000754 1763350113.369780 trunc from 0x036af5 to 0x03c9fa (0x5f06 bytes) 000755 1763350113.431371 trunc from 0x020186 to 0x0263fc (0x6277 bytes) 000756 1763350113.436381 punch from 0x002aad to 0x006dbe (0x4312 bytes) 000757 1763350113.441706 trunc from 0x020517 to 0x0248c9 (0x43b3 bytes) 000758 1763350113.446920 mapwrite 0x03a03c thru 0x03a8e6 (0x8ab bytes) 000761 1763350113.476081 punch from 0x03af1b to 0x03c9fa (0x1ae0 bytes) 000762 1763350113.488266 trunc from 0x0011c7 to 0x004249 (0x3083 bytes) 000763 1763350113.501691 mapread 0x0381b5 thru 0x03c9fa (0x4846 bytes) 000764 1763350113.544998 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c9fb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9fb): Mirror 3: 0xbe7121bc (stale) Mirror 4: 0xbe7121bc (stale) Mirror 6: 0xbe7121bc (stale) Mirror 7: 0xbe7121bc (stale) CRC-32 checksum value for chunk [0, 0x3c9fb): Mirror 1: 0x94e5126e Mirror 8: 0x94e5126e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9fb: skipped 000765 1763350114.498425 write 0x005100 thru 0x005889 (0x78a bytes) 000766 1763350114.547084 mapread 0x01a42a thru 0x022135 (0x7d0c bytes) 000767 1763350114.563460 write 0x018be1 thru 0x021605 (0x8a25 bytes) 000768 1763350114.565166 mapread 0x00ef7d thru 0x01ec81 (0xfd05 bytes) 000769 1763350114.575882 fallocate 0x00e92e thru 0x01ac9c (0xc36f bytes) 000770 1763350114.582082 fallocate 0x010f5f thru 0x0132e5 (0x2387 bytes) 000771 1763350114.587921 mapread 0x01c969 thru 0x0273d3 (0xaa6b bytes) 000773 1763350114.600177 fallocate 0x036855 thru 0x03ffff (0x97ab bytes) 000774 1763350114.607546 write 0x009380 thru 0x014f1b (0xbb9c bytes) 000775 1763350114.609767 fallocate 0x02c943 thru 0x02ded8 (0x1596 bytes) 000776 1763350114.616423 read 0x0117b6 thru 0x01dd3a (0xc585 bytes) 000777 1763350114.617892 mapwrite 0x02d0df thru 0x03b296 (0xe1b8 bytes) 000778 1763350114.669246 trunc from 0x0312d4 to 0x03c9fa (0xb727 bytes) 000779 1763350114.680201 read 0x02d69a thru 0x035aad (0x8414 bytes) 000780 1763350114.692242 trunc from 0x03c9fb to 0x02bec1 (0x10b39 bytes) 000781 1763350114.768391 mapwrite 0x009ca9 thru 0x00c3a8 (0x2700 bytes) 000782 1763350114.797619 punch from 0x027777 to 0x02bec1 (0x474b bytes) 000783 1763350114.807744 mapread 0x0263dd thru 0x02bec1 (0x5ae5 bytes) 000786 1763350114.868262 fallocate 0x03b659 thru 0x03ffff (0x49a7 bytes) 000787 1763350114.872993 fallocate 0x006452 thru 0x00c9f3 (0x65a2 bytes) 000788 1763350114.882229 mapwrite 0x03c007 thru 0x03ffff (0x3ff9 bytes) 000789 1763350114.945505 punch from 0x00455d to 0x008cfb (0x479f bytes) 000790 1763350114.955312 mapread 0x02eb3d thru 0x03b7c9 (0xcc8d bytes) 000792 1763350115.109100 mapread 0x019003 thru 0x024f6a (0xbf68 bytes) 000793 1763350115.117842 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8daa63b7 (stale) Mirror 4: 0x8daa63b7 (stale) Mirror 6: 0x8daa63b7 (stale) Mirror 7: 0x8daa63b7 (stale) Mirror 8: 0xe98dd484 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8e51e570 Mirror 9: 0x8e51e570 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1763350116.299831 trunc from 0x00bcca to 0x018bcd (0xcf04 bytes) 000795 1763350116.351572 mapread 0x03cc96 thru 0x03ffff (0x336a bytes) 000796 1763350116.379159 mapwrite 0x02d922 thru 0x038d99 (0xb478 bytes) 000797 1763350116.422116 MIRROR_RESYNC 7 mirrors 000799 1763350118.741872 punch from 0x003d59 to 0x003f54 (0x1fc bytes) 000800 1763350118.780162 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xca425b07 (stale) Mirror 4: 0xca425b07 (stale) Mirror 6: 0xca425b07 (stale) Mirror 7: 0xca425b07 (stale) Mirror 8: 0xca425b07 (stale) Mirror 9: 0xca425b07 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xca425b07 Mirror 10: 0xca425b07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1763350119.818674 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xca425b07 (stale) Mirror 4: 0xca425b07 (stale) Mirror 6: 0xca425b07 (stale) Mirror 7: 0xca425b07 (stale) Mirror 8: 0xca425b07 (stale) Mirror 9: 0xca425b07 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xca425b07 Mirror 10: 0xca425b07 Mirror 11: 0xca425b07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000803 1763350120.722435 MIRROR_RESYNC 9 mirrors 000804 1763350122.727006 fallocate 0x019959 thru 0x028cfe (0xf3a6 bytes) 000805 1763350122.777533 mapread 0x0335e7 thru 0x03bbf4 (0x860e bytes) 000806 1763350122.818024 trunc from 0x040000 to 0x031b9d (0xe462 bytes) 000807 1763350122.849195 punch from 0x025bac to 0x02b299 (0x56ee bytes) 000808 1763350122.854247 mapread 0x02d954 thru 0x031b9d (0x424a bytes) 000809 1763350122.864743 write 0x005666 thru 0x014a28 (0xf3c3 bytes) 000810 1763350122.880057 mapread 0x01d721 thru 0x027c84 (0xa564 bytes) 000811 1763350122.890864 read 0x025ca2 thru 0x02b674 (0x59d3 bytes) 000812 1763350122.892529 fallocate 0x00b603 thru 0x013a46 (0x8444 bytes) 000813 1763350122.898612 trunc from 0x02a4dd to 0x031b9d (0x76c1 bytes) 000814 1763350122.905107 mapread 0x02d377 thru 0x031b9d (0x4827 bytes) 000815 1763350122.921129 fallocate 0x024f5b thru 0x02e679 (0x971f bytes) 000816 1763350122.925938 mapwrite 0x0379a9 thru 0x03e667 (0x6cbf bytes) 000817 1763350122.974208 mapwrite 0x017b63 thru 0x022f45 (0xb3e3 bytes) 000818 1763350123.005034 mapwrite 0x02c3bd thru 0x03723d (0xae81 bytes) 000820 1763350123.044261 trunc from 0x01fa09 to 0x02f256 (0xf84e bytes) 000821 1763350123.051390 trunc from 0x037f21 to 0x03e667 (0x6747 bytes) 000822 1763350123.059007 write 0x02d240 thru 0x02defb (0xcbc bytes) 000823 1763350123.070059 write 0x012a2d thru 0x019bfb (0x71cf bytes) 000824 1763350123.072033 read 0x013e9b thru 0x0179e0 (0x3b46 bytes) 000825 1763350123.072627 trunc from 0x03e668 to 0x02066e (0x1dff9 bytes) 000826 1763350123.094633 punch from 0x00d256 to 0x016b69 (0x9914 bytes) 000827 1763350123.114695 trunc from 0x02066f to 0x039050 (0x189e2 bytes) 000828 1763350123.133469 fallocate 0x011c6a thru 0x018ae0 (0x6e77 bytes) 000829 1763350123.141090 trunc from 0x039051 to 0x026606 (0x12a4a bytes) 000831 1763350123.161185 trunc from 0x003696 to 0x011510 (0xde7b bytes) 000832 1763350123.173864 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26607): Mirror 4: 0x93901edc (stale) Mirror 6: 0x93901edc (stale) Mirror 7: 0x93901edc (stale) Mirror 8: 0x93901edc (stale) Mirror 9: 0x93901edc (stale) Mirror 10: 0x93901edc (stale) Mirror 11: 0x93901edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26607): Mirror 4: 0x93901edc (stale) Mirror 6: 0x93901edc (stale) Mirror 7: 0x93901edc (stale) Mirror 8: 0x93901edc (stale) Mirror 9: 0x93901edc (stale) Mirror 10: 0x93901edc (stale) Mirror 11: 0x93901edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26607: skipped 000833 1763350123.610960 read 0x00d742 thru 0x019111 (0xb9d0 bytes) 000835 1763350123.619972 write 0x0378c3 thru 0x039024 (0x1762 bytes) 000836 1763350123.627633 fallocate 0x00d418 thru 0x011e6b (0x4a54 bytes) 000837 1763350123.638350 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39025): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39025): Mirror 4: 0xb09e77fe (stale) Mirror 6: 0xb09e77fe (stale) Mirror 7: 0xb09e77fe (stale) Mirror 8: 0xb09e77fe (stale) Mirror 9: 0xb09e77fe (stale) Mirror 10: 0xb09e77fe (stale) Mirror 11: 0xb09e77fe (stale) CRC-32 checksum value for chunk [0, 0x39025): Mirror 1: 0xa167a87 Mirror 12: 0xa167a87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39025: skipped 000838 1763350124.754192 mapwrite 0x025971 thru 0x029f12 (0x45a2 bytes) 000840 1763350124.899220 fallocate 0x03dc53 thru 0x03ffff (0x23ad bytes) 000841 1763350124.905601 mapwrite 0x00fc95 thru 0x017131 (0x749d bytes) 000842 1763350124.948308 mapread 0x02b4c0 thru 0x0340ce (0x8c0f bytes) 000843 1763350124.958638 mapwrite 0x011301 thru 0x01ffd6 (0xecd6 bytes) 000844 1763350125.010474 trunc from 0x040000 to 0x02cb00 (0x134ff bytes) 000845 1763350125.047474 mapwrite 0x036119 thru 0x03e9f5 (0x88dd bytes) 000846 1763350125.100903 punch from 0x01405f to 0x017222 (0x31c4 bytes) 000847 1763350125.106868 read 0x032eb8 thru 0x036a0c (0x3b55 bytes) 000848 1763350125.107504 mapread 0x00822c thru 0x013e6a (0xbc3f bytes) 000849 1763350125.108606 trunc from 0x03e9f6 to 0x002136 (0x3c8bf bytes) 000850 1763350125.126558 mapwrite 0x034d2f thru 0x03aa97 (0x5d69 bytes) 000851 1763350125.167833 punch from 0x015262 to 0x0157a4 (0x543 bytes) 000853 1763350125.173597 mapread 0x035e13 thru 0x03aa97 (0x4c85 bytes) 000854 1763350125.175229 write 0x000294 thru 0x0042df (0x404c bytes) 000855 1763350125.182231 trunc from 0x038dfa to 0x03aa97 (0x1c9e bytes) 000856 1763350125.187077 punch from 0x03008b to 0x03aa97 (0xaa0d bytes) 000857 1763350125.192532 punch from 0x030f01 to 0x032fc8 (0x20c8 bytes) 000858 1763350125.200449 fallocate 0x011146 thru 0x013fc8 (0x2e83 bytes) 000859 1763350125.207759 trunc from 0x03aa98 to 0x0075c5 (0x334d2 bytes) 000860 1763350125.226858 trunc from 0x00050a to 0x002ead (0x29a4 bytes) 000861 1763350125.245080 read 0x00456c thru 0x0075c5 (0x305a bytes) 000862 1763350125.251890 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x75c6): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x75c6): Mirror 4: 0xecc86332 (stale) Mirror 6: 0xecc86332 (stale) Mirror 7: 0xecc86332 (stale) Mirror 8: 0xecc86332 (stale) Mirror 9: 0xecc86332 (stale) Mirror 10: 0xecc86332 (stale) Mirror 11: 0xecc86332 (stale) Mirror 12: 0xecc86332 (stale) CRC-32 checksum value for chunk [0, 0x75c6): Mirror 1: 0x7cb1991e Mirror 13: 0x7cb1991e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x75c6: skipped 000863 1763350126.236489 trunc from 0x0075c6 to 0x038bfc (0x31637 bytes) 000864 1763350126.319836 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38bfd): Mirror 4: 0xa5a9eccf (stale) Mirror 6: 0xa5a9eccf (stale) Mirror 7: 0xa5a9eccf (stale) Mirror 8: 0xa5a9eccf (stale) Mirror 10: 0xa5a9eccf (stale) Mirror 11: 0xa5a9eccf (stale) Mirror 12: 0x46f5a0c4 (stale) Mirror 13: 0x76bd058b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bfd): Mirror 4: 0xa5a9eccf (stale) Mirror 6: 0xa5a9eccf (stale) Mirror 7: 0xa5a9eccf (stale) Mirror 8: 0xa5a9eccf (stale) Mirror 10: 0xa5a9eccf (stale) Mirror 11: 0xa5a9eccf (stale) Mirror 12: 0x46f5a0c4 (stale) Mirror 13: 0x76bd058b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bfd: skipped 000865 1763350126.857543 fallocate 0x02ffba thru 0x03e55e (0xe5a5 bytes) 000867 1763350126.864022 punch from 0x012b38 to 0x0177f8 (0x4cc1 bytes) 000868 1763350126.870811 punch from 0x0381fc to 0x03b96a (0x376f bytes) 000870 1763350126.882921 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e55f): Mirror 4: 0x6d8a4a77 (stale) Mirror 6: 0x6d8a4a77 (stale) Mirror 7: 0x6d8a4a77 (stale) Mirror 8: 0x6d8a4a77 (stale) Mirror 10: 0x6d8a4a77 (stale) Mirror 12: 0xfe29e6a0 (stale) Mirror 13: 0xa0b161fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55f): Mirror 4: 0x6d8a4a77 (stale) Mirror 6: 0x6d8a4a77 (stale) Mirror 7: 0x6d8a4a77 (stale) Mirror 8: 0x6d8a4a77 (stale) Mirror 10: 0x6d8a4a77 (stale) Mirror 12: 0xfe29e6a0 (stale) Mirror 13: 0xa0b161fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55f: skipped 000871 1763350127.305300 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e55f): Mirror 4: 0x6d8a4a77 (stale) Mirror 6: 0x6d8a4a77 (stale) Mirror 7: 0x6d8a4a77 (stale) Mirror 8: 0x6d8a4a77 (stale) Mirror 12: 0xfe29e6a0 (stale) Mirror 13: 0xa0b161fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55f): Mirror 4: 0x6d8a4a77 (stale) Mirror 6: 0x6d8a4a77 (stale) Mirror 7: 0x6d8a4a77 (stale) Mirror 8: 0x6d8a4a77 (stale) Mirror 12: 0xfe29e6a0 (stale) Mirror 13: 0xa0b161fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55f: skipped 000872 1763350127.789921 write 0x004729 thru 0x006d7f (0x2657 bytes) 000873 1763350127.808907 mapread 0x019ccc thru 0x0213ce (0x7703 bytes) 000874 1763350127.819152 trunc from 0x027288 to 0x02f38c (0x8105 bytes) 000875 1763350127.824490 trunc from 0x03e55f to 0x030a18 (0xdb46 bytes) 000876 1763350127.844116 trunc from 0x030a19 to 0x02c1e4 (0x4834 bytes) 000877 1763350127.863615 write 0x02de23 thru 0x032517 (0x46f5 bytes) 000878 1763350127.865689 mapwrite 0x03dd2f thru 0x03ffff (0x22d1 bytes) 000879 1763350127.923331 read 0x037290 thru 0x038e9b (0x1c0c bytes) 000880 1763350127.924012 trunc from 0x02571b to 0x0351ce (0xfab4 bytes) 000882 1763350127.935146 trunc from 0x005caf to 0x00a251 (0x45a3 bytes) 000883 1763350127.943972 read 0x035ca9 thru 0x03e4de (0x8836 bytes) 000884 1763350127.974852 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xca425b07 (stale) Mirror 6: 0xca425b07 (stale) Mirror 7: 0xca425b07 (stale) Mirror 8: 0xca425b07 (stale) Mirror 12: 0x9d716666 (stale) Mirror 13: 0x8d713b5a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69789957 Mirror 14: 0x69789957 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1763350129.050959 read 0x00c7b3 thru 0x01569f (0x8eed bytes) 000886 1763350129.065797 trunc from 0x015cf9 to 0x0257a4 (0xfaac bytes) 000887 1763350129.121499 mapwrite 0x00cb60 thru 0x00cc71 (0x112 bytes) 000888 1763350129.152841 mapread 0x02fd7c thru 0x03cd03 (0xcf88 bytes) 000889 1763350129.191264 read 0x00904a thru 0x01858d (0xf544 bytes) 000890 1763350129.202723 trunc from 0x040000 to 0x02a255 (0x15daa bytes) 000891 1763350129.233590 read 0x00d158 thru 0x015ce7 (0x8b90 bytes) 000892 1763350129.234500 trunc from 0x02a256 to 0x02ccf6 (0x2aa1 bytes) 000893 1763350129.262727 fallocate 0x00ff71 thru 0x019df0 (0x9e80 bytes) 000894 1763350129.272023 read 0x01c676 thru 0x027180 (0xab0b bytes) 000895 1763350129.272639 punch from 0x012b90 to 0x014897 (0x1d08 bytes) 000897 1763350129.282510 mapread 0x028f14 thru 0x02b424 (0x2511 bytes) 000898 1763350129.294805 write 0x0242f7 thru 0x028fb9 (0x4cc3 bytes) 000899 1763350129.295911 mapread 0x01386c thru 0x01dbae (0xa343 bytes) 000900 1763350129.296872 mapwrite 0x000cc6 thru 0x010962 (0xfc9d bytes) 000901 1763350129.332579 mapwrite 0x0315cc thru 0x033ee7 (0x291c bytes) 000902 1763350129.403191 punch from 0x02a0a6 to 0x031a65 (0x79c0 bytes) 000903 1763350129.430701 MIRROR_RESYNC 8 mirrors 000904 1763350131.263873 trunc from 0x033ee8 to 0x01c017 (0x17ed0 bytes) 000906 1763350131.302032 write 0x0347aa thru 0x03ffff (0xb856 bytes) 000907 1763350131.304783 write 0x03c1e8 thru 0x03ffff (0x3e18 bytes) 000908 1763350131.307639 mapread 0x007b6e thru 0x010ff2 (0x9485 bytes) 000910 1763350131.333800 fallocate 0x01bbf3 thru 0x02a803 (0xec11 bytes) 000911 1763350131.341930 trunc from 0x03c9bb to 0x03ffff (0x3645 bytes) 000912 1763350131.358332 fallocate 0x0258d0 thru 0x026a9d (0x11ce bytes) 000913 1763350131.372836 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xabd38af6 (stale) Mirror 6: 0xabd38af6 (stale) Mirror 7: 0xabd38af6 (stale) Mirror 8: 0xabd38af6 (stale) Mirror 12: 0xabd38af6 (stale) Mirror 13: 0xabd38af6 (stale) Mirror 14: 0xabd38af6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc2e893da Mirror 15: 0xc2e893da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000914 1763350132.556704 read 0x016daf thru 0x020e0c (0xa05e bytes) 000915 1763350132.573077 write 0x01d928 thru 0x01e7ac (0xe85 bytes) 000916 1763350132.656737 mapwrite 0x034074 thru 0x039961 (0x58ee bytes) 000917 1763350132.684103 punch from 0x0191bd to 0x01a47a (0x12be bytes) 000918 1763350132.690303 write 0x03650c thru 0x03ffff (0x9af4 bytes) 000919 1763350132.691876 mapwrite 0x032371 thru 0x03dae4 (0xb774 bytes) 000921 1763350132.719676 fallocate 0x0385fd thru 0x03bbe5 (0x35e9 bytes) 000922 1763350132.732339 punch from 0x005763 to 0x013b25 (0xe3c3 bytes) 000923 1763350132.742401 trunc from 0x00befc to 0x011c11 (0x5d16 bytes) 000924 1763350132.752441 write 0x010d4e thru 0x012abe (0x1d71 bytes) 000925 1763350132.774132 read 0x02faea thru 0x03c447 (0xc95e bytes) 000926 1763350132.777150 read 0x025e34 thru 0x030c87 (0xae54 bytes) 000927 1763350132.796439 read 0x035215 thru 0x0377b2 (0x259e bytes) 000929 1763350132.800431 read 0x007773 thru 0x01658b (0xee19 bytes) 000930 1763350132.801153 punch from 0x021b79 to 0x02b498 (0x9920 bytes) 000931 1763350132.817225 punch from 0x014ff0 to 0x0173f4 (0x2405 bytes) 000932 1763350132.826349 fallocate 0x03fc40 thru 0x03ffff (0x3c0 bytes) 000933 1763350132.835231 trunc from 0x040000 to 0x000d7f (0x3f280 bytes) 000934 1763350132.881528 read 0x0002e0 thru 0x000d7f (0xaa0 bytes) 000935 1763350132.883704 read 0x000729 thru 0x000d7f (0x657 bytes) 000936 1763350132.885528 read 0x00098f thru 0x000d7f (0x3f1 bytes) 000938 1763350132.887310 trunc from 0x000d80 to 0x031075 (0x302f6 bytes) 000939 1763350132.913336 write 0x021fb5 thru 0x02e76e (0xc7ba bytes) 000940 1763350132.924831 fallocate 0x00a824 thru 0x018472 (0xdc4f bytes) 000941 1763350132.929850 read 0x010d2a thru 0x0165f9 (0x58d0 bytes) 000943 1763350132.930965 fallocate 0x03808a thru 0x03f8fd (0x7874 bytes) 000944 1763350132.937114 fallocate 0x02a366 thru 0x02c388 (0x2023 bytes) 000945 1763350132.945395 mapwrite 0x01a6c5 thru 0x0219cf (0x730b bytes) 000946 1763350132.964863 trunc from 0x016a82 to 0x026887 (0xfe06 bytes) 000947 1763350132.972465 trunc from 0x03f8fe to 0x02cd7f (0x12b7e bytes) 000948 1763350132.991783 write 0x032192 thru 0x03f130 (0xcf9f bytes) 000949 1763350132.993538 punch from 0x03a30c to 0x03f130 (0x4e25 bytes) 000950 1763350133.020149 fallocate 0x028308 thru 0x0287be (0x4b7 bytes) 000951 1763350133.025258 write 0x003ab9 thru 0x012d87 (0xf2cf bytes) 000952 1763350133.030153 write 0x01e965 thru 0x02d3cf (0xea6b bytes) 000953 1763350133.049233 trunc from 0x03f131 to 0x006f31 (0x381ff bytes) 000954 1763350133.079692 punch from 0x002376 to 0x006f31 (0x4bbc bytes) 000955 1763350133.090562 trunc from 0x006f32 to 0x03fd2d (0x38dfc bytes) 000956 1763350133.107910 read 0x0206f1 thru 0x025ca1 (0x55b1 bytes) 000958 1763350133.122852 trunc from 0x039e2e to 0x03b2a1 (0x1474 bytes) 000959 1763350133.130471 read 0x01ec9d thru 0x02eb63 (0xfec7 bytes) 000960 1763350133.132418 trunc from 0x022ac1 to 0x02e61b (0xbb5b bytes) 000961 1763350133.139132 mapread 0x016b99 thru 0x020b35 (0x9f9d bytes) 000962 1763350133.141227 trunc from 0x03fd2e to 0x03de15 (0x1f18 bytes) 000963 1763350133.167985 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3de16): Mirror 4: 0xd84416bf (stale) Mirror 6: 0xd84416bf (stale) Mirror 7: 0xd84416bf (stale) Mirror 8: 0xd84416bf (stale) Mirror 12: 0xd84416bf (stale) Mirror 13: 0xd84416bf (stale) Mirror 15: 0x672a4874 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de16): Mirror 4: 0xd84416bf (stale) Mirror 6: 0xd84416bf (stale) Mirror 7: 0xd84416bf (stale) Mirror 8: 0xd84416bf (stale) Mirror 12: 0xd84416bf (stale) Mirror 13: 0xd84416bf (stale) Mirror 15: 0x672a4874 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de16: skipped 000964 1763350133.696046 trunc from 0x03de16 to 0x011ef1 (0x2bf24 bytes) 000965 1763350133.725637 write 0x030f8c thru 0x03c896 (0xb90b bytes) 000966 1763350133.727515 trunc from 0x03c897 to 0x039759 (0x313d bytes) 000967 1763350133.748515 mapwrite 0x016f1c thru 0x0242a2 (0xd387 bytes) 000968 1763350133.810424 punch from 0x035133 to 0x036328 (0x11f6 bytes) 000969 1763350133.817131 mapread 0x0086bb thru 0x00f36a (0x6cb0 bytes) 000970 1763350133.829930 fallocate 0x0279f8 thru 0x034594 (0xcb9d bytes) 000971 1763350133.835350 fallocate 0x01c639 thru 0x02c17e (0xfb46 bytes) 000972 1763350133.841302 punch from 0x01ef6e to 0x02ba9e (0xcb31 bytes) 000973 1763350133.848386 write 0x0394b3 thru 0x03ffff (0x6b4d bytes) 000974 1763350133.849826 fallocate 0x01c2d3 thru 0x01dd20 (0x1a4e bytes) 000975 1763350133.856566 mapread 0x014da0 thru 0x01640b (0x166c bytes) 000976 1763350133.857500 fallocate 0x0115a0 thru 0x012590 (0xff1 bytes) 000977 1763350133.873480 trunc from 0x031d24 to 0x03d377 (0xb654 bytes) 000978 1763350133.893394 read 0x004ead thru 0x010a54 (0xbba8 bytes) 000979 1763350133.899274 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xabd38af6 (stale) Mirror 6: 0xabd38af6 (stale) Mirror 7: 0xabd38af6 (stale) Mirror 8: 0xabd38af6 (stale) Mirror 12: 0xabd38af6 (stale) Mirror 13: 0xabd38af6 (stale) Mirror 15: 0xc2e893da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x32b98206 Mirror 16: 0x32b98206 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000981 1763350134.969571 mapread 0x01a3de thru 0x022f60 (0x8b83 bytes) 000982 1763350135.003045 punch from 0x028129 to 0x02be5c (0x3d34 bytes) 000984 1763350135.076507 fallocate 0x027d8b thru 0x030173 (0x83e9 bytes) 000985 1763350135.083414 trunc from 0x01d889 to 0x027765 (0x9edd bytes) 000986 1763350135.089247 mapread 0x02423c thru 0x02a6d0 (0x6495 bytes) 000987 1763350135.097382 write 0x01208b thru 0x01857e (0x64f4 bytes) 000988 1763350135.113011 mapwrite 0x0102ae thru 0x01782c (0x757f bytes) 000989 1763350135.144858 trunc from 0x040000 to 0x01bfa0 (0x2405f bytes) 000990 1763350135.165184 punch from 0x0001b1 to 0x003d72 (0x3bc2 bytes) 000991 1763350135.170162 mapwrite 0x00bacd thru 0x012674 (0x6ba8 bytes) 000994 1763350135.217406 punch from 0x004f23 to 0x00cf8a (0x8068 bytes) 000995 1763350135.223975 trunc from 0x001954 to 0x00d398 (0xba45 bytes) 000996 1763350135.228722 punch from 0x009dfe to 0x00e033 (0x4236 bytes) 000997 1763350135.234113 trunc from 0x01bfa1 to 0x00f813 (0xc78d bytes) 000998 1763350135.254089 trunc from 0x00d0fb to 0x00f813 (0x2719 bytes) 000999 1763350135.277733 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf814): Mirror 4: 0x94a5cb (stale) Mirror 6: 0x94a5cb (stale) Mirror 7: 0x94a5cb (stale) Mirror 8: 0x94a5cb (stale) Mirror 12: 0x94a5cb (stale) Mirror 13: 0x94a5cb (stale) Mirror 15: 0x94a5cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf814): Mirror 4: 0x94a5cb (stale) Mirror 6: 0x94a5cb (stale) Mirror 7: 0x94a5cb (stale) Mirror 8: 0x94a5cb (stale) Mirror 12: 0x94a5cb (stale) Mirror 13: 0x94a5cb (stale) Mirror 15: 0x94a5cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf814: skipped All operations completed A-OK! PASS 70a (70s) == sanity-flr test 71: check mirror extend parent fid ==== 22:29:00 (1763350140) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x1e7:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 200a: stress test ===================== 22:29:09 (1763350149) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6078976 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4039328 .. failed Extending file size to 6825184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8790624 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9000704 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9128768 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59095 59097 59098 59099 59101 done 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) 192.168.201.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:30:25 (1763350225) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4950336 .. Extending file size to 4475424 .. Extending file size to 6641600 .. Extending file size to 6022720 .. Extending file size to 6601728 .. Extending file size to 7093152 .. Extending file size to 6654528 .. done Extending file size to 9028064 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8613664 .. Extending file size to 9004032 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9065888 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9168672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9380384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61312 61313 61314 61317 61320 done 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) 192.168.201.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:31:41 (1763350301) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2340 1268964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23148 3583872 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14952 3592068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38100 7175940 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2340 1268964 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23148 3583872 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14952 3592068 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38100 7175940 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0938989 s, 22.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190268 s, 11.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xaa8738b0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:31:59 (1763350319) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10537 s, 19.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 22:32:06 (1763350326) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:32:11 (1763350331) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0740945 s, 28.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x264:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:32:18 (1763350338) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0891175 s, 23.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:32:24 (1763350344) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920539 s, 22.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:32:55 (1763350375) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836289 s, 12.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x266:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:33:40 (1763350420) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0908942 s, 23.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:34:31 (1763350471) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg106-server: Pool lustre.test_204f created Adding targets to pool oleg106-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg106-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145348 s, 14.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg106-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg106-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg106-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:35:37 (1763350537) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:35:44 (1763350544) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:35:49 (1763350549) oleg106-server: Pool lustre.test_206 created Adding targets to pool oleg106-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg106-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg106-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg106-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg106-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:36:15 (1763350575) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659436 s, 15.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243884 s, 43.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf991dd6 Mirror 3: 0xaf991dd6 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:36:24 (1763350584) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:36:27 (1763350587) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:36:30 (1763350590) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0992918 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xce0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x14b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x14b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x14b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e30000 Read this (4 bytes): BBBB 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:37:05 (1763350625) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0156622 s, 523 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:37:12 (1763350632) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106983 s, 9.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 22:37:15.000000000 -0500 Modify: 2025-11-16 22:37:15.000000000 -0500 Change: 2025-11-16 22:37:17.000000000 -0500 Birth: 2025-11-16 22:37:16.000000000 -0500 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:37:23 (1763350643) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542828 s, 19.3 MB/s oleg106-server: debugfs 1.47.3-wc1 (8-Aug-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg106-server: debugfs 1.47.3-wc1 (8-Aug-2025) oleg106-server: O/280000401/d25/b9: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:37:46 (1763350666) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.233308 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0274589 s, 149 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 22:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 22:37 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:37:54 (1763350674) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:38:00 (1763350680) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1514 sec =========== 22:38:08 (1763350688) === sanity-flr: start cleanup 22:38:09 (1763350689) === === sanity-flr: finish cleanup 22:38:17 (1763350697) ===