-----============= acceptance-small: sanity-flr ============----- Wed Dec 3 01:52:03 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg140-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:52:12 (1764744732) === oleg140-client.virtnet: executing check_config_client /mnt/lustre oleg140-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg140-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99ce03974000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99ce03974000.idle_timeout=debug disable quota as required oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:52:28 (1764744748) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8192] == sanity-flr test 0a: lfs mirror create with -N option == 01:52:29 (1764744749) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:52:35 (1764744755) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:52:37 (1764744757) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:52:38 (1764744758) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:52:49 (1764744769) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:52:50 (1764744770) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 01:52:52 (1764744772) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99ce03974000.stats=clear osc.lustre-OST0001-osc-ffff99ce03974000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:52:55 (1764744775) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:53:00 (1764744780) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:53:03 (1764744783) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:53:05 (1764744785) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 01:53:08 (1764744788) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:53:11 (1764744791) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:53:13 (1764744793) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:53:16 (1764744796) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:53:19 (1764744799) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:53:21 (1764744801) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.215774 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252737 s, 41.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:53:24 (1764744804) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174289 s, 60.2 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:53:26 (1764744806) Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:53:55 (1764744835) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.232417 s, 49.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:54:06 (1764744846) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:54:31 (1764744871) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.763242 s, 68.7 MB/s -rw-r--r-- 1 root root 52428800 Dec 3 01:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.177s user 0m0.001s sys 0m0.141s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.204s user 0m0.001s sys 0m0.158s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.201s user 0m0.001s sys 0m0.148s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg140-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ce03974000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:54:58 (1764744898) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:54:59 (1764744899) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:55:00 (1764744900) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:55:01 (1764744901) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 01:55:03 (1764744903) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.163666 s, 76.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:55:07 (1764744907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109952 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140066 s, 74.9 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:55:09 (1764744909) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:55:12 (1764744912) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000614219 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000535865 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000702215 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000551901 s, 1.9 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 01:55:15 (1764744915) create test files with size 15729926 .. 0c0365a1cf13ba2c6c26c52220d11179 - 4f713a65e750b312bc6d45ba59c64c9c - 384c82c0c9359d1cc2024dcdb50bf9a2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 01:55:21 (1764744921) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:55:25 (1764744925) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:55:27 (1764744927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254121 s, 41.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111785 s, 93.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 01:55:30 (1764744930) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0341248 s, 61.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.109813 s, 38.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0424213 s, 49.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 01:55:35 (1764744935) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:55:36 (1764744936) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:55:37 (1764744937) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124564 s, 84.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 01:55:39 (1764744939) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0911664 s, 34.5 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:55:42 (1764744942) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:55:45 (1764744945) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.101685 s, 103 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:56:03 (1764744963) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff99ce29114800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.094215 s, 111 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff99ce0562c000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:56:08 (1764744968) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:56:19 (1764744979) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:56:30 (1764744990) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:56:31 (1764744991) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0757536 s, 41.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0528747 s, 59.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdc303091 (stale) Mirror 2: 0xdc303091 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: cd2387f557fd4f75147ef6d59504a6e1 - mirror 1 checksum: 80e9179daadbc5b83e8e9676b7ed6b80 - mirror 2 checksum: cd2387f557fd4f75147ef6d59504a6e1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 01:56:33 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 01:56:33 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cd2387f557fd4f75147ef6d59504a6e1 - mirror 1 checksum: 80e9179daadbc5b83e8e9676b7ed6b80 - mirror 2 checksum: 80e9179daadbc5b83e8e9676b7ed6b80 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 01:56:33 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 01:56:33 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:56:34 (1764744994) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105325 s, 99.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:56:36 (1764744996) Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 01:56:44 (1764745004) shut down facet: mds1 facet_host: oleg140-server facet_failover_host: oleg140-server Failover mds1 to oleg140-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 01:56:58 (1764745018) targets are mounted 01:56:58 (1764745018) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00983498 s, 107 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:57:06 (1764745026) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00941228 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00554159 s, 3.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:57:09 (1764745029) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00890722 s, 2.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0031268 s, 6.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:57:11 (1764745031) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00226963 s, 7.2 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:57:14 (1764745034) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00880067 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:57:16 (1764745036) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00196094 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00828759 s, 494 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:57:18 (1764745038) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:57:24 (1764745044) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:57:43 (1764745063) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:58:00 (1764745080) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:58:18 (1764745098) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 01:58:20 (1764745100) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55867 mirror split pid 55868 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55867 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55868 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 01:59:22 (1764745162) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5163 skipping zero length punch hole 000002 1764745163.495110 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x1d02b 000003 1764745163.574526 trunc from 00000000 to 0x01d02a (0x1d02b bytes) fallocating to largest ever: 0x22c78 000004 1764745163.588241 fallocate 0x02281c thru 0x022c77 (0x45c bytes) 000005 1764745163.589602 trunc from 0x01f187 to 0x022c77 (0x3af1 bytes) 000006 1764745163.590800 mapwrite 0x03a14a thru 0x03ec49 (0x4b00 bytes) 000007 1764745163.604826 punch from 0x000b09 to 0x0055df (0x4ad7 bytes) 000008 1764745163.606409 write 0x03e81b thru 0x03ffff (0x17e5 bytes) 000009 1764745163.606971 write 0x037dc1 thru 0x03ffff (0x823f bytes) 000010 1764745163.607734 mapwrite 0x01f805 thru 0x022b8d (0x3389 bytes) 000011 1764745163.618902 trunc from 0x040000 to 0x004cfa (0x3b305 bytes) 000012 1764745163.623364 mapread 0x000dd2 thru 0x004cfa (0x3f29 bytes) 000013 1764745163.626135 trunc from 0x004cfb to 0x01bc14 (0x16f1a bytes) 000014 1764745163.629991 mapwrite 0x02da93 thru 0x03120d (0x377b bytes) 000015 1764745163.640992 punch from 0x0227f4 to 0x02bb69 (0x9376 bytes) zero_range to largest ever: 0x3120e 000017 1764745163.642719 trunc from 0x02331a to 0x03120d (0xdef4 bytes) 000018 1764745163.644440 write 0x031bb4 thru 0x03d19d (0xb5ea bytes) fallocating to largest ever: 0x3e768 000019 1764745163.647885 fallocate 0x038da2 thru 0x03e767 (0x59c6 bytes) 000020 1764745163.649749 fallocate 0x026755 thru 0x034acd (0xe379 bytes) 000021 1764745163.651435 write 0x036fcf thru 0x038445 (0x1477 bytes) 000022 1764745163.652018 mapread 0x0328da thru 0x0343c9 (0x1af0 bytes) 000023 1764745163.652538 write 0x03edaa thru 0x03ffff (0x1256 bytes) 000024 1764745163.654678 write 0x03243a thru 0x03d9c5 (0xb58c bytes) fallocating to largest ever: 0x3fdd9 000025 1764745163.655700 fallocate 0x033d18 thru 0x03fdd8 (0xc0c1 bytes) 000026 1764745163.658236 trunc from 0x040000 to 0x0137ea (0x2c815 bytes) 000027 1764745163.663277 trunc from 0x0137eb to 0x0061c5 (0xd625 bytes) 000028 1764745163.668117 punch from 0x002699 to 0x0031db (0xb43 bytes) 000029 1764745163.669594 mapwrite 0x03c74f thru 0x03fa9a (0x334c bytes) 000030 1764745163.682868 punch from 0x007571 to 0x015582 (0xe012 bytes) 000031 1764745163.684849 fallocate 0x0144aa thru 0x01e8d3 (0xa42a bytes) 000032 1764745163.686579 fallocate 0x023e22 thru 0x0270bb (0x329a bytes) 000033 1764745163.688340 fallocate 0x0054d1 thru 0x012dc2 (0xd8f2 bytes) 000034 1764745163.690463 mapread 0x01e58f thru 0x024220 (0x5c92 bytes) 000035 1764745163.700117 write 0x0162e9 thru 0x019f77 (0x3c8f bytes) 000036 1764745163.702435 mapwrite 0x01766d thru 0x01f3c9 (0x7d5d bytes) 000037 1764745163.710473 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000039 1764745163.749634 write 0x02ef80 thru 0x030747 (0x17c8 bytes) 000040 1764745163.753603 write 0x01038f thru 0x014f0c (0x4b7e bytes) 000042 1764745163.757639 fallocate 0x014650 thru 0x022006 (0xd9b7 bytes) 000043 1764745163.759374 mapread 0x016e81 thru 0x02615f (0xf2df bytes) 000044 1764745163.771882 mapwrite 0x016dce thru 0x01b152 (0x4385 bytes) 000047 1764745163.778930 read 0x01eb9a thru 0x021e6b (0x32d2 bytes) 000048 1764745163.779318 trunc from 0x008225 to 0x009f39 (0x1d15 bytes) 000049 1764745163.780983 mapwrite 0x01b09b thru 0x022f7a (0x7ee0 bytes) 000050 1764745163.786973 write 0x01a94a thru 0x02895e (0xe015 bytes) 000051 1764745163.789045 mapread 0x01f5d9 thru 0x02d8f7 (0xe31f bytes) 000052 1764745163.791425 fallocate 0x01c79d thru 0x021c42 (0x54a6 bytes) 000053 1764745163.792975 mapwrite 0x029ef5 thru 0x02b53c (0x1648 bytes) 000056 1764745163.798665 read 0x0226df thru 0x023c40 (0x1562 bytes) 000057 1764745163.799010 punch from 0x008197 to 0x00f560 (0x73ca bytes) 000058 1764745163.800446 trunc from 0x01025d to 0x015898 (0x563c bytes) 000059 1764745163.802129 fallocate 0x0080f8 thru 0x00ad99 (0x2ca2 bytes) 000060 1764745163.803682 read 0x02200d thru 0x0242cd (0x22c1 bytes) 000061 1764745163.803988 fallocate 0x02db79 thru 0x03c1ba (0xe642 bytes) 000062 1764745163.805515 read 0x02bb9f thru 0x038610 (0xca72 bytes) fallocating to largest ever: 0x40000 000063 1764745163.808944 fallocate 0x036d6d thru 0x03ffff (0x9293 bytes) 000064 1764745163.810462 mapread 0x026936 thru 0x02b492 (0x4b5d bytes) 000065 1764745163.810968 trunc from 0x024c16 to 0x02829d (0x3688 bytes) 000066 1764745163.812508 mapwrite 0x03f84a thru 0x03ffff (0x7b6 bytes) 000068 1764745163.816658 punch from 0x006808 to 0x00819b (0x1994 bytes) 000069 1764745163.818272 read 0x031ffc thru 0x037a27 (0x5a2c bytes) 000070 1764745163.818658 trunc from 0x0248f9 to 0x02f4ba (0xabc2 bytes) 000071 1764745163.821567 MIRROR_EXTEND to 2 mirrors 000073 1764745163.997488 write 0x028573 thru 0x0381f0 (0xfc7e bytes) 000074 1764745164.013208 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000075 1764745164.057912 mapwrite 0x037bcf thru 0x03ffff (0x8431 bytes) 000076 1764745164.070064 punch from 0x00fd8d to 0x014346 (0x45ba bytes) 000077 1764745164.071581 write 0x030b26 thru 0x036aea (0x5fc5 bytes) 000078 1764745164.075654 trunc from 0x02c03b to 0x03a36d (0xe333 bytes) 000079 1764745164.080361 MIRROR_EXTEND to 2 mirrors 000080 1764745164.246704 trunc from 0x03e365 to 0x03ffff (0x1c9b bytes) 000081 1764745164.255606 mapwrite 0x03f2c5 thru 0x03ffff (0xd3b bytes) 000082 1764745164.260696 mapwrite 0x00aa18 thru 0x0104a3 (0x5a8c bytes) 000084 1764745164.271190 read 0x02a5b1 thru 0x033acd (0x951d bytes) 000086 1764745164.273838 mapread 0x026b85 thru 0x0301e8 (0x9664 bytes) 000087 1764745164.277821 write 0x0004be thru 0x00d899 (0xd3dc bytes) 000088 1764745164.280832 fallocate 0x00c991 thru 0x01786a (0xaeda bytes) 000089 1764745164.282470 mapread 0x02afae thru 0x03a364 (0xf3b7 bytes) 000090 1764745164.288152 mapread 0x03f837 thru 0x03ffff (0x7c9 bytes) 000091 1764745164.289730 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9de66669 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5e644c30 Mirror 3: 0x5e644c30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000093 1764745164.509007 mapwrite 0x02357f thru 0x03355c (0xffde bytes) 000096 1764745164.555856 trunc from 0x023285 to 0x026106 (0x2e82 bytes) 000097 1764745164.557494 punch from 0x012285 to 0x021474 (0xf1f0 bytes) 000098 1764745164.559153 trunc from 0x040000 to 0x016cc1 (0x2933e bytes) 000099 1764745164.570845 write 0x027c46 thru 0x037893 (0xfc4e bytes) 000100 1764745164.572383 write 0x03e4e3 thru 0x03ffff (0x1b1d bytes) 000101 1764745164.574576 MIRROR_RESYNC 3 mirrors 000102 1764745164.804834 trunc from 0x036503 to 0x037dae (0x18ac bytes) 000103 1764745164.813544 trunc from 0x01bfae to 0x01d4a6 (0x14f9 bytes) 000104 1764745164.815269 fallocate 0x0228d0 thru 0x026125 (0x3856 bytes) 000105 1764745164.817276 punch from 0x019173 to 0x01f843 (0x66d1 bytes) 000106 1764745164.819053 read 0x0330cb thru 0x03fc54 (0xcb8a bytes) 000107 1764745164.822401 write 0x03c67b thru 0x03ffff (0x3985 bytes) 000108 1764745164.825031 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf3993326 (stale) Mirror 3: 0xf3993326 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x252cd3e4 Mirror 4: 0x252cd3e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000109 1764745165.047889 trunc from 0x0065ca to 0x00a503 (0x3f3a bytes) 000110 1764745165.057537 write 0x03a9db thru 0x03ffff (0x5625 bytes) 000111 1764745165.060671 fallocate 0x02b5ca thru 0x0335ea (0x8021 bytes) 000112 1764745165.063579 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf3993326 (stale) Mirror 3: 0xf3993326 (stale) Mirror 4: 0x252cd3e4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf78c987c Mirror 5: 0xf78c987c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1764745165.311550 mapread 0x019755 thru 0x0283a4 (0xec50 bytes) 000115 1764745165.329362 mapread 0x02e97d thru 0x03bca2 (0xd326 bytes) 000116 1764745165.343544 trunc from 0x003ab4 to 0x00cd02 (0x924f bytes) 000117 1764745165.354257 trunc from 0x040000 to 0x033fab (0xc054 bytes) 000118 1764745165.361286 write 0x02394b thru 0x026bed (0x32a3 bytes) 000121 1764745165.365530 write 0x020dee thru 0x02ad7a (0x9f8d bytes) 000122 1764745165.369218 trunc from 0x032731 to 0x033fab (0x187b bytes) 000123 1764745165.370579 trunc from 0x033fac to 0x0051df (0x2edcc bytes) 000124 1764745165.374470 trunc from 0x0051e0 to 0x01e6ba (0x194db bytes) 000125 1764745165.378119 read 0x018c15 thru 0x01e6ba (0x5aa6 bytes) 000126 1764745165.383080 write 0x023b8e thru 0x027332 (0x37a5 bytes) 000127 1764745165.383994 trunc from 0x00aa3c to 0x01a041 (0xf606 bytes) 000128 1764745165.385730 mapwrite 0x02a41b thru 0x02b57e (0x1164 bytes) 000129 1764745165.395413 punch from 0x00a711 to 0x00fb08 (0x53f8 bytes) 000130 1764745165.396953 read 0x023709 thru 0x0273c2 (0x3cba bytes) 000131 1764745165.397322 fallocate 0x0141b3 thru 0x01cd1f (0x8b6d bytes) 000132 1764745165.398994 mapread 0x00e293 thru 0x0132c5 (0x5033 bytes) 000133 1764745165.402352 mapwrite 0x00599d thru 0x0146a0 (0xed04 bytes) 000134 1764745165.410199 fallocate 0x00154b thru 0x0081b1 (0x6c67 bytes) 000135 1764745165.412004 trunc from 0x02b568 to 0x02b57e (0x17 bytes) 000136 1764745165.413682 write 0x0029e0 thru 0x0110b0 (0xe6d1 bytes) 000137 1764745165.415086 write 0x02c887 thru 0x0311fc (0x4976 bytes) 000138 1764745165.416075 mapread 0x004699 thru 0x0083d9 (0x3d41 bytes) 000140 1764745165.416943 trunc from 0x0311fd to 0x003e74 (0x2d388 bytes) 000141 1764745165.421708 write 0x01fb6d thru 0x0222b2 (0x2746 bytes) 000142 1764745165.422513 mapread 0x01d5ad thru 0x020b5d (0x35b1 bytes) 000143 1764745165.425318 read 0x00f03e thru 0x015063 (0x6026 bytes) 000144 1764745165.427678 read 0x004c31 thru 0x00da8e (0x8e5e bytes) 000145 1764745165.428158 trunc from 0x0222b3 to 0x003342 (0x1ef70 bytes) 000146 1764745165.432418 trunc from 0x003343 to 0x023aac (0x2076a bytes) 000147 1764745165.436298 trunc from 0x003f64 to 0x011625 (0xd6c2 bytes) 000148 1764745165.439392 write 0x03d19b thru 0x03eb3a (0x19a0 bytes) 000151 1764745165.441112 fallocate 0x0062a5 thru 0x00e964 (0x86c0 bytes) 000152 1764745165.442812 trunc from 0x03eb3b to 0x004ebf (0x39c7b bytes) 000153 1764745165.446630 read 0x000745 thru 0x004ebf (0x477b bytes) 000154 1764745165.448387 mapwrite 0x037880 thru 0x03bab0 (0x4231 bytes) 000155 1764745165.459612 mapread 0x03a35c thru 0x03bab0 (0x1755 bytes) 000157 1764745165.460556 mapread 0x02952c thru 0x029bf4 (0x6c9 bytes) 000158 1764745165.462207 MIRROR_RESYNC 5 mirrors 000159 1764745165.667230 fallocate 0x001494 thru 0x001be8 (0x755 bytes) 000160 1764745165.676443 trunc from 0x03bab1 to 0x001452 (0x3a65e bytes) 000161 1764745165.686021 fallocate 0x008c8e thru 0x00a640 (0x19b3 bytes) 000162 1764745165.687800 trunc from 0x00a641 to 0x002ede (0x7762 bytes) 000163 1764745165.691458 write 0x018259 thru 0x01c6af (0x4457 bytes) 000164 1764745165.692336 read 0x01432a thru 0x01c6af (0x8386 bytes) 000166 1764745165.695522 MIRROR_RESYNC 5 mirrors 000167 1764745165.893980 read 0x006a87 thru 0x01617c (0xf6f6 bytes) 000168 1764745165.896998 mapread 0x004fb0 thru 0x00893d (0x398e bytes) 000169 1764745165.900133 fallocate 0x008f31 thru 0x013e86 (0xaf56 bytes) 000170 1764745165.910345 trunc from 0x0034bf to 0x00abd6 (0x7718 bytes) 000171 1764745165.911973 write 0x0030ee thru 0x012d95 (0xfca8 bytes) 000173 1764745165.916466 read 0x011c8a thru 0x016c3a (0x4fb1 bytes) 000174 1764745165.918909 mapread 0x00bf9a thru 0x01a816 (0xe87d bytes) 000175 1764745165.919644 read 0x01b3cc thru 0x01c6af (0x12e4 bytes) 000176 1764745165.920001 mapwrite 0x02ab39 thru 0x038486 (0xd94e bytes) 000177 1764745165.939072 write 0x027819 thru 0x0344e9 (0xccd1 bytes) 000178 1764745165.941287 MIRROR_RESYNC 5 mirrors 000179 1764745166.228080 mapread 0x017ddf thru 0x02779b (0xf9bd bytes) 000180 1764745166.241442 trunc from 0x033469 to 0x038486 (0x501e bytes) 000181 1764745166.249975 read 0x004d34 thru 0x00b876 (0x6b43 bytes) 000182 1764745166.252190 trunc from 0x038487 to 0x03a7b7 (0x2331 bytes) 000183 1764745166.257183 read 0x025110 thru 0x02c8ee (0x77df bytes) 000184 1764745166.260544 trunc from 0x013094 to 0x0199c6 (0x6933 bytes) 000185 1764745166.262199 mapwrite 0x028ee0 thru 0x02e7b3 (0x58d4 bytes) 000186 1764745166.267972 mapread 0x011217 thru 0x018d32 (0x7b1c bytes) 000187 1764745166.275160 write 0x0153c8 thru 0x0200f0 (0xad29 bytes) 000188 1764745166.276020 fallocate 0x03c204 thru 0x03fa31 (0x382e bytes) 000189 1764745166.277561 trunc from 0x017872 to 0x02036a (0x8af9 bytes) 000191 1764745166.281502 mapread 0x003898 thru 0x00e4cb (0xac34 bytes) 000192 1764745166.283441 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a7b8): Mirror 3: 0xaa0d6cce (stale) Mirror 4: 0xaa0d6cce (stale) Mirror 5: 0xaa0d6cce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7b8): Mirror 3: 0xaa0d6cce (stale) Mirror 4: 0xaa0d6cce (stale) Mirror 5: 0xaa0d6cce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7b8: skipped 000193 1764745166.365506 trunc from 0x034d8b to 0x03a7b7 (0x5a2d bytes) 000194 1764745166.367230 write 0x03b692 thru 0x03ffff (0x496e bytes) 000195 1764745166.369233 read 0x0200f8 thru 0x02e0ca (0xdfd3 bytes) 000196 1764745166.371840 trunc from 0x040000 to 0x00d01f (0x32fe0 bytes) 000197 1764745166.376498 trunc from 0x00d020 to 0x034aea (0x27acb bytes) 000198 1764745166.380576 mapread 0x027e5f thru 0x029bbf (0x1d61 bytes) 000199 1764745166.385845 mapread 0x01d591 thru 0x029f28 (0xc998 bytes) 000200 1764745166.390630 trunc from 0x005ef9 to 0x007753 (0x185b bytes) 000201 1764745166.392405 mapread 0x01d4b7 thru 0x02a505 (0xd04f bytes) 000202 1764745166.393021 read 0x000044 thru 0x00ffa7 (0xff64 bytes) 000204 1764745166.396189 trunc from 0x0109c7 to 0x014b55 (0x418f bytes) 000205 1764745166.397876 punch from 0x025225 to 0x03164c (0xc428 bytes) 000206 1764745166.399370 punch from 0x02a99c to 0x033eda (0x953f bytes) 000207 1764745166.401027 read 0x01e9b0 thru 0x02da62 (0xf0b3 bytes) 000208 1764745166.405217 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34aeb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34aeb): Mirror 3: 0x7bc2d3e (stale) Mirror 4: 0x7bc2d3e (stale) Mirror 5: 0x7bc2d3e (stale) CRC-32 checksum value for chunk [0, 0x34aeb): Mirror 1: 0x3e9dcaa1 Mirror 6: 0x3e9dcaa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34aeb: skipped 000209 1764745166.628140 mapwrite 0x005b28 thru 0x00ac4a (0x5123 bytes) 000210 1764745166.663295 mapwrite 0x020e0e thru 0x02bfbc (0xb1af bytes) 000211 1764745166.680663 trunc from 0x026bd2 to 0x030704 (0x9b33 bytes) 000212 1764745166.682620 punch from 0x014246 to 0x015036 (0xdf1 bytes) 000213 1764745166.684209 mapread 0x02c0de thru 0x034aea (0x8a0d bytes) 000214 1764745166.687325 read 0x00d43b thru 0x01cc22 (0xf7e8 bytes) 000215 1764745166.689698 write 0x035018 thru 0x03f2ba (0xa2a3 bytes) 000216 1764745166.692052 punch from 0x03d270 to 0x03da37 (0x7c8 bytes) 000217 1764745166.695775 trunc from 0x01aa31 to 0x0207b1 (0x5d81 bytes) 000218 1764745166.697315 fallocate 0x0098fd thru 0x00d831 (0x3f35 bytes) 000219 1764745166.699069 mapread 0x0285c2 thru 0x02ba75 (0x34b4 bytes) 000220 1764745166.699878 trunc from 0x01b031 to 0x01d4e2 (0x24b2 bytes) 000221 1764745166.701250 mapwrite 0x034330 thru 0x03c0b7 (0x7d88 bytes) 000222 1764745166.707300 write 0x0078e8 thru 0x0141fa (0xc913 bytes) 000223 1764745166.708567 write 0x008974 thru 0x015742 (0xcdcf bytes) 000224 1764745166.711592 MIRROR_RESYNC 5 mirrors 000226 1764745166.987393 read 0x02e224 thru 0x0302b0 (0x208d bytes) 000227 1764745166.989085 mapwrite 0x00a57e thru 0x010d71 (0x67f4 bytes) 000228 1764745167.021175 trunc from 0x03f2bb to 0x00cb55 (0x32765 bytes) 000229 1764745167.030861 fallocate 0x0332f7 thru 0x03ffff (0xcd09 bytes) 000230 1764745167.032592 read 0x036c46 thru 0x03ffff (0x93ba bytes) 000231 1764745167.035852 fallocate 0x011fa0 thru 0x0143fb (0x245c bytes) 000232 1764745167.037808 mapwrite 0x020a29 thru 0x021dde (0x13b6 bytes) 000234 1764745167.043080 fallocate 0x02add1 thru 0x02dc58 (0x2e88 bytes) 000235 1764745167.045142 trunc from 0x00e72b to 0x012f3b (0x4811 bytes) 000236 1764745167.046610 read 0x03042c thru 0x035b91 (0x5766 bytes) 000237 1764745167.047000 read 0x017b75 thru 0x026b96 (0xf022 bytes) 000238 1764745167.047364 fallocate 0x0014fb thru 0x003449 (0x1f4f bytes) 000239 1764745167.048975 read 0x00f34b thru 0x016081 (0x6d37 bytes) 000240 1764745167.051381 trunc from 0x01561c to 0x01c1d3 (0x6bb8 bytes) 000241 1764745167.053091 mapread 0x00c991 thru 0x00f590 (0x2c00 bytes) 000242 1764745167.053792 mapwrite 0x0247d0 thru 0x02f9f3 (0xb224 bytes) 000243 1764745167.060129 mapread 0x01e7f4 thru 0x02c532 (0xdd3f bytes) 000244 1764745167.060689 trunc from 0x040000 to 0x0070cb (0x38f34 bytes) 000245 1764745167.065237 punch from 0x005b4e to 0x0070cb (0x157e bytes) 000246 1764745167.067082 trunc from 0x005f33 to 0x0070cb (0x1199 bytes) 000247 1764745167.068419 mapread 0x002e67 thru 0x0070cb (0x4265 bytes) 000249 1764745167.072407 punch from 0x001a2d to 0x005f1b (0x44ef bytes) 000250 1764745167.075174 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x70cc): Mirror 3: 0xb6990e78 (stale) Mirror 4: 0xb6990e78 (stale) Mirror 6: 0xb6990e78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x70cc): Mirror 3: 0xb6990e78 (stale) Mirror 4: 0xb6990e78 (stale) Mirror 6: 0xb6990e78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x70cc: skipped 000251 1764745167.149384 fallocate 0x007ebb thru 0x00daec (0x5c32 bytes) 000252 1764745167.150833 mapwrite 0x020ab2 thru 0x028f49 (0x8498 bytes) 000255 1764745167.164853 mapwrite 0x03aa83 thru 0x03f5ad (0x4b2b bytes) 000256 1764745167.175112 read 0x01205b thru 0x020788 (0xe72e bytes) 000257 1764745167.178705 MIRROR_RESYNC 4 mirrors 000258 1764745167.440792 read 0x0069cb thru 0x008b1c (0x2152 bytes) 000259 1764745167.443562 MIRROR_RESYNC 4 mirrors 000260 1764745167.558728 fallocate 0x00dcb3 thru 0x00f0ee (0x143c bytes) 000261 1764745167.567997 trunc from 0x00fe2d to 0x013fd3 (0x41a7 bytes) 000262 1764745167.569488 punch from 0x005a20 to 0x0131bf (0xd7a0 bytes) 000263 1764745167.572247 MIRROR_RESYNC 4 mirrors 000264 1764745167.819820 write 0x03ef6a thru 0x03ffff (0x1096 bytes) 000265 1764745167.829599 trunc from 0x015b11 to 0x0244e8 (0xe9d8 bytes) 000266 1764745167.831657 read 0x0065ec thru 0x00dc28 (0x763d bytes) 000267 1764745167.834396 mapwrite 0x039569 thru 0x03ffff (0x6a97 bytes) 000268 1764745167.847078 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7504c882 (stale) Mirror 4: 0x7504c882 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7504c882 (stale) Mirror 4: 0x7504c882 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1764745167.932616 MIRROR_RESYNC 3 mirrors 000270 1764745168.146879 mapread 0x019f20 thru 0x028fba (0xf09b bytes) 000271 1764745168.163126 punch from 0x03dc4a to 0x03ffff (0x23b6 bytes) 000272 1764745168.173066 MIRROR_RESYNC 3 mirrors 000274 1764745168.393731 mapread 0x00f17f thru 0x019295 (0xa117 bytes) 000275 1764745168.405228 trunc from 0x040000 to 0x00513b (0x3aec4 bytes) 000276 1764745168.415501 write 0x035ec4 thru 0x03687d (0x9ba bytes) 000277 1764745168.416456 trunc from 0x03687e to 0x033cc8 (0x2bb5 bytes) 000280 1764745168.422633 write 0x008fb8 thru 0x00d230 (0x4279 bytes) 000281 1764745168.425907 trunc from 0x01eab3 to 0x02127d (0x27cb bytes) 000282 1764745168.427669 write 0x019cb2 thru 0x01b8b8 (0x1c07 bytes) 000283 1764745168.430565 punch from 0x009980 to 0x009c2c (0x2ad bytes) 000285 1764745168.437160 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33cc9): Mirror 4: 0x1eb40081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cc9): Mirror 4: 0x1eb40081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cc9: skipped 000286 1764745168.518830 write 0x030884 thru 0x030bc7 (0x344 bytes) 000287 1764745168.522624 trunc from 0x033cc9 to 0x0051ec (0x2eadc bytes) 000288 1764745168.527613 mapread 0x0038b2 thru 0x0051ec (0x193b bytes) 000289 1764745168.532807 fallocate 0x006d4a thru 0x00b33c (0x45f3 bytes) 000290 1764745168.535595 fallocate 0x004005 thru 0x00f601 (0xb5fd bytes) 000291 1764745168.538617 mapwrite 0x0140dd thru 0x020031 (0xbf55 bytes) 000292 1764745168.560219 read 0x01413d thru 0x0141b0 (0x74 bytes) 000293 1764745168.560698 punch from 0x012872 to 0x014ebb (0x264a bytes) 000294 1764745168.562709 punch from 0x011a18 to 0x01a64b (0x8c34 bytes) 000295 1764745168.564541 read 0x017a7b thru 0x020031 (0x85b7 bytes) 000296 1764745168.567786 mapread 0x00ce20 thru 0x016740 (0x9921 bytes) 000297 1764745168.568587 fallocate 0x01296d thru 0x020264 (0xd8f8 bytes) 000298 1764745168.570823 trunc from 0x01e974 to 0x020031 (0x16be bytes) 000299 1764745168.572534 punch from 0x01cf65 to 0x020031 (0x30cd bytes) 000300 1764745168.574043 write 0x037704 thru 0x03e105 (0x6a02 bytes) 000301 1764745168.574952 trunc from 0x0170e4 to 0x01fdc4 (0x8ce1 bytes) 000302 1764745168.576746 punch from 0x037af8 to 0x03e105 (0x660e bytes) 000303 1764745168.580642 trunc from 0x014230 to 0x01be08 (0x7bd9 bytes) 000304 1764745168.582083 mapread 0x006539 thru 0x00d131 (0x6bf9 bytes) 000306 1764745168.582999 mapread 0x00cc8d thru 0x012212 (0x5586 bytes) 000307 1764745168.583536 mapread 0x034ca0 thru 0x03e105 (0x9466 bytes) 000308 1764745168.587068 punch from 0x024584 to 0x02e586 (0xa003 bytes) 000309 1764745168.588985 mapwrite 0x014b9f thru 0x01d0ad (0x850f bytes) 000310 1764745168.607288 write 0x02acef thru 0x02ff4a (0x525c bytes) 000311 1764745168.609904 mapwrite 0x01079c thru 0x020285 (0xfaea bytes) 000312 1764745168.622151 punch from 0x00d6f8 to 0x00fb98 (0x24a1 bytes) 000313 1764745168.623752 mapwrite 0x02e5da thru 0x0340d5 (0x5afc bytes) 000314 1764745168.628963 fallocate 0x0160b1 thru 0x020198 (0xa0e8 bytes) 000315 1764745168.630565 mapwrite 0x03cf27 thru 0x03ffff (0x30d9 bytes) 000316 1764745168.640172 punch from 0x031025 to 0x032b2b (0x1b07 bytes) 000317 1764745168.642049 punch from 0x002b4d to 0x00f7e6 (0xcc9a bytes) 000318 1764745168.643667 read 0x006da4 thru 0x014518 (0xd775 bytes) 000319 1764745168.646698 write 0x028d47 thru 0x037f58 (0xf212 bytes) 000320 1764745168.647718 fallocate 0x01587f thru 0x023c2b (0xe3ad bytes) 000321 1764745168.649728 punch from 0x009a15 to 0x015a13 (0xbfff bytes) 000322 1764745168.651905 fallocate 0x003ade thru 0x008066 (0x4589 bytes) 000323 1764745168.653843 mapwrite 0x033507 thru 0x03da8e (0xa588 bytes) 000324 1764745168.661186 trunc from 0x040000 to 0x0369a7 (0x9658 bytes) 000325 1764745168.666117 fallocate 0x02cb1a thru 0x03094b (0x3e32 bytes) 000326 1764745168.668074 fallocate 0x013074 thru 0x02305b (0xffe8 bytes) 000327 1764745168.670513 mapwrite 0x02e7b4 thru 0x0338a7 (0x50f4 bytes) 000329 1764745168.678690 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x369a8): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369a8): Mirror 4: 0x35f6fe92 (stale) CRC-32 checksum value for chunk [0, 0x369a8): Mirror 1: 0x613fb0b Mirror 5: 0x613fb0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369a8: skipped 000330 1764745168.899202 trunc from 0x0369a8 to 0x022cb4 (0x13cf3 bytes) 000331 1764745168.913724 fallocate 0x02f5c0 thru 0x0390fc (0x9b3d bytes) 000332 1764745168.915169 trunc from 0x022cb5 to 0x02eeff (0xc24b bytes) 000333 1764745168.919329 fallocate 0x00b11a thru 0x0182b2 (0xd199 bytes) 000334 1764745168.920947 trunc from 0x02ef00 to 0x03c82d (0xd92e bytes) 000335 1764745168.924711 trunc from 0x032075 to 0x0390f6 (0x7082 bytes) 000336 1764745168.926369 read 0x02a81b thru 0x02b5e1 (0xdc7 bytes) 000338 1764745168.928799 mapwrite 0x036b1f thru 0x03ffff (0x94e1 bytes) 000340 1764745168.944496 mapread 0x03bc2d thru 0x03ffff (0x43d3 bytes) 000341 1764745168.946674 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1c333af3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1c333af3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000342 1764745169.018586 trunc from 0x02df02 to 0x037060 (0x915f bytes) 000343 1764745169.020373 punch from 0x03aaac to 0x03ffff (0x5554 bytes) 000344 1764745169.022176 mapread 0x0317ea thru 0x038e3a (0x7651 bytes) 000345 1764745169.032484 write 0x037898 thru 0x03cdcb (0x5534 bytes) 000346 1764745169.034719 punch from 0x000ec6 to 0x004791 (0x38cc bytes) 000347 1764745169.036422 mapwrite 0x000183 thru 0x00246e (0x22ec bytes) 000348 1764745169.043720 mapread 0x01c8fb thru 0x0239d8 (0x70de bytes) 000349 1764745169.053159 fallocate 0x004f45 thru 0x008fb6 (0x4072 bytes) 000350 1764745169.055154 fallocate 0x006679 thru 0x00cbfa (0x6582 bytes) 000352 1764745169.058757 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000354 1764745169.096317 punch from 0x03268f to 0x03b0c0 (0x8a32 bytes) 000355 1764745169.098341 fallocate 0x00c65a thru 0x01705b (0xaa02 bytes) 000357 1764745169.100476 write 0x01346f thru 0x01ce92 (0x9a24 bytes) 000359 1764745169.105259 punch from 0x01a63d to 0x025005 (0xa9c9 bytes) 000360 1764745169.109681 mapwrite 0x01faff thru 0x02d0a1 (0xd5a3 bytes) 000362 1764745169.132541 mapread 0x00ec13 thru 0x012019 (0x3407 bytes) 000363 1764745169.140043 trunc from 0x039a83 to 0x03ff75 (0x64f3 bytes) 000364 1764745169.141720 read 0x02ac3d thru 0x037c2b (0xcfef bytes) 000365 1764745169.144652 read 0x036b1e thru 0x03c1bc (0x569f bytes) 000366 1764745169.147818 trunc from 0x040000 to 0x02cfd1 (0x1302e bytes) 000368 1764745169.152753 write 0x03bade thru 0x03ffff (0x4522 bytes) 000370 1764745169.154098 trunc from 0x040000 to 0x00087b (0x3f784 bytes) 000371 1764745169.158249 read 0x0001e0 thru 0x00087b (0x69c bytes) 000372 1764745169.158559 fallocate 0x037f2a thru 0x03ffff (0x80d6 bytes) 000373 1764745169.160676 trunc from 0x00b6fe to 0x00fbbc (0x44bf bytes) 000374 1764745169.162130 trunc from 0x040000 to 0x03dc6a (0x2395 bytes) 000375 1764745169.165951 mapwrite 0x00b8aa thru 0x0188ea (0xd041 bytes) 000376 1764745169.176497 MIRROR_EXTEND to 2 mirrors 000377 1764745169.325819 MIRROR_EXTEND to 3 mirrors 000378 1764745169.475186 write 0x031e0e thru 0x033ffa (0x21ed bytes) 000379 1764745169.486561 trunc from 0x03dc6b to 0x02d378 (0x108f2 bytes) 000380 1764745169.495145 punch from 0x0247b2 to 0x02d378 (0x8bc7 bytes) 000381 1764745169.497620 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d379): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d379): Mirror 2: 0xb35db385 (stale) Mirror 3: 0xb35db385 (stale) CRC-32 checksum value for chunk [0, 0x2d379): Mirror 1: 0xb35db385 Mirror 4: 0xb35db385 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d379: skipped 000382 1764745169.712110 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d379): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d379): Mirror 2: 0xb35db385 (stale) Mirror 3: 0xb35db385 (stale) CRC-32 checksum value for chunk [0, 0x2d379): Mirror 1: 0xb35db385 Mirror 4: 0xb35db385 Mirror 5: 0xb35db385 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d379: skipped 000383 1764745169.938033 mapread 0x01ac60 thru 0x01e362 (0x3703 bytes) 000384 1764745169.942738 trunc from 0x02d379 to 0x010e86 (0x1c4f2 bytes) 000385 1764745169.960013 mapwrite 0x01aaf4 thru 0x01b728 (0xc35 bytes) 000386 1764745169.971157 trunc from 0x00fdc3 to 0x013a5c (0x3c9a bytes) 000387 1764745169.972619 mapread 0x002687 thru 0x011a03 (0xf37d bytes) 000388 1764745169.977653 trunc from 0x006aec to 0x00db47 (0x705c bytes) 000389 1764745169.979254 mapread 0x00345c thru 0x010b39 (0xd6de bytes) 000390 1764745169.984152 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b729): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b729): Mirror 2: 0x1d01172a (stale) Mirror 3: 0x1d01172a (stale) Mirror 4: 0x1d01172a (stale) Mirror 5: 0x1d01172a (stale) CRC-32 checksum value for chunk [0, 0x1b729): Mirror 1: 0x96574563 Mirror 6: 0x96574563 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b729: skipped 000391 1764745170.260005 mapwrite 0x02687b thru 0x035484 (0xec0a bytes) 000392 1764745170.285380 write 0x026bfa thru 0x0352ad (0xe6b4 bytes) 000394 1764745170.286552 trunc from 0x01e57e to 0x02c336 (0xddb9 bytes) 000395 1764745170.291106 read 0x00d2a3 thru 0x00ed91 (0x1aef bytes) 000396 1764745170.293028 read 0x010280 thru 0x01e2f2 (0xe073 bytes) 000397 1764745170.296434 trunc from 0x01f42e to 0x022dab (0x397e bytes) 000399 1764745170.298987 mapwrite 0x034bee thru 0x037c37 (0x304a bytes) 000400 1764745170.313786 trunc from 0x01ec72 to 0x01f0db (0x46a bytes) 000401 1764745170.316093 write 0x00938b thru 0x015d3c (0xc9b2 bytes) 000402 1764745170.317198 mapread 0x02cf58 thru 0x02e0fc (0x11a5 bytes) 000403 1764745170.317802 mapread 0x005bc3 thru 0x00adc1 (0x51ff bytes) 000404 1764745170.318499 trunc from 0x0207f1 to 0x029694 (0x8ea4 bytes) 000405 1764745170.320673 fallocate 0x03f75c thru 0x03ffff (0x8a4 bytes) 000407 1764745170.323130 mapread 0x011f14 thru 0x012db7 (0xea4 bytes) 000408 1764745170.323944 trunc from 0x0215b6 to 0x02605b (0x4aa6 bytes) 000409 1764745170.326204 trunc from 0x037c38 to 0x02ca2e (0xb209 bytes) 000410 1764745170.331669 trunc from 0x009dbe to 0x00d2f6 (0x3539 bytes) 000411 1764745170.336186 mapwrite 0x00dbf6 thru 0x0106cd (0x2ad8 bytes) 000413 1764745170.344072 read 0x00df59 thru 0x016e20 (0x8ec8 bytes) 000414 1764745170.344558 mapwrite 0x00d114 thru 0x015a70 (0x895d bytes) 000415 1764745170.351387 trunc from 0x02ca2f to 0x00bbf2 (0x20e3c bytes) 000416 1764745170.355864 trunc from 0x00bbf3 to 0x027320 (0x1b72e bytes) 000417 1764745170.359960 punch from 0x014c63 to 0x01aeaa (0x6248 bytes) 000418 1764745170.361485 write 0x021fc4 thru 0x02e9ca (0xca07 bytes) 000419 1764745170.365275 mapread 0x02e302 thru 0x02e9ca (0x6c9 bytes) 000420 1764745170.365807 fallocate 0x028539 thru 0x033428 (0xaef0 bytes) 000421 1764745170.367890 mapwrite 0x02e6b3 thru 0x030fb3 (0x2901 bytes) 000422 1764745170.379295 read 0x00aab9 thru 0x01012f (0x5677 bytes) 000423 1764745170.379814 read 0x00e6fa thru 0x0159d9 (0x72e0 bytes) 000424 1764745170.380231 write 0x0070c5 thru 0x012363 (0xb29f bytes) 000426 1764745170.381434 punch from 0x014946 to 0x01e3ef (0x9aaa bytes) 000427 1764745170.383366 read 0x000fad thru 0x00e131 (0xd185 bytes) 000428 1764745170.383754 punch from 0x01e2fe to 0x020324 (0x2027 bytes) 000429 1764745170.385276 trunc from 0x004a53 to 0x00aa76 (0x6024 bytes) 000430 1764745170.389255 mapwrite 0x025369 thru 0x02682a (0x14c2 bytes) 000431 1764745170.393664 read 0x005c0d thru 0x00c197 (0x658b bytes) 000432 1764745170.397185 trunc from 0x015215 to 0x025187 (0xff73 bytes) 000433 1764745170.399563 mapread 0x0250b3 thru 0x025509 (0x457 bytes) 000434 1764745170.401629 trunc from 0x0231ed to 0x02db17 (0xa92b bytes) 000435 1764745170.403474 fallocate 0x005e39 thru 0x013be4 (0xddac bytes) 000437 1764745170.405748 write 0x021c4e thru 0x0240c3 (0x2476 bytes) 000438 1764745170.408620 mapwrite 0x03278d thru 0x03ffff (0xd873 bytes) 000440 1764745170.423693 write 0x012e08 thru 0x0134d9 (0x6d2 bytes) 000441 1764745170.424505 mapwrite 0x03a753 thru 0x03ffff (0x58ad bytes) 000442 1764745170.431834 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83947e20 (stale) Mirror 3: 0x83947e20 (stale) Mirror 4: 0x83947e20 (stale) Mirror 5: 0x83947e20 (stale) Mirror 6: 0x9b81791e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf65f0a28 Mirror 7: 0xf65f0a28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1764745170.687639 punch from 0x009f2f to 0x017d7e (0xde50 bytes) 000444 1764745170.700661 MIRROR_RESYNC 7 mirrors 000445 1764745171.145655 mapwrite 0x00d996 thru 0x016a18 (0x9083 bytes) 000446 1764745171.192166 read 0x02b2bb thru 0x032eb7 (0x7bfd bytes) 000448 1764745171.195336 trunc from 0x0365bc to 0x03845f (0x1ea4 bytes) 000449 1764745171.197221 trunc from 0x040000 to 0x02074d (0x1f8b2 bytes) 000450 1764745171.203650 MIRROR_RESYNC 7 mirrors 000451 1764745171.539754 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000452 1764745171.630656 mapread 0x014adf thru 0x01eca6 (0xa1c8 bytes) 000453 1764745171.636923 mapwrite 0x01419b thru 0x01b079 (0x6edf bytes) 000454 1764745171.669511 fallocate 0x023ec6 thru 0x032613 (0xe74e bytes) 000455 1764745171.671332 fallocate 0x031ada thru 0x0328f8 (0xe1f bytes) 000456 1764745171.673271 fallocate 0x00b3e0 thru 0x018af4 (0xd715 bytes) 000457 1764745171.675027 trunc from 0x0328f9 to 0x00a3e5 (0x28513 bytes) 000458 1764745171.685785 read 0x0096df thru 0x00975c (0x7e bytes) 000459 1764745171.687646 punch from 0x005692 to 0x00a3e5 (0x4d54 bytes) 000460 1764745171.689174 punch from 0x00393a to 0x009c6e (0x6335 bytes) 000461 1764745171.690488 punch from 0x002eca to 0x0086fe (0x5835 bytes) 000462 1764745171.691817 mapwrite 0x010949 thru 0x018b5b (0x8213 bytes) 000463 1764745171.703397 trunc from 0x018b5c to 0x01f4a4 (0x6949 bytes) 000465 1764745171.708842 MIRROR_RESYNC 6 mirrors 000466 1764745171.972389 fallocate 0x003a61 thru 0x008eb9 (0x5459 bytes) 000467 1764745171.982407 fallocate 0x01d657 thru 0x01fa98 (0x2442 bytes) 000468 1764745171.984104 punch from 0x00292c to 0x00efc9 (0xc69e bytes) 000469 1764745171.985575 read 0x019cb3 thru 0x01fa98 (0x5de6 bytes) 000470 1764745171.988253 mapwrite 0x000629 thru 0x0038e3 (0x32bb bytes) 000471 1764745171.998118 mapread 0x01cddc thru 0x01fa98 (0x2cbd bytes) 000472 1764745171.998994 trunc from 0x01db2f to 0x01f101 (0x15d3 bytes) 000473 1764745172.000630 trunc from 0x0131c0 to 0x01d251 (0xa092 bytes) 000474 1764745172.002383 fallocate 0x0128bc thru 0x018913 (0x6058 bytes) 000475 1764745172.004212 punch from 0x00be28 to 0x01260f (0x67e8 bytes) 000477 1764745172.005986 read 0x011975 thru 0x015ff7 (0x4683 bytes) 000478 1764745172.008729 write 0x0187a7 thru 0x0218c3 (0x911d bytes) 000479 1764745172.011073 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x218c4): Mirror 2: 0x42489a7f (stale) Mirror 5: 0x42489a7f (stale) Mirror 6: 0x42489a7f (stale) Mirror 7: 0x42489a7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x218c4): Mirror 2: 0x42489a7f (stale) Mirror 5: 0x42489a7f (stale) Mirror 6: 0x42489a7f (stale) Mirror 7: 0x42489a7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x218c4: skipped 000480 1764745172.104669 write 0x03d9d6 thru 0x03ffff (0x262a bytes) 000481 1764745172.106715 punch from 0x00d066 to 0x013871 (0x680c bytes) 000482 1764745172.108366 trunc from 0x010b0c to 0x018596 (0x7a8b bytes) 000483 1764745172.109935 read 0x032355 thru 0x03ffff (0xdcab bytes) 000484 1764745172.112127 read 0x012122 thru 0x0139d3 (0x18b2 bytes) 000485 1764745172.114863 read 0x02bf85 thru 0x02d1f4 (0x1270 bytes) 000486 1764745172.115241 trunc from 0x040000 to 0x0170e4 (0x28f1b bytes) 000487 1764745172.122478 trunc from 0x0170e5 to 0x02945b (0x12377 bytes) 000488 1764745172.125574 trunc from 0x012432 to 0x01d58d (0xb15c bytes) 000489 1764745172.127002 fallocate 0x0247e5 thru 0x0311d9 (0xc9f5 bytes) 000490 1764745172.128351 read 0x027b91 thru 0x02e5e9 (0x6a59 bytes) 000491 1764745172.132502 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x311da): Mirror 2: 0xc1071033 (stale) Mirror 5: 0xc1071033 (stale) Mirror 7: 0xc1071033 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311da): Mirror 2: 0xc1071033 (stale) Mirror 5: 0xc1071033 (stale) Mirror 7: 0xc1071033 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311da: skipped 000492 1764745172.213258 punch from 0x012e0b to 0x017d29 (0x4f1f bytes) 000494 1764745172.215165 mapwrite 0x004b8e thru 0x005a8b (0xefe bytes) 000495 1764745172.221385 trunc from 0x0092a2 to 0x00b1cd (0x1f2c bytes) 000496 1764745172.222642 write 0x003406 thru 0x009269 (0x5e64 bytes) 000497 1764745172.226992 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x311da): Mirror 5: 0xc1071033 (stale) Mirror 7: 0xc1071033 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311da): Mirror 5: 0xc1071033 (stale) Mirror 7: 0xc1071033 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311da: skipped 000498 1764745172.297528 MIRROR_RESYNC 3 mirrors 000499 1764745172.443932 trunc from 0x0311da to 0x0043a7 (0x2ce32 bytes) 000500 1764745172.457186 mapwrite 0x01a44b thru 0x0262e2 (0xbe98 bytes) 000501 1764745172.474012 punch from 0x00435f to 0x007290 (0x2f32 bytes) 000502 1764745172.475407 fallocate 0x01a920 thru 0x02976d (0xee4e bytes) 000503 1764745172.476840 punch from 0x01e899 to 0x029212 (0xa97a bytes) 000504 1764745172.478126 mapwrite 0x03b653 thru 0x03df82 (0x2930 bytes) 000505 1764745172.487550 write 0x02037a thru 0x028099 (0x7d20 bytes) 000506 1764745172.490880 mapwrite 0x039e17 thru 0x03ffff (0x61e9 bytes) 000507 1764745172.502338 mapread 0x02e82b thru 0x03cce4 (0xe4ba bytes) 000509 1764745172.510914 write 0x03bbf8 thru 0x03ffff (0x4408 bytes) 000510 1764745172.511646 mapread 0x02a402 thru 0x0340fa (0x9cf9 bytes) 000511 1764745172.512374 mapread 0x01d3c1 thru 0x020ee6 (0x3b26 bytes) 000512 1764745172.512892 read 0x023a3a thru 0x02572d (0x1cf4 bytes) 000513 1764745172.513235 mapwrite 0x00de99 thru 0x01a810 (0xc978 bytes) 000515 1764745172.530423 MIRROR_RESYNC 3 mirrors 000516 1764745172.731356 trunc from 0x03c4a7 to 0x03ffff (0x3b59 bytes) 000517 1764745172.739964 mapread 0x020c2d thru 0x0255b6 (0x498a bytes) 000518 1764745172.748536 mapread 0x0345f7 thru 0x0392f9 (0x4d03 bytes) 000519 1764745172.755651 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa37796c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa37796c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000520 1764745172.822874 fallocate 0x006095 thru 0x00d48f (0x73fb bytes) 000521 1764745172.824681 trunc from 0x040000 to 0x032f42 (0xd0bd bytes) 000522 1764745172.833309 write 0x03c9f8 thru 0x03ffff (0x3608 bytes) 000523 1764745172.835255 MIRROR_RESYNC 2 mirrors 000524 1764745173.001569 MIRROR_EXTEND to 3 mirrors 000525 1764745173.166820 mapwrite 0x01a931 thru 0x01c60d (0x1cdd bytes) 000526 1764745173.198814 MIRROR_RESYNC 3 mirrors 000527 1764745173.397373 write 0x00503b thru 0x013324 (0xe2ea bytes) 000528 1764745173.405488 write 0x023904 thru 0x02b50e (0x7c0b bytes) 000529 1764745173.408553 mapread 0x017f2f thru 0x0276ba (0xf78c bytes) 000530 1764745173.414063 mapread 0x015da1 thru 0x01cb14 (0x6d74 bytes) 000531 1764745173.414526 fallocate 0x000cd3 thru 0x00f3b9 (0xe6e7 bytes) 000532 1764745173.416000 mapwrite 0x01c676 thru 0x01e5ee (0x1f79 bytes) 000534 1764745173.420840 trunc from 0x040000 to 0x02d192 (0x12e6d bytes) 000535 1764745173.426952 MIRROR_RESYNC 3 mirrors 000536 1764745173.614236 mapread 0x027c00 thru 0x02d192 (0x5593 bytes) 000537 1764745173.619007 fallocate 0x017bb5 thru 0x0197f8 (0x1c44 bytes) 000538 1764745173.628035 mapwrite 0x031b00 thru 0x03d633 (0xbb34 bytes) 000539 1764745173.649820 fallocate 0x014b85 thru 0x0234f3 (0xe96f bytes) 000540 1764745173.651740 mapread 0x02540b thru 0x02decc (0x8ac2 bytes) 000541 1764745173.657755 read 0x038f4a thru 0x03d633 (0x46ea bytes) 000542 1764745173.658314 trunc from 0x03d634 to 0x0269b1 (0x16c82 bytes) 000543 1764745173.663235 read 0x00b975 thru 0x017041 (0xb6cd bytes) 000544 1764745173.666790 read 0x004c85 thru 0x00a3db (0x5757 bytes) 000545 1764745173.667331 trunc from 0x020342 to 0x022d8d (0x2a4c bytes) 000547 1764745173.669617 mapwrite 0x011bd9 thru 0x01c883 (0xacab bytes) 000549 1764745173.676729 read 0x01216d thru 0x019971 (0x7805 bytes) 000550 1764745173.677145 write 0x0201df thru 0x02a229 (0xa04b bytes) 000551 1764745173.679808 trunc from 0x008617 to 0x00ee03 (0x67ed bytes) 000552 1764745173.681645 trunc from 0x0008ee to 0x009b41 (0x9254 bytes) 000555 1764745173.683848 write 0x006e79 thru 0x00b05b (0x41e3 bytes) 000556 1764745173.686949 fallocate 0x0338fa thru 0x03c5a5 (0x8cac bytes) 000557 1764745173.690137 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5a6): Mirror 6: 0xcd3023de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5a6): Mirror 6: 0xcd3023de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5a6: skipped 000558 1764745173.763071 mapwrite 0x009582 thru 0x009f88 (0xa07 bytes) 000559 1764745173.768458 write 0x02fcfb thru 0x0350be (0x53c4 bytes) 000561 1764745173.774869 MIRROR_RESYNC 2 mirrors 000562 1764745173.972607 MIRROR_EXTEND to 3 mirrors 000563 1764745174.169426 trunc from 0x03c5a6 to 0x013ae3 (0x28ac2 bytes) 000564 1764745174.180927 mapwrite 0x0320e1 thru 0x03a031 (0x7f51 bytes) 000565 1764745174.195489 fallocate 0x00d4a8 thru 0x0191b8 (0xbd11 bytes) 000566 1764745174.197097 mapread 0x0366ff thru 0x03a031 (0x3933 bytes) 000567 1764745174.197596 write 0x011988 thru 0x0158b7 (0x3f30 bytes) 000568 1764745174.200936 read 0x022bd5 thru 0x031d1c (0xf148 bytes) 000569 1764745174.203263 punch from 0x0219be to 0x031555 (0xfb98 bytes) 000570 1764745174.204979 mapwrite 0x03edf2 thru 0x03ffff (0x120e bytes) 000571 1764745174.215889 fallocate 0x03a5c1 thru 0x03ffff (0x5a3f bytes) 000572 1764745174.217775 fallocate 0x00c473 thru 0x018250 (0xbdde bytes) 000573 1764745174.219286 mapwrite 0x0082a6 thru 0x011883 (0x95de bytes) 000574 1764745174.230132 mapread 0x010939 thru 0x0171c4 (0x688c bytes) 000575 1764745174.234785 mapread 0x018e30 thru 0x027d5a (0xef2b bytes) 000576 1764745174.246104 mapread 0x02f9ae thru 0x03ddf3 (0xe446 bytes) 000577 1764745174.250083 punch from 0x00f948 to 0x01f666 (0xfd1f bytes) 000578 1764745174.252278 trunc from 0x01336b to 0x016fe4 (0x3c7a bytes) 000580 1764745174.254587 read 0x033d6a thru 0x0390dd (0x5374 bytes) 000581 1764745174.255178 write 0x0106be thru 0x01a577 (0x9eba bytes) 000582 1764745174.258927 punch from 0x00eeed to 0x01997c (0xaa90 bytes) 000583 1764745174.264391 trunc from 0x040000 to 0x026c72 (0x1938d bytes) 000584 1764745174.269533 trunc from 0x026c73 to 0x035921 (0xecaf bytes) 000585 1764745174.274280 trunc from 0x02ef8f to 0x02f615 (0x687 bytes) 000586 1764745174.276099 read 0x02e3d2 thru 0x032d51 (0x4980 bytes) 000587 1764745174.279173 trunc from 0x035922 to 0x0178f7 (0x1e02a bytes) 000588 1764745174.283417 punch from 0x004ef1 to 0x00b893 (0x69a3 bytes) 000589 1764745174.285253 fallocate 0x0318ac thru 0x036188 (0x48dd bytes) 000590 1764745174.286990 mapwrite 0x03b7bf thru 0x03ffff (0x4841 bytes) 000591 1764745174.297237 fallocate 0x029a8f thru 0x0350a5 (0xb617 bytes) 000592 1764745174.298750 write 0x03c072 thru 0x03ffff (0x3f8e bytes) 000594 1764745174.299618 read 0x032ce2 thru 0x03ffff (0xd31e bytes) 000595 1764745174.302635 trunc from 0x016571 to 0x01fa3a (0x94ca bytes) 000596 1764745174.304408 mapwrite 0x0309fb thru 0x03e539 (0xdb3f bytes) 000597 1764745174.311612 trunc from 0x00397e to 0x00498f (0x1012 bytes) 000599 1764745174.313343 trunc from 0x040000 to 0x0390f6 (0x6f09 bytes) 000600 1764745174.317330 read 0x00af97 thru 0x01128b (0x62f5 bytes) 000601 1764745174.317873 punch from 0x025694 to 0x030dd3 (0xb740 bytes) 000602 1764745174.319544 fallocate 0x008d31 thru 0x012e71 (0xa141 bytes) 000603 1764745174.321156 fallocate 0x03f3d6 thru 0x03ffff (0xc2a bytes) 000604 1764745174.322799 trunc from 0x0173d9 to 0x01bdb8 (0x49e0 bytes) 000605 1764745174.324264 punch from 0x02ac36 to 0x02ad41 (0x10c bytes) 000606 1764745174.325484 mapread 0x009abc thru 0x01418f (0xa6d4 bytes) 000607 1764745174.326134 read 0x027a8e thru 0x02dc0a (0x617d bytes) 000608 1764745174.329184 fallocate 0x022c4d thru 0x029a44 (0x6df8 bytes) 000609 1764745174.330785 fallocate 0x00514c thru 0x00def4 (0x8da9 bytes) 000610 1764745174.332263 trunc from 0x040000 to 0x022bf0 (0x1d40f bytes) 000611 1764745174.338242 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22bf1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22bf1): Mirror 6: 0x13f32378 (stale) Mirror 7: 0x13f32378 (stale) CRC-32 checksum value for chunk [0, 0x22bf1): Mirror 1: 0xed20bff9 Mirror 8: 0xed20bff9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22bf1: skipped 000612 1764745174.549598 mapread 0x00b6fa thru 0x012091 (0x6998 bytes) 000613 1764745174.560650 trunc from 0x022bf1 to 0x03ab73 (0x17f83 bytes) 000614 1764745174.578563 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ab74): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab74): Mirror 6: 0xfbda13de (stale) Mirror 7: 0xfbda13de (stale) Mirror 8: 0xd4ab5b92 (stale) CRC-32 checksum value for chunk [0, 0x3ab74): Mirror 1: 0xd4ab5b92 Mirror 9: 0xd4ab5b92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab74: skipped 000616 1764745174.811048 read 0x004ed6 thru 0x00ffd3 (0xb0fe bytes) 000617 1764745174.815199 fallocate 0x036658 thru 0x03ffff (0x99a8 bytes) 000618 1764745174.826647 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf43ba9c (stale) Mirror 7: 0xbf43ba9c (stale) Mirror 8: 0x664b4223 (stale) Mirror 9: 0x664b4223 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x664b4223 Mirror 10: 0x664b4223 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1764745175.066554 write 0x00abf4 thru 0x0183dc (0xd7e9 bytes) 000621 1764745175.084904 write 0x01bddb thru 0x02a52a (0xe750 bytes) 000622 1764745175.090037 mapwrite 0x0032ee thru 0x00a018 (0x6d2b bytes) 000623 1764745175.099953 read 0x02da5c thru 0x032d0d (0x52b2 bytes) 000624 1764745175.102724 mapwrite 0x03949f thru 0x03ffff (0x6b61 bytes) 000625 1764745175.111464 mapread 0x03f835 thru 0x03fad1 (0x29d bytes) 000626 1764745175.113578 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf43ba9c (stale) Mirror 7: 0xbf43ba9c (stale) Mirror 8: 0x664b4223 (stale) Mirror 9: 0x664b4223 (stale) Mirror 10: 0x664b4223 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe09ac4e0 Mirror 11: 0xe09ac4e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000627 1764745175.361930 mapwrite 0x01b0ee thru 0x02aaab (0xf9be bytes) 000629 1764745175.416680 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf43ba9c (stale) Mirror 7: 0xbf43ba9c (stale) Mirror 8: 0x664b4223 (stale) Mirror 9: 0x664b4223 (stale) Mirror 10: 0x664b4223 (stale) Mirror 11: 0xe09ac4e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe86d33d6 Mirror 12: 0xe86d33d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1764745175.678218 read 0x021807 thru 0x02388d (0x2087 bytes) 000631 1764745175.682470 punch from 0x023dd7 to 0x031b34 (0xdd5e bytes) 000632 1764745175.694333 trunc from 0x040000 to 0x000990 (0x3f66f bytes) 000633 1764745175.706373 trunc from 0x000991 to 0x038ec8 (0x38538 bytes) 000634 1764745175.712350 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ec9): Mirror 6: 0xef8f85bf (stale) Mirror 7: 0xef8f85bf (stale) Mirror 8: 0x60ca10bb (stale) Mirror 9: 0x60ca10bb (stale) Mirror 11: 0x83320f0 (stale) Mirror 12: 0xcd9e1a39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ec9): Mirror 6: 0xef8f85bf (stale) Mirror 7: 0xef8f85bf (stale) Mirror 8: 0x60ca10bb (stale) Mirror 9: 0x60ca10bb (stale) Mirror 11: 0x83320f0 (stale) Mirror 12: 0xcd9e1a39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ec9: skipped 000635 1764745175.831162 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ec9): Mirror 6: 0xef8f85bf (stale) Mirror 7: 0xef8f85bf (stale) Mirror 9: 0x60ca10bb (stale) Mirror 11: 0x83320f0 (stale) Mirror 12: 0xcd9e1a39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ec9): Mirror 6: 0xef8f85bf (stale) Mirror 7: 0xef8f85bf (stale) Mirror 9: 0x60ca10bb (stale) Mirror 11: 0x83320f0 (stale) Mirror 12: 0xcd9e1a39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ec9: skipped 000636 1764745175.934711 punch from 0x022239 to 0x02e5b1 (0xc379 bytes) 000638 1764745175.936729 punch from 0x0335a4 to 0x038ec8 (0x5925 bytes) 000639 1764745175.938492 trunc from 0x00511a to 0x006894 (0x177b bytes) 000640 1764745175.940405 fallocate 0x019640 thru 0x028939 (0xf2fa bytes) 000641 1764745175.942192 punch from 0x005039 to 0x011c38 (0xcc00 bytes) 000642 1764745175.944003 fallocate 0x02d548 thru 0x032138 (0x4bf1 bytes) 000643 1764745175.946070 write 0x0321f6 thru 0x03ffff (0xde0a bytes) 000644 1764745175.950418 write 0x002312 thru 0x00accc (0x89bb bytes) 000645 1764745175.954454 read 0x00aced thru 0x00d9ca (0x2cde bytes) 000646 1764745175.959827 fallocate 0x00a986 thru 0x00e6d7 (0x3d52 bytes) 000647 1764745175.962783 mapread 0x037be7 thru 0x03ffff (0x8419 bytes) 000648 1764745175.963634 write 0x008a16 thru 0x0162d0 (0xd8bb bytes) 000650 1764745175.965330 trunc from 0x02dce6 to 0x03a72e (0xca49 bytes) 000651 1764745175.973704 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbf43ba9c (stale) Mirror 7: 0xbf43ba9c (stale) Mirror 9: 0x664b4223 (stale) Mirror 11: 0xe09ac4e0 (stale) Mirror 12: 0xe86d33d6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x980067de Mirror 13: 0x980067de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000652 1764745176.243869 read 0x01e7b7 thru 0x02cd1e (0xe568 bytes) 000653 1764745176.248552 trunc from 0x040000 to 0x0097da (0x36825 bytes) 000655 1764745176.262161 read 0x002371 thru 0x0097da (0x746a bytes) 000656 1764745176.264504 trunc from 0x0097db to 0x03291d (0x29143 bytes) 000657 1764745176.270909 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3291e): Mirror 7: 0x194cf83a (stale) Mirror 9: 0x5080df19 (stale) Mirror 11: 0xbecbd50b (stale) Mirror 12: 0x4035fd13 (stale) Mirror 13: 0x14785809 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3291e): Mirror 7: 0x194cf83a (stale) Mirror 9: 0x5080df19 (stale) Mirror 11: 0xbecbd50b (stale) Mirror 12: 0x4035fd13 (stale) Mirror 13: 0x14785809 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3291e: skipped 000658 1764745176.381827 trunc from 0x002513 to 0x00a185 (0x7c73 bytes) 000659 1764745176.383525 trunc from 0x02cfb9 to 0x03291d (0x5965 bytes) 000660 1764745176.384998 write 0x0394b0 thru 0x03af25 (0x1a76 bytes) 000661 1764745176.388455 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af26): Mirror 9: 0x50a5933f (stale) Mirror 11: 0xa63f1d57 (stale) Mirror 12: 0xa09f8790 (stale) Mirror 13: 0x51c20edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af26): Mirror 9: 0x50a5933f (stale) Mirror 11: 0xa63f1d57 (stale) Mirror 12: 0xa09f8790 (stale) Mirror 13: 0x51c20edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af26: skipped 000663 1764745176.486113 fallocate 0x03bfc2 thru 0x03d895 (0x18d4 bytes) 000664 1764745176.487972 trunc from 0x03d896 to 0x03e2a3 (0xa0e bytes) 000665 1764745176.492538 write 0x02b39b thru 0x0366e2 (0xb348 bytes) 000666 1764745176.495724 read 0x03a120 thru 0x03e08c (0x3f6d bytes) 000667 1764745176.498071 trunc from 0x0340b1 to 0x03cf03 (0x8e53 bytes) 000668 1764745176.503640 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e2a4): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2a4): Mirror 9: 0x8bf2d878 (stale) Mirror 11: 0x3ba1ddbc (stale) Mirror 12: 0x5a15c87d (stale) Mirror 13: 0x42227ff2 (stale) CRC-32 checksum value for chunk [0, 0x3e2a4): Mirror 1: 0x768be90f Mirror 14: 0x768be90f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2a4: skipped 000669 1764745176.736035 MIRROR_RESYNC 6 mirrors 000670 1764745177.038130 trunc from 0x03e2a4 to 0x01fbd5 (0x1e6ce bytes) 000671 1764745177.053660 trunc from 0x01fbd6 to 0x036b51 (0x16f7c bytes) 000672 1764745177.058643 fallocate 0x035811 thru 0x03ffff (0xa7ef bytes) 000673 1764745177.060243 mapread 0x0283f7 thru 0x03715d (0xed67 bytes) 000674 1764745177.076441 trunc from 0x01d400 to 0x028fce (0xbbcf bytes) 000675 1764745177.078296 trunc from 0x040000 to 0x002014 (0x3dfeb bytes) 000677 1764745177.083955 read 0x001ee0 thru 0x002014 (0x135 bytes) 000678 1764745177.085935 trunc from 0x0013f0 to 0x002014 (0xc25 bytes) 000679 1764745177.087650 trunc from 0x0016ea to 0x002014 (0x92b bytes) 000680 1764745177.089180 mapwrite 0x0251ff thru 0x02fc8b (0xaa8d bytes) 000681 1764745177.105783 mapwrite 0x020890 thru 0x02d3c1 (0xcb32 bytes) 000682 1764745177.112148 trunc from 0x012f85 to 0x022417 (0xf493 bytes) 000683 1764745177.113776 mapread 0x01d682 thru 0x020252 (0x2bd1 bytes) 000684 1764745177.117969 read 0x0122e3 thru 0x01b6a5 (0x93c3 bytes) 000685 1764745177.121478 fallocate 0x004021 thru 0x011199 (0xd179 bytes) 000686 1764745177.123541 fallocate 0x00be10 thru 0x01773e (0xb92f bytes) 000687 1764745177.126314 MIRROR_RESYNC 6 mirrors 000688 1764745177.423692 MIRROR_RESYNC 6 mirrors 000689 1764745177.573124 trunc from 0x020f83 to 0x02a7dd (0x985b bytes) 000690 1764745177.585400 MIRROR_RESYNC 6 mirrors 000692 1764745177.870510 punch from 0x02ed03 to 0x02fc8b (0xf89 bytes) 000693 1764745177.880207 read 0x027300 thru 0x02fc8b (0x898c bytes) 000694 1764745177.882510 punch from 0x0227ac to 0x02fc8b (0xd4e0 bytes) 000695 1764745177.884127 punch from 0x003573 to 0x004ae1 (0x156f bytes) 000696 1764745177.885490 mapwrite 0x01a206 thru 0x0261be (0xbfb9 bytes) 000697 1764745177.903965 trunc from 0x02fc8c to 0x005129 (0x2ab62 bytes) 000699 1764745177.914394 punch from 0x003033 to 0x005129 (0x20f7 bytes) 000701 1764745177.916260 punch from 0x000eb0 to 0x003c71 (0x2dc2 bytes) 000702 1764745177.917592 mapread 0x004eb0 thru 0x005129 (0x27a bytes) 000703 1764745177.920075 mapwrite 0x022d4c thru 0x0279d7 (0x4c8c bytes) 000704 1764745177.931668 fallocate 0x0244fe thru 0x028a58 (0x455b bytes) 000705 1764745177.933417 read 0x002161 thru 0x00a06e (0x7f0e bytes) 000706 1764745177.939024 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28a59): Mirror 9: 0xf9e0207b (stale) Mirror 11: 0xf9e0207b (stale) Mirror 12: 0xf9e0207b (stale) Mirror 14: 0xf9e0207b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28a59): Mirror 9: 0xf9e0207b (stale) Mirror 11: 0xf9e0207b (stale) Mirror 12: 0xf9e0207b (stale) Mirror 14: 0xf9e0207b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28a59: skipped 000707 1764745178.050252 trunc from 0x028a59 to 0x031610 (0x8bb8 bytes) 000709 1764745178.057119 MIRROR_RESYNC 5 mirrors 000710 1764745178.318889 MIRROR_RESYNC 5 mirrors 000711 1764745178.440975 trunc from 0x004e46 to 0x0131d6 (0xe391 bytes) 000712 1764745178.448478 read 0x012cba thru 0x021443 (0xe78a bytes) 000713 1764745178.451205 read 0x01b189 thru 0x02662e (0xb4a6 bytes) 000714 1764745178.454249 read 0x025e42 thru 0x02d2db (0x749a bytes) 000715 1764745178.454682 write 0x02bb5a thru 0x02ee33 (0x32da bytes) 000716 1764745178.455294 mapread 0x00bb32 thru 0x0153d0 (0x989f bytes) 000721 1764745178.457337 mapread 0x002f71 thru 0x0128df (0xf96f bytes) 000722 1764745178.457956 punch from 0x019139 to 0x02565d (0xc525 bytes) 000723 1764745178.460394 punch from 0x019ac4 to 0x01a680 (0xbbd bytes) 000724 1764745178.462125 write 0x000518 thru 0x00d8c1 (0xd3aa bytes) 000726 1764745178.463414 trunc from 0x0258fb to 0x031610 (0xbd16 bytes) 000727 1764745178.468058 read 0x02a150 thru 0x030813 (0x66c4 bytes) 000728 1764745178.471738 trunc from 0x02ef4f to 0x02f007 (0xb9 bytes) 000729 1764745178.473790 mapread 0x029f01 thru 0x02c166 (0x2266 bytes) 000731 1764745178.476541 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31611): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31611): Mirror 9: 0x957c8c4c (stale) Mirror 11: 0x957c8c4c (stale) Mirror 12: 0x957c8c4c (stale) Mirror 14: 0x957c8c4c (stale) CRC-32 checksum value for chunk [0, 0x31611): Mirror 1: 0x49556300 Mirror 15: 0x49556300 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31611: skipped 000733 1764745178.729965 mapread 0x01aef9 thru 0x02604d (0xb155 bytes) 000735 1764745178.749730 trunc from 0x031611 to 0x00a089 (0x27587 bytes) 000736 1764745178.766943 write 0x02e83c thru 0x03bcd6 (0xd49b bytes) 000737 1764745178.768162 fallocate 0x0314dc thru 0x0351aa (0x3ccf bytes) 000738 1764745178.770312 mapwrite 0x015272 thru 0x01cc46 (0x79d5 bytes) 000739 1764745178.782070 read 0x00868d thru 0x0129b4 (0xa328 bytes) 000741 1764745178.785231 fallocate 0x015053 thru 0x01bfb4 (0x6f62 bytes) 000742 1764745178.788194 MIRROR_RESYNC 6 mirrors 000743 1764745179.122000 mapread 0x03bb28 thru 0x03bcd6 (0x1af bytes) 000744 1764745179.124657 MIRROR_RESYNC 6 mirrors 000745 1764745179.290350 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000746 1764745179.395166 trunc from 0x038492 to 0x0386bb (0x22a bytes) 000747 1764745179.404615 trunc from 0x03bcd7 to 0x037dd7 (0x3eff bytes) 000748 1764745179.410276 mapread 0x0058d6 thru 0x00be5c (0x6587 bytes) 000749 1764745179.414806 punch from 0x002ec8 to 0x00d324 (0xa45d bytes) 000750 1764745179.416312 punch from 0x00eaff to 0x00ee2d (0x32f bytes) 000751 1764745179.417715 punch from 0x01341b to 0x01c5dc (0x91c2 bytes) 000752 1764745179.419172 trunc from 0x007846 to 0x00c306 (0x4ac1 bytes) 000753 1764745179.420878 trunc from 0x037dd8 to 0x0280b1 (0xfd26 bytes) 000754 1764745179.424975 trunc from 0x0280b2 to 0x0298a2 (0x17f1 bytes) 000755 1764745179.428682 trunc from 0x0298a3 to 0x031f69 (0x86c7 bytes) 000756 1764745179.432630 mapread 0x029286 thru 0x02dbb0 (0x492b bytes) 000757 1764745179.435415 read 0x01106a thru 0x01dec7 (0xce5e bytes) 000758 1764745179.437978 mapread 0x027928 thru 0x030324 (0x89fd bytes) 000759 1764745179.438650 write 0x00feca thru 0x01cc86 (0xcdbd bytes) 000760 1764745179.439592 trunc from 0x006038 to 0x010a88 (0xaa51 bytes) 000761 1764745179.443524 trunc from 0x031f6a to 0x02bba0 (0x63c9 bytes) 000762 1764745179.447479 mapwrite 0x02ad72 thru 0x03a8d8 (0xfb67 bytes) 000763 1764745179.464920 trunc from 0x03a8d9 to 0x011fef (0x288e9 bytes) 000764 1764745179.469326 read 0x00c273 thru 0x011fef (0x5d7d bytes) 000765 1764745179.472004 punch from 0x006b56 to 0x0076f9 (0xba4 bytes) 000766 1764745179.473775 mapwrite 0x00b659 thru 0x00c084 (0xa2c bytes) 000767 1764745179.478091 mapread 0x00e0ee thru 0x00e4b4 (0x3c7 bytes) 000768 1764745179.479891 MIRROR_RESYNC 5 mirrors 000769 1764745179.743154 trunc from 0x011ff0 to 0x007f6b (0xa084 bytes) 000770 1764745179.760950 trunc from 0x00062a to 0x007099 (0x6a70 bytes) 000772 1764745179.763245 fallocate 0x03670f thru 0x03ffff (0x98f1 bytes) 000774 1764745179.765470 mapwrite 0x015f26 thru 0x022272 (0xc34d bytes) 000775 1764745179.785707 mapwrite 0x03ff2b thru 0x03ffff (0xd5 bytes) 000777 1764745179.796342 fallocate 0x00cea1 thru 0x00d9b7 (0xb17 bytes) 000778 1764745179.798119 write 0x025290 thru 0x02f202 (0x9f73 bytes) 000779 1764745179.801582 trunc from 0x040000 to 0x0363b6 (0x9c49 bytes) 000780 1764745179.806778 fallocate 0x0000be thru 0x00e999 (0xe8dc bytes) 000781 1764745179.808630 trunc from 0x0363b7 to 0x00b902 (0x2aab4 bytes) 000782 1764745179.814586 MIRROR_RESYNC 5 mirrors 000783 1764745180.013467 write 0x00677e thru 0x00c05f (0x58e2 bytes) 000785 1764745180.022488 trunc from 0x00c060 to 0x019bb4 (0xdb55 bytes) 000786 1764745180.031204 mapwrite 0x02ad09 thru 0x0329d0 (0x7cc8 bytes) 000787 1764745180.044827 fallocate 0x00c731 thru 0x00ccb0 (0x580 bytes) 000788 1764745180.046630 trunc from 0x005c5d to 0x00badd (0x5e81 bytes) 000789 1764745180.048193 punch from 0x018e11 to 0x01c764 (0x3954 bytes) 000791 1764745180.050350 trunc from 0x029caf to 0x02aa24 (0xd76 bytes) 000792 1764745180.052230 punch from 0x0291ca to 0x031944 (0x877b bytes) 000793 1764745180.054351 write 0x000aa3 thru 0x00fce8 (0xf246 bytes) 000794 1764745180.060856 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x329d1): Mirror 12: 0x67bef73e (stale) Mirror 14: 0x67bef73e (stale) Mirror 15: 0x67bef73e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329d1): Mirror 12: 0x67bef73e (stale) Mirror 14: 0x67bef73e (stale) Mirror 15: 0x67bef73e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329d1: skipped 000796 1764745180.172568 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x329d1): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329d1): Mirror 12: 0x67bef73e (stale) Mirror 14: 0x67bef73e (stale) Mirror 15: 0x67bef73e (stale) CRC-32 checksum value for chunk [0, 0x329d1): Mirror 1: 0x6a436469 Mirror 16: 0x6a436469 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329d1: skipped 000797 1764745180.435757 read 0x01a556 thru 0x029422 (0xeecd bytes) 000798 1764745180.441021 fallocate 0x036e25 thru 0x03ffff (0x91db bytes) 000799 1764745180.454490 mapwrite 0x02749b thru 0x0371f6 (0xfd5c bytes) 000801 1764745180.484195 trunc from 0x023515 to 0x02bfa5 (0x8a91 bytes) 000802 1764745180.486991 trunc from 0x02a94d to 0x031053 (0x6707 bytes) 000803 1764745180.489516 mapwrite 0x024274 thru 0x02b2d5 (0x7062 bytes) 000804 1764745180.506212 punch from 0x0184c2 to 0x021ce3 (0x9822 bytes) 000805 1764745180.508065 punch from 0x0263d4 to 0x02bf8f (0x5bbc bytes) 000806 1764745180.509944 punch from 0x01a439 to 0x01eec3 (0x4a8b bytes) 000807 1764745180.511522 mapread 0x02899f thru 0x02ec14 (0x6276 bytes) 000808 1764745180.514969 mapwrite 0x00903b thru 0x0091d7 (0x19d bytes) 000809 1764745180.521739 fallocate 0x014cc4 thru 0x01ecc7 (0xa004 bytes) 000810 1764745180.523678 mapread 0x035da6 thru 0x0371f6 (0x1451 bytes) 000811 1764745180.524363 trunc from 0x02101c to 0x02eb81 (0xdb66 bytes) 000812 1764745180.526591 fallocate 0x00bf00 thru 0x017dfb (0xbefc bytes) 000814 1764745180.528564 punch from 0x001d56 to 0x010308 (0xe5b3 bytes) 000815 1764745180.530393 read 0x012e65 thru 0x014811 (0x19ad bytes) 000816 1764745180.534648 write 0x00ee35 thru 0x011888 (0x2a54 bytes) 000817 1764745180.535621 trunc from 0x027492 to 0x0348d6 (0xd445 bytes) 000818 1764745180.539870 MIRROR_RESYNC 5 mirrors 000819 1764745180.841279 write 0x02ed59 thru 0x0321a6 (0x344e bytes) 000821 1764745180.853447 read 0x023ef8 thru 0x024f12 (0x101b bytes) 000822 1764745180.855503 trunc from 0x0371f7 to 0x03937a (0x2184 bytes) 000823 1764745180.865039 trunc from 0x029aae to 0x02edba (0x530d bytes) 000824 1764745180.868974 read 0x014475 thru 0x01ab69 (0x66f5 bytes) 000825 1764745180.873497 MIRROR_RESYNC 5 mirrors 000826 1764745181.204388 fallocate 0x02b24e thru 0x02b4ef (0x2a2 bytes) 000828 1764745181.212365 write 0x0010c7 thru 0x00b274 (0xa1ae bytes) 000829 1764745181.215495 mapwrite 0x028892 thru 0x02e6b7 (0x5e26 bytes) 000830 1764745181.223441 mapwrite 0x013c87 thru 0x01a5ed (0x6967 bytes) 000831 1764745181.236992 trunc from 0x03937b to 0x0213d2 (0x17fa8 bytes) 000832 1764745181.242439 trunc from 0x0213d3 to 0x00d1ac (0x14226 bytes) 000833 1764745181.246294 mapread 0x00bd16 thru 0x00d1ac (0x1497 bytes) 000834 1764745181.246870 punch from 0x008269 to 0x00d1ac (0x4f44 bytes) 000835 1764745181.248422 trunc from 0x00d1ad to 0x029cb1 (0x1cb05 bytes) 000836 1764745181.252852 punch from 0x023884 to 0x029cb1 (0x642e bytes) 000837 1764745181.254438 punch from 0x004542 to 0x00c837 (0x82f6 bytes) 000838 1764745181.256175 mapread 0x0030bc thru 0x010f2d (0xde72 bytes) 000839 1764745181.260810 trunc from 0x00e036 to 0x016177 (0x8142 bytes) 000840 1764745181.262600 read 0x00a40c thru 0x00ac37 (0x82c bytes) 000841 1764745181.263177 write 0x028d9c thru 0x033777 (0xa9dc bytes) 000842 1764745181.266331 fallocate 0x02fc87 thru 0x036740 (0x6aba bytes) 000843 1764745181.268478 punch from 0x01a90c to 0x02a627 (0xfd1c bytes) 000844 1764745181.273506 punch from 0x0241d8 to 0x02731b (0x3144 bytes) 000845 1764745181.276535 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36741): Mirror 12: 0xcec00956 (stale) Mirror 14: 0xcec00956 (stale) Mirror 16: 0xcec00956 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36741): Mirror 12: 0xcec00956 (stale) Mirror 14: 0xcec00956 (stale) Mirror 16: 0xcec00956 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36741: skipped 000846 1764745181.360225 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36741): Mirror 14: 0xcec00956 (stale) Mirror 16: 0xcec00956 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36741): Mirror 14: 0xcec00956 (stale) Mirror 16: 0xcec00956 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36741: skipped 000847 1764745181.434929 trunc from 0x036741 to 0x00996a (0x2cdd6 bytes) 000849 1764745181.440477 punch from 0x002391 to 0x00996a (0x75da bytes) 000852 1764745181.442778 mapwrite 0x038785 thru 0x03ffff (0x787b bytes) 000853 1764745181.456461 fallocate 0x01f15f thru 0x025647 (0x64e9 bytes) 000854 1764745181.458455 trunc from 0x040000 to 0x014bba (0x2b445 bytes) 000855 1764745181.462427 mapread 0x007eb5 thru 0x00bc15 (0x3d61 bytes) 000856 1764745181.465096 mapwrite 0x01d08f thru 0x0211b5 (0x4127 bytes) 000857 1764745181.476572 write 0x02aedf thru 0x038631 (0xd753 bytes) 000859 1764745181.477964 read 0x027147 thru 0x029403 (0x22bd bytes) 000860 1764745181.479639 mapwrite 0x03a0f8 thru 0x03ffff (0x5f08 bytes) 000861 1764745181.491257 fallocate 0x030257 thru 0x03a2d7 (0xa081 bytes) 000862 1764745181.494177 MIRROR_RESYNC 3 mirrors 000863 1764745181.698206 fallocate 0x01d1e9 thru 0x0271e4 (0x9ffc bytes) 000864 1764745181.706286 trunc from 0x040000 to 0x023e8b (0x1c174 bytes) 000865 1764745181.714598 read 0x0009ce thru 0x008089 (0x76bc bytes) 000866 1764745181.716944 fallocate 0x0337bb thru 0x03f2ce (0xbb14 bytes) 000867 1764745181.718489 punch from 0x011b1b to 0x01bb54 (0xa03a bytes) 000868 1764745181.719954 read 0x008a3e thru 0x00d523 (0x4ae6 bytes) 000870 1764745181.722983 fallocate 0x00a915 thru 0x01428a (0x9976 bytes) 000871 1764745181.724846 mapread 0x023d3f thru 0x023e8b (0x14d bytes) 000873 1764745181.725958 punch from 0x020b85 to 0x023e8b (0x3307 bytes) 000874 1764745181.727858 trunc from 0x023e8c to 0x017123 (0xcd68 bytes) 000875 1764745181.733453 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17124): Mirror 16: 0xefd803ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17124): Mirror 16: 0xefd803ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17124: skipped 000876 1764745181.802332 mapread 0x01568c thru 0x017123 (0x1a98 bytes) 000877 1764745181.807618 read 0x00b561 thru 0x017123 (0xbbc3 bytes) 000878 1764745181.810403 mapread 0x004bcc thru 0x008f0c (0x4341 bytes) 000879 1764745181.812937 trunc from 0x014c32 to 0x017123 (0x24f2 bytes) 000880 1764745181.814427 trunc from 0x014f40 to 0x017123 (0x21e4 bytes) 000881 1764745181.816966 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) 000882 1764745181.848314 punch from 0x016317 to 0x017123 (0xe0d bytes) 000883 1764745181.849671 trunc from 0x001450 to 0x00df09 (0xcaba bytes) 000884 1764745181.851126 write 0x03a2c0 thru 0x03ffff (0x5d40 bytes) 000885 1764745181.852979 fallocate 0x01ed34 thru 0x0242fc (0x55c9 bytes) 000887 1764745181.855210 trunc from 0x014ed6 to 0x02143c (0xc567 bytes) 000888 1764745181.856572 trunc from 0x040000 to 0x038e18 (0x71e7 bytes) 000889 1764745181.860339 write 0x0081df thru 0x00d8dd (0x56ff bytes) 000890 1764745181.863150 read 0x00b3d5 thru 0x0113e2 (0x600e bytes) 000891 1764745181.864712 trunc from 0x038e19 to 0x000db7 (0x38061 bytes) 000892 1764745181.868739 mapread 0x0000b1 thru 0x000db7 (0xd07 bytes) 000893 1764745181.870493 mapread 0x00000a thru 0x000db7 (0xdae bytes) 000894 1764745181.870951 read 0x0008d6 thru 0x000db7 (0x4e2 bytes) 000895 1764745181.871245 fallocate 0x010e12 thru 0x013c12 (0x2e01 bytes) 000896 1764745181.872928 trunc from 0x013c13 to 0x02c689 (0x18a77 bytes) 000898 1764745181.876890 fallocate 0x039868 thru 0x03ffff (0x6798 bytes) 000900 1764745181.878747 punch from 0x01f119 to 0x02be6e (0xcd56 bytes) 000901 1764745181.879970 mapread 0x014f1d thru 0x01a7a5 (0x5889 bytes) 000902 1764745181.883409 read 0x0014cc thru 0x00907c (0x7bb1 bytes) 000903 1764745181.883961 trunc from 0x02c68a to 0x00eb0e (0x1db7b bytes) 000904 1764745181.888200 mapwrite 0x024718 thru 0x02a356 (0x5c3f bytes) 000905 1764745181.899079 read 0x01c64e thru 0x020024 (0x39d7 bytes) 000906 1764745181.899484 fallocate 0x02c5c3 thru 0x02cdaf (0x7ed bytes) 000907 1764745181.901117 trunc from 0x02cdb0 to 0x00f579 (0x1d836 bytes) 000908 1764745181.904991 trunc from 0x00f57a to 0x0278a6 (0x1832d bytes) 000910 1764745181.908870 mapwrite 0x00113a thru 0x0103de (0xf2a5 bytes) 000911 1764745181.918255 punch from 0x01ebd9 to 0x022c9a (0x40c2 bytes) 000912 1764745181.919935 trunc from 0x0278a7 to 0x0063c1 (0x214e5 bytes) 000913 1764745181.924253 trunc from 0x003f9f to 0x0063c1 (0x2423 bytes) 000914 1764745181.925909 mapread 0x00620a thru 0x0063c1 (0x1b8 bytes) 000915 1764745181.927959 trunc from 0x0063c2 to 0x02ca52 (0x26691 bytes) 000916 1764745181.932050 mapwrite 0x014bf1 thru 0x01952b (0x493b bytes) 000917 1764745181.944525 fallocate 0x027708 thru 0x02af7f (0x3878 bytes) 000918 1764745181.945999 mapread 0x01ec6d thru 0x027cf8 (0x908c bytes) 000919 1764745181.957271 trunc from 0x01eb4b to 0x02601e (0x74d4 bytes) 000920 1764745181.958636 mapwrite 0x03ce1b thru 0x03ffff (0x31e5 bytes) 000921 1764745181.969092 mapwrite 0x02e889 thru 0x03b03a (0xc7b2 bytes) 000922 1764745181.975171 trunc from 0x040000 to 0x023dc9 (0x1c236 bytes) 000923 1764745181.978829 trunc from 0x0177a8 to 0x023dc9 (0xc622 bytes) 000925 1764745181.980713 read 0x006363 thru 0x015203 (0xeea1 bytes) 000926 1764745181.983314 trunc from 0x023dca to 0x0237a1 (0x628 bytes) 000927 1764745181.987750 MIRROR_EXTEND to 2 mirrors 000928 1764745182.164083 trunc from 0x0237a2 to 0x02d587 (0x9de6 bytes) 000929 1764745182.175329 punch from 0x014b02 to 0x0236e6 (0xebe5 bytes) 000930 1764745182.176798 fallocate 0x01158b thru 0x015877 (0x42ed bytes) 000932 1764745182.178416 mapread 0x004124 thru 0x0123d9 (0xe2b6 bytes) 000933 1764745182.186031 mapread 0x005fe1 thru 0x012862 (0xc882 bytes) 000934 1764745182.186544 mapread 0x0075c5 thru 0x00b112 (0x3b4e bytes) 000935 1764745182.187021 write 0x00b67e thru 0x018d22 (0xd6a5 bytes) 000936 1764745182.189143 punch from 0x00e8f4 to 0x018644 (0x9d51 bytes) 000937 1764745182.192519 mapread 0x01bcf3 thru 0x01d69e (0x19ac bytes) 000938 1764745182.195071 punch from 0x022339 to 0x023b4a (0x1812 bytes) 000939 1764745182.196275 trunc from 0x02d588 to 0x01ab28 (0x12a5f bytes) 000940 1764745182.201003 write 0x00674c thru 0x00a0e3 (0x3998 bytes) 000942 1764745182.201973 fallocate 0x000379 thru 0x00845b (0x80e3 bytes) 000943 1764745182.203429 punch from 0x0027d9 to 0x00cf43 (0xa76b bytes) 000944 1764745182.206746 mapread 0x0107bc thru 0x01ab28 (0xa36d bytes) 000945 1764745182.211551 mapread 0x012222 thru 0x01a580 (0x835f bytes) 000946 1764745182.212129 fallocate 0x0195d7 thru 0x0273fc (0xde26 bytes) 000947 1764745182.213946 mapread 0x019767 thru 0x01ab28 (0x13c2 bytes) 000950 1764745182.214971 trunc from 0x01ab29 to 0x0223e1 (0x78b9 bytes) 000951 1764745182.219026 punch from 0x01b53e to 0x0223e1 (0x6ea4 bytes) 000952 1764745182.220809 write 0x006ec5 thru 0x00e873 (0x79af bytes) 000953 1764745182.224111 punch from 0x0205e1 to 0x020f0c (0x92c bytes) 000954 1764745182.225862 mapwrite 0x038bb3 thru 0x03ffff (0x744d bytes) 000955 1764745182.238632 trunc from 0x01992c to 0x01fa69 (0x613e bytes) 000956 1764745182.240403 fallocate 0x00359f thru 0x00d937 (0xa399 bytes) 000957 1764745182.242202 trunc from 0x0225ac to 0x025bbf (0x3614 bytes) 000958 1764745182.243644 write 0x026284 thru 0x026718 (0x495 bytes) 000959 1764745182.246152 read 0x0108ac thru 0x019eae (0x9603 bytes) 000960 1764745182.247934 write 0x037cf5 thru 0x03a84f (0x2b5b bytes) 000961 1764745182.248743 punch from 0x038b8d to 0x03f6e4 (0x6b58 bytes) 000962 1764745182.254644 MIRROR_RESYNC 2 mirrors 000964 1764745182.453316 MIRROR_RESYNC 2 mirrors 000965 1764745182.548204 mapwrite 0x0123db thru 0x01ece2 (0xc908 bytes) 000966 1764745182.586814 read 0x02c616 thru 0x035170 (0x8b5b bytes) 000967 1764745182.589535 mapread 0x019f78 thru 0x0240bc (0xa145 bytes) 000968 1764745182.595992 trunc from 0x0181be to 0x019764 (0x15a7 bytes) 000969 1764745182.597857 read 0x036ad8 thru 0x037d24 (0x124d bytes) 000970 1764745182.600722 trunc from 0x040000 to 0x0362fa (0x9d05 bytes) 000972 1764745182.611160 trunc from 0x0362fb to 0x035421 (0xed9 bytes) 000973 1764745182.615233 mapread 0x027987 thru 0x032793 (0xae0d bytes) 000974 1764745182.616018 trunc from 0x035422 to 0x021495 (0x13f8c bytes) 000975 1764745182.620427 read 0x00bf59 thru 0x00dbf2 (0x1c9a bytes) 000976 1764745182.620996 trunc from 0x021496 to 0x034eca (0x13a35 bytes) 000977 1764745182.625708 write 0x01befd thru 0x01c0e3 (0x1e7 bytes) 000979 1764745182.627049 punch from 0x02090f to 0x02575c (0x4e4e bytes) 000980 1764745182.629198 punch from 0x00d198 to 0x010190 (0x2ff9 bytes) 000981 1764745182.631175 trunc from 0x034ecb to 0x03260f (0x28bb bytes) 000982 1764745182.635684 mapwrite 0x03e777 thru 0x03ffff (0x1889 bytes) 000983 1764745182.648657 mapread 0x018401 thru 0x020394 (0x7f94 bytes) 000984 1764745182.649377 mapwrite 0x00c4ec thru 0x014e1f (0x8934 bytes) 000985 1764745182.657255 trunc from 0x035804 to 0x03ffff (0xa7fc bytes) 000986 1764745182.660980 MIRROR_RESYNC 2 mirrors 000987 1764745182.817698 mapwrite 0x025a97 thru 0x02f523 (0x9a8d bytes) 000988 1764745182.848951 mapwrite 0x03b091 thru 0x03ffff (0x4f6f bytes) 000989 1764745182.856227 read 0x00a06b thru 0x01377d (0x9713 bytes) 000990 1764745182.858705 mapwrite 0x02718d thru 0x035145 (0xdfb9 bytes) 000991 1764745182.870064 mapwrite 0x03af15 thru 0x03ffff (0x50eb bytes) 000992 1764745182.875018 trunc from 0x03d4ee to 0x03ffff (0x2b12 bytes) 000993 1764745182.876388 mapwrite 0x017b94 thru 0x02754f (0xf9bc bytes) 000994 1764745182.884940 mapread 0x018b09 thru 0x0262c5 (0xd7bd bytes) 000995 1764745182.885518 trunc from 0x040000 to 0x00645d (0x39ba2 bytes) 000996 1764745182.895666 mapread 0x002935 thru 0x00645d (0x3b29 bytes) 000998 1764745182.898478 read 0x003aca thru 0x004ca6 (0x11dd bytes) 000999 1764745182.898887 mapwrite 0x01d49c thru 0x0211a5 (0x3d0a bytes) 001000 1764745182.911324 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x717:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 71: check mirror extend parent fid ==== 01:59:44 (1764745184) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x750:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 71: check append on FLR file ========== 01:59:47 (1764745187) /mnt/lustre/f71.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0529171 s, 39.6 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314851 s, 33.3 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f71.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 01:59:50 (1764745190) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6408448 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8233952 .. Extending file size to 7543744 .. failed Extending file size to 7996864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8562272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8773792 .. Extending file size to 8889312 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9178784 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8999200 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9170816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9202368 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 63819 63820 63822 63824 63825 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:00:55 (1764745255) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3463008 .. Extending file size to 2416512 .. done Extending file size to 4159936 .. Extending file size to 8139712 .. Extending file size to 4048576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8452288 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8853568 .. Extending file size to 8914880 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9189600 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9196992 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 67770 67771 67773 67775 67777 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:01:59 (1764745319) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2564 1268740 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2984 1268320 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25760 3581260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17912 3589108 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43672 7170368 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2564 1268740 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2984 1268320 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25760 3581260 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17912 3589108 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43672 7170368 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0412692 s, 50.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0562916 s, 37.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x222f8aa5 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:02:05 (1764745325) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0381737 s, 54.9 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 02:02:08 (1764745328) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:02:10 (1764745330) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0266869 s, 78.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xab7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xab8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xab9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:02:13 (1764745333) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0253677 s, 82.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xab9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xaba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xab8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xab9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xaba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xabb:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:02:16 (1764745336) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0290623 s, 72.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:02:40 (1764745360) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210776 s, 49.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:03:13 (1764745393) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0276951 s, 75.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:03:49 (1764745429) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg140-server: Pool lustre.test_204f created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.027911 s, 75.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg140-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:04:37 (1764745477) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:04:40 (1764745480) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:04:42 (1764745482) oleg140-server: Pool lustre.test_206 created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg140-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:04:57 (1764745497) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259902 s, 40.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138799 s, 75.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1507f616 Mirror 3: 0x1507f616 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:05:00 (1764745500) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:05:01 (1764745501) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:05:02 (1764745502) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0134398 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1720000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1350000 Read this (4 bytes): BBBB 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:05:18 (1764745518) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00304588 s, 2.7 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:05:20 (1764745520) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136401 s, 76.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-03 02:05:21.000000000 -0500 Modify: 2025-12-03 02:05:21.000000000 -0500 Change: 2025-12-03 02:05:21.000000000 -0500 Birth: 2025-12-03 02:05:22.000000000 -0500 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:05:23 (1764745523) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016508 s, 63.5 MB/s oleg140-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg140-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg140-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:05:38 (1764745538) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0399477 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00551904 s, 742 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 3 02:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 3 02:05 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:05:40 (1764745540) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:05:43 (1764745543) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 822 sec ============ 02:05:46 (1764745546) === sanity-flr: start cleanup 02:05:46 (1764745546) === === sanity-flr: finish cleanup 02:05:48 (1764745548) ===