-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 15:08:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg202-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:08:57 (1786561737) === oleg202-client.virtnet: executing check_config_client /mnt/lustre oleg202-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg202-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92d68c35b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92d68c35b800.idle_timeout=debug disable quota as required oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:09:13 (1786561753) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 15:09:14 (1786561754) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:09:22 (1786561762) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:09:23 (1786561763) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:09:24 (1786561764) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:09:34 (1786561774) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:09:37 (1786561777) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 15:09:39 (1786561779) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92d68c35b800.stats=clear osc.lustre-OST0001-osc-ffff92d68c35b800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:09:45 (1786561785) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:09:51 (1786561791) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:09:58 (1786561798) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:10:01 (1786561801) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created error: lfs extend: layout is NULL /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000402 lmm_object_id: 0x6 lmm_fid: [0x240000402:0x6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xf:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000402 lmm_object_id: 0x6 lmm_fid: [0x240000402:0x6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x10:0x0] } sanity-flr test_0m: @@@@@@ FAIL: failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:899:test_0m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:909:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0m.*.1786561805.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.debug_log.oleg202-server.1786561805.log.tzW32c" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.dmesg.oleg202-server.1786561805.log.DMHrEA" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 0m (10s) == sanity-flr test 0n: lfs mirror extend with --no-resync option ========================================================== 15:10:11 (1786561811) striped dir -i0 -c2 -H all_char /mnt/lustre/d0n.sanity-flr 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.620436 s, 18.6 MB/s - original checksum: dad7d1963d2f9546685f6934ebb7b50c - extend 1 mirror check the 2nd mirror - mirror 2 checksum: dad7d1963d2f9546685f6934ebb7b50c - extend 2 more mirrors check the stale 3rd mirror - mirror 3 checksum: d41d8cd98f00b204e9800998ecf8427e - check the stale 4th mirror - mirror 4 checksum: d41d8cd98f00b204e9800998ecf8427e - resync the file check the 3rd mirror after resync - mirror 3 checksum after resync: dad7d1963d2f9546685f6934ebb7b50c - check the 4th mirror after resync - mirror 4 checksum after resync: dad7d1963d2f9546685f6934ebb7b50c - PASS 0n (24s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:10:35 (1786561835) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x12:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 15:10:39 (1786561839) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:10:43 (1786561843) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:10:46 (1786561846) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:10:50 (1786561850) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:10:55 (1786561855) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:10:58 (1786561858) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.360763 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053689 s, 19.5 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:11:03 (1786561863) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.201834 s, 67.5 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:11:07 (1786561867) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:11:11 (1786561871) Stopping /mnt/lustre-ost1 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg202-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg202-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (30s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:11:41 (1786561901) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0654401 s, 48.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg202-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:11:57 (1786561917) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg202-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg202-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:12:33 (1786561953) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48858 s, 35.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 15:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.456s user 0m0.001s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.397s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg202-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg202-server oleg202-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.411s user 0m0.001s sys 0m0.171s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg202-server: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg202-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d68c35b800.ost_server_uuid in FULL state after 0 sec PASS 33b (41s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:13:14 (1786561994) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:13:16 (1786561996) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:13:18 (1786561998) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 15:13:19 (1786561999) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 15:13:23 (1786562003) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.226546 s, 55.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:13:29 (1786562009) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239944 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352062 s, 29.8 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:13:33 (1786562013) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:13:38 (1786562018) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000860058 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122471 s, 836 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000975561 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000869878 s, 1.2 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 15:13:43 (1786562023) create test files with size 15758162 .. 43d2c254d5b9d6b185a04067e3c7ce91 - 2c4fe317e0a04d289397dd8db8484747 - b1d0c3e0d4db18d5215fda35c3da56c7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg202-client.virtnet /mnt/lustre (opts:) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 15:13:54 (1786562034) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:14:01 (1786562041) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:14:04 (1786562044) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383351 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139021 s, 75.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb1:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd1:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 15:14:08 (1786562048) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0573494 s, 36.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.143944 s, 29.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.08263 s, 25.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 15:14:16 (1786562056) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 15:14:18 (1786562058) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:14:20 (1786562060) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268927 s, 39.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 15:14:23 (1786562063) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171022 s, 18.4 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:14:29 (1786562069) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:14:34 (1786562074) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.141697 s, 74.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg202-client.virtnet /mnt/lustre (opts:) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:14:43 (1786562083) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff92d6c0f22000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.165815 s, 63.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg202-client.virtnet /mnt/lustre (opts:) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff92d6834ce800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:14:50 (1786562090) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:15:14 (1786562114) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (24s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:15:38 (1786562138) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 15:15:40 (1786562140) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.164225 s, 19.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0769315 s, 40.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x53da897a (nosync,stale) Mirror 2: 0x53da897a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8c481903cfd555469909a8df8034b0dc - mirror 1 checksum: a928f8c5b0af564b27f7b4d2c1b8c9d3 - mirror 2 checksum: 8c481903cfd555469909a8df8034b0dc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:15:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:15:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8c481903cfd555469909a8df8034b0dc - mirror 1 checksum: a928f8c5b0af564b27f7b4d2c1b8c9d3 - mirror 2 checksum: a928f8c5b0af564b27f7b4d2c1b8c9d3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:15:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:15:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 48 (7s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:15:47 (1786562147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187935 s, 55.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 398336.. 399359: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:15:52 (1786562152) Failing mds1 on oleg202-server Stopping /mnt/lustre-mds1 (opts:) on oleg202-server 15:16:01 (1786562161) shut down facet: mds1 facet_host: oleg202-server facet_failover_host: oleg202-server Failover mds1 to oleg202-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-MDT0000 15:16:19 (1786562179) targets are mounted 15:16:19 (1786562179) facet_failover done oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238543 s, 44.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:16:29 (1786562189) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.030522 s, 671 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0102902 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:16:34 (1786562194) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.018953 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00573422 s, 3.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:16:39 (1786562199) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00485035 s, 3.4 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:16:44 (1786562204) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0271612 s, 754 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:16:48 (1786562208) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00368837 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0230993 s, 177 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:16:52 (1786562212) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:17:05 (1786562225) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:17:27 (1786562247) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:17:46 (1786562266) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:18:05 (1786562285) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 15:18:09 (1786562289) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 59940 mirror split pid 59941 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 59940 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 59941 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 15:19:12 (1786562352) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2353 000001 1786562353.275788 mapwrite 0x02c57c thru 0x0371ee (0xac73 bytes) fallocating to largest ever: 0x26f05 000002 1786562353.327530 fallocate 0x02517e thru 0x026f04 (0x1d87 bytes) 000003 1786562353.331111 mapwrite 0x02a1ef thru 0x03a0e7 (0xfef9 bytes) 000004 1786562353.357382 read 0x006230 thru 0x00b151 (0x4f22 bytes) 000005 1786562353.362147 mapread 0x011204 thru 0x016738 (0x5535 bytes) 000006 1786562353.368781 MIRROR_EXTEND to 2 mirrors 000007 1786562353.684219 mapread 0x031c1a thru 0x03a0e7 (0x84ce bytes) 000008 1786562353.703417 mapwrite 0x019250 thru 0x024711 (0xb4c2 bytes) 000009 1786562353.775158 write 0x02937c thru 0x0363be (0xd043 bytes) 000011 1786562353.782035 mapwrite 0x004372 thru 0x0102d1 (0xbf60 bytes) 000012 1786562353.805065 write 0x00efef thru 0x018ab4 (0x9ac6 bytes) 000013 1786562353.806536 punch from 0x036555 to 0x03a0e7 (0x3b93 bytes) fallocating to largest ever: 0x40000 000014 1786562353.811487 fallocate 0x03e4c0 thru 0x03ffff (0x1b40 bytes) 000015 1786562353.815397 trunc from 0x03a0e8 to 0x03ab40 (0xa59 bytes) 000017 1786562353.839323 punch from 0x026938 to 0x02fb98 (0x9261 bytes) 000018 1786562353.842900 trunc from 0x035d20 to 0x03ab40 (0x4e21 bytes) 000019 1786562353.847274 mapread 0x00e817 thru 0x015848 (0x7032 bytes) 000020 1786562353.848452 trunc from 0x020aa4 to 0x02f796 (0xecf3 bytes) 000021 1786562353.852395 write 0x02bdaf thru 0x0398de (0xdb30 bytes) 000022 1786562353.859123 fallocate 0x02a6cd thru 0x037344 (0xcc78 bytes) 000023 1786562353.862115 read 0x02f4da thru 0x03ab40 (0xb667 bytes) 000024 1786562353.865084 write 0x003e5b thru 0x00b6fa (0x78a0 bytes) 000025 1786562353.866388 mapread 0x01c915 thru 0x02a260 (0xd94c bytes) 000027 1786562353.876634 punch from 0x02af49 to 0x031cb5 (0x6d6d bytes) 000028 1786562353.889281 write 0x02e884 thru 0x033ace (0x524b bytes) 000029 1786562353.895167 trunc from 0x026060 to 0x028393 (0x2334 bytes) 000030 1786562353.899613 mapread 0x014324 thru 0x0221d1 (0xdeae bytes) 000031 1786562353.900852 mapwrite 0x013412 thru 0x01f010 (0xbbff bytes) 000032 1786562353.916488 mapwrite 0x004c9e thru 0x0112f7 (0xc65a bytes) 000033 1786562353.934960 read 0x00a194 thru 0x00e8cc (0x4739 bytes) 000034 1786562353.935578 punch from 0x00c1ed to 0x019cbe (0xdad2 bytes) 000036 1786562353.940549 trunc from 0x00f33c to 0x016bfe (0x78c3 bytes) 000037 1786562353.949664 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000038 1786562354.059806 read 0x023de5 thru 0x02a3c9 (0x65e5 bytes) 000039 1786562354.069275 trunc from 0x03ab41 to 0x01709b (0x23aa5 bytes) 000040 1786562354.090983 fallocate 0x009a7c thru 0x00e433 (0x49b8 bytes) 000041 1786562354.097740 fallocate 0x01bcf6 thru 0x01cf14 (0x121f bytes) 000042 1786562354.105326 trunc from 0x0146d6 to 0x01709b (0x29c6 bytes) 000045 1786562354.110705 trunc from 0x01709c to 0x026c34 (0xfb99 bytes) 000046 1786562354.127810 MIRROR_EXTEND to 2 mirrors 000047 1786562354.473103 read 0x009f17 thru 0x012861 (0x894b bytes) 000048 1786562354.486064 write 0x028e14 thru 0x031bc4 (0x8db1 bytes) 000050 1786562354.506564 trunc from 0x00753c to 0x00ce77 (0x593c bytes) 000051 1786562354.511190 read 0x011bea thru 0x01ef1d (0xd334 bytes) 000052 1786562354.516023 mapread 0x030247 thru 0x03051d (0x2d7 bytes) 000053 1786562354.516720 trunc from 0x031bc5 to 0x016c55 (0x1af6f bytes) 000054 1786562354.535382 write 0x0338ae thru 0x038932 (0x5085 bytes) 000055 1786562354.536910 fallocate 0x017e13 thru 0x026d77 (0xef65 bytes) 000057 1786562354.540512 mapread 0x026680 thru 0x03549d (0xee1e bytes) 000058 1786562354.586406 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38933): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38933): Mirror 2: 0xa6b1bc0a (stale) CRC-32 checksum value for chunk [0, 0x38933): Mirror 1: 0x819b635 Mirror 3: 0x819b635 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38933: skipped 000059 1786562355.067071 read 0x01b32f thru 0x0260a6 (0xad78 bytes) 000060 1786562355.076675 trunc from 0x037be3 to 0x038932 (0xd50 bytes) 000061 1786562355.095575 trunc from 0x038933 to 0x036c38 (0x1cfa bytes) 000063 1786562355.115063 fallocate 0x00ab53 thru 0x011f05 (0x73b3 bytes) 000064 1786562355.118895 mapread 0x028bf9 thru 0x0349f7 (0xbdff bytes) 000065 1786562355.140634 fallocate 0x00ddcf thru 0x00fe94 (0x20c6 bytes) 000066 1786562355.146763 trunc from 0x036c39 to 0x0335cc (0x366c bytes) 000067 1786562355.161243 trunc from 0x029edd to 0x0335cc (0x96f0 bytes) 000069 1786562355.168677 read 0x024ff4 thru 0x02e960 (0x996d bytes) 000070 1786562355.177315 punch from 0x0224f1 to 0x025cbf (0x37cf bytes) 000072 1786562355.185985 write 0x036c7e thru 0x03ffff (0x9382 bytes) 000073 1786562355.188625 punch from 0x02c9e8 to 0x03031a (0x3933 bytes) 000074 1786562355.193030 punch from 0x01b3d1 to 0x02731b (0xbf4b bytes) 000075 1786562355.198975 trunc from 0x016c3e to 0x02247a (0xb83d bytes) 000076 1786562355.203680 fallocate 0x037798 thru 0x03ffff (0x8868 bytes) 000077 1786562355.211474 MIRROR_RESYNC 3 mirrors 000078 1786562355.787015 trunc from 0x0333c4 to 0x036090 (0x2ccd bytes) 000079 1786562355.814092 write 0x01b1f8 thru 0x01b44a (0x253 bytes) 000081 1786562355.818545 mapread 0x008a4e thru 0x00b245 (0x27f8 bytes) 000082 1786562355.835669 read 0x0275f1 thru 0x028435 (0xe45 bytes) 000083 1786562355.839776 mapwrite 0x019763 thru 0x02822f (0xeacd bytes) 000084 1786562355.883055 read 0x0376d7 thru 0x038e23 (0x174d bytes) 000085 1786562355.895475 trunc from 0x03c70d to 0x03ffff (0x38f3 bytes) 000086 1786562355.900200 trunc from 0x040000 to 0x00628d (0x39d72 bytes) 000087 1786562355.932138 trunc from 0x001e51 to 0x00628d (0x443d bytes) 000088 1786562355.936993 trunc from 0x00628e to 0x00357f (0x2d0e bytes) 000089 1786562355.951193 read 0x000b9b thru 0x00357f (0x29e5 bytes) 000090 1786562355.955203 mapread 0x0029a3 thru 0x00357f (0xbdd bytes) 000091 1786562355.956290 fallocate 0x039aa3 thru 0x03e1a7 (0x4705 bytes) 000092 1786562355.962462 trunc from 0x03e1a8 to 0x03b875 (0x2932 bytes) 000094 1786562355.974783 write 0x00e2ff thru 0x01c45f (0xe161 bytes) 000095 1786562355.985674 write 0x00cd3e thru 0x013ba6 (0x6e69 bytes) 000096 1786562355.986667 mapwrite 0x017035 thru 0x019eb0 (0x2e7c bytes) 000097 1786562356.007994 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b876): Mirror 3: 0x8e2ca87b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b876): Mirror 3: 0x8e2ca87b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b876: skipped 000098 1786562356.187024 write 0x024994 thru 0x032e32 (0xe49f bytes) 000099 1786562356.205051 write 0x00b6fe thru 0x00f3e4 (0x3ce7 bytes) 000100 1786562356.224162 trunc from 0x008c9d to 0x016a31 (0xdd95 bytes) 000101 1786562356.237386 write 0x016bdd thru 0x01b9f8 (0x4e1c bytes) 000102 1786562356.247751 read 0x021979 thru 0x0242e3 (0x296b bytes) 000103 1786562356.253267 fallocate 0x01346e thru 0x013a05 (0x598 bytes) 000104 1786562356.262193 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b876): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b876): Mirror 3: 0x8e2ca87b (stale) CRC-32 checksum value for chunk [0, 0x3b876): Mirror 1: 0x32956ad0 Mirror 4: 0x32956ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b876: skipped 000105 1786562356.698717 trunc from 0x004452 to 0x00b1d1 (0x6d80 bytes) 000106 1786562356.734887 mapwrite 0x02295d thru 0x02925c (0x6900 bytes) 000107 1786562356.777787 trunc from 0x018c21 to 0x01dd1e (0x50fe bytes) 000108 1786562356.783760 trunc from 0x00f7b5 to 0x01069e (0xeea bytes) 000109 1786562356.790672 mapwrite 0x01d129 thru 0x01ee93 (0x1d6b bytes) 000110 1786562356.813551 punch from 0x023319 to 0x02fc67 (0xc94f bytes) 000113 1786562356.823986 punch from 0x03240d to 0x03b875 (0x9469 bytes) 000114 1786562356.832062 mapwrite 0x011ca8 thru 0x0139cb (0x1d24 bytes) 000115 1786562356.872108 MIRROR_RESYNC 3 mirrors 000116 1786562357.609969 read 0x0092d6 thru 0x010c28 (0x7953 bytes) 000117 1786562357.615293 mapread 0x0208e6 thru 0x02c845 (0xbf60 bytes) 000118 1786562357.626907 read 0x032f10 thru 0x033895 (0x986 bytes) 000119 1786562357.627298 mapwrite 0x00d689 thru 0x01aa07 (0xd37f bytes) 000120 1786562357.665215 MIRROR_RESYNC 3 mirrors 000121 1786562358.129916 write 0x0146b9 thru 0x01de64 (0x97ac bytes) 000122 1786562358.156607 punch from 0x01ff39 to 0x02d36e (0xd436 bytes) 000124 1786562358.159935 trunc from 0x03b876 to 0x02f329 (0xc54c bytes) 000125 1786562358.170329 fallocate 0x02b4f8 thru 0x031544 (0x604d bytes) 000126 1786562358.178115 write 0x0213ae thru 0x026ed8 (0x5b2b bytes) 000128 1786562358.192987 mapread 0x023092 thru 0x031544 (0xe4b3 bytes) 000129 1786562358.209945 punch from 0x008e15 to 0x017ef8 (0xf0e4 bytes) 000130 1786562358.221650 trunc from 0x006a0c to 0x00a23c (0x3831 bytes) 000131 1786562358.227468 mapwrite 0x0031c8 thru 0x009aa0 (0x68d9 bytes) 000132 1786562358.248552 fallocate 0x03724e thru 0x03ffff (0x8db2 bytes) 000134 1786562358.254523 fallocate 0x01132a thru 0x01fdad (0xea84 bytes) 000135 1786562358.264795 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31545): Mirror 4: 0xf43fe70e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31545): Mirror 4: 0xf43fe70e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31545: skipped 000136 1786562358.442822 trunc from 0x031545 to 0x01c797 (0x14dad bytes) 000137 1786562358.453969 mapwrite 0x038812 thru 0x03f35b (0x6b4a bytes) 000138 1786562358.491918 mapwrite 0x03ef61 thru 0x03ffff (0x109f bytes) 000139 1786562358.511976 fallocate 0x005a8a thru 0x008418 (0x298f bytes) 000140 1786562358.517332 trunc from 0x018580 to 0x01e64f (0x60d0 bytes) 000141 1786562358.523753 mapwrite 0x03ebf9 thru 0x03ffff (0x1407 bytes) 000142 1786562358.538672 trunc from 0x03ec61 to 0x03ffff (0x139f bytes) 000144 1786562358.549144 MIRROR_RESYNC 2 mirrors 000145 1786562359.055063 mapwrite 0x00aef8 thru 0x00d60e (0x2717 bytes) 000146 1786562359.145182 read 0x0287a9 thru 0x02c5e6 (0x3e3e bytes) 000147 1786562359.153493 trunc from 0x040000 to 0x00ab56 (0x354a9 bytes) 000148 1786562359.194329 mapread 0x0039e4 thru 0x00ab56 (0x7173 bytes) 000149 1786562359.205069 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000150 1786562359.407764 fallocate 0x032dfd thru 0x037b48 (0x4d4c bytes) 000151 1786562359.422143 fallocate 0x00ef3b thru 0x01d3c9 (0xe48f bytes) 000152 1786562359.429150 trunc from 0x037b49 to 0x036105 (0x1a43 bytes) 000153 1786562359.452118 trunc from 0x036106 to 0x0342f9 (0x1e0c bytes) 000155 1786562359.480758 trunc from 0x004142 to 0x0059db (0x189a bytes) 000156 1786562359.490415 mapwrite 0x013dbe thru 0x02166d (0xd8b0 bytes) 000159 1786562359.581338 read 0x005349 thru 0x00fdf3 (0xaaab bytes) 000161 1786562359.594434 write 0x0276b7 thru 0x02e1ce (0x6b18 bytes) 000162 1786562359.608510 write 0x0075f8 thru 0x016377 (0xed80 bytes) 000164 1786562359.611664 read 0x00428b thru 0x011669 (0xd3df bytes) 000165 1786562359.621812 trunc from 0x002ad7 to 0x007431 (0x495b bytes) 000166 1786562359.642016 read 0x02d588 thru 0x02d7b3 (0x22c bytes) 000167 1786562359.643035 mapread 0x004b57 thru 0x0058d4 (0xd7e bytes) 000169 1786562359.656931 read 0x034097 thru 0x0342f9 (0x263 bytes) 000170 1786562359.658047 mapwrite 0x00412c thru 0x0074f9 (0x33ce bytes) 000171 1786562359.676104 read 0x016d7a thru 0x01b3cb (0x4652 bytes) 000172 1786562359.678172 read 0x030da0 thru 0x0342f9 (0x355a bytes) 000173 1786562359.679077 mapwrite 0x0397bb thru 0x03ffff (0x6845 bytes) 000174 1786562359.724663 MIRROR_EXTEND to 2 mirrors 000175 1786562360.338513 trunc from 0x01b345 to 0x029206 (0xdec2 bytes) 000176 1786562360.383644 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000177 1786562360.548568 mapread 0x029dbc thru 0x02efd1 (0x5216 bytes) 000180 1786562360.580211 punch from 0x0243b9 to 0x02a504 (0x614c bytes) 000182 1786562360.596548 trunc from 0x040000 to 0x01a9f3 (0x2560c bytes) 000185 1786562360.638126 mapread 0x00ac0a thru 0x01a259 (0xf650 bytes) 000186 1786562360.669528 mapwrite 0x008622 thru 0x01085a (0x8239 bytes) 000187 1786562360.705475 trunc from 0x01a9f4 to 0x02db90 (0x1319d bytes) 000189 1786562360.721638 write 0x03ad09 thru 0x03ffff (0x52f7 bytes) 000190 1786562360.724300 mapwrite 0x01d937 thru 0x01f33c (0x1a06 bytes) 000191 1786562360.757248 mapwrite 0x03bb87 thru 0x03fbab (0x4025 bytes) 000192 1786562360.769971 mapread 0x00d707 thru 0x017a9c (0xa396 bytes) 000193 1786562360.770977 punch from 0x03433f to 0x03ffff (0xbcc1 bytes) 000194 1786562360.775935 read 0x000c6d thru 0x00fc2b (0xefbf bytes) 000195 1786562360.784410 write 0x032506 thru 0x0332f0 (0xdeb bytes) 000196 1786562360.804658 fallocate 0x00145e thru 0x00b564 (0xa107 bytes) 000197 1786562360.817697 MIRROR_EXTEND to 2 mirrors 000198 1786562361.411228 trunc from 0x0341b7 to 0x03627a (0x20c4 bytes) 000199 1786562361.466217 MIRROR_RESYNC 2 mirrors 000200 1786562362.450115 MIRROR_RESYNC 2 mirrors 000201 1786562362.672932 trunc from 0x040000 to 0x035e6a (0xa195 bytes) 000202 1786562362.704090 mapwrite 0x0083e8 thru 0x0177dd (0xf3f6 bytes) 000203 1786562362.749210 punch from 0x015051 to 0x021345 (0xc2f5 bytes) 000204 1786562362.758374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000205 1786562362.915512 mapread 0x008b90 thru 0x00b572 (0x29e3 bytes) 000206 1786562362.948119 read 0x0173f6 thru 0x01cc0d (0x5818 bytes) 000207 1786562362.977036 read 0x01ecdd thru 0x029a3b (0xad5f bytes) 000209 1786562362.994895 write 0x005e73 thru 0x00b3c2 (0x5550 bytes) 000210 1786562363.003639 trunc from 0x035e6b to 0x02b060 (0xae0a bytes) 000211 1786562363.023530 mapread 0x01b6c0 thru 0x02af81 (0xf8c2 bytes) 000214 1786562363.025808 fallocate 0x03fffa thru 0x03ffff (0x6 bytes) 000215 1786562363.029597 mapwrite 0x02b0ab thru 0x034f65 (0x9ebb bytes) 000217 1786562363.123310 trunc from 0x0072ab to 0x013e03 (0xcb59 bytes) 000218 1786562363.131015 punch from 0x021577 to 0x0310b7 (0xfb41 bytes) 000219 1786562363.137541 trunc from 0x0213d1 to 0x021f28 (0xb58 bytes) 000220 1786562363.145342 write 0x011828 thru 0x018d4e (0x7527 bytes) 000224 1786562363.171254 read 0x02ae7b thru 0x034f65 (0xa0eb bytes) 000225 1786562363.183185 mapwrite 0x02773b thru 0x02f574 (0x7e3a bytes) 000227 1786562363.230248 fallocate 0x039f82 thru 0x03ffff (0x607e bytes) 000228 1786562363.238563 fallocate 0x0116be thru 0x011ba1 (0x4e4 bytes) 000230 1786562363.248496 fallocate 0x02d003 thru 0x02f9a1 (0x299f bytes) 000231 1786562363.267402 trunc from 0x0015fe to 0x009eed (0x88f0 bytes) 000232 1786562363.278570 read 0x00dacf thru 0x019901 (0xbe33 bytes) 000233 1786562363.280204 mapwrite 0x0211c1 thru 0x021e3e (0xc7e bytes) 000234 1786562363.298042 fallocate 0x03377b thru 0x037cc7 (0x454d bytes) 000238 1786562363.323140 fallocate 0x0056d8 thru 0x00a405 (0x4d2e bytes) 000239 1786562363.337692 punch from 0x01fe68 to 0x028749 (0x88e2 bytes) 000240 1786562363.343663 trunc from 0x034f66 to 0x010f3d (0x24028 bytes) 000241 1786562363.389375 write 0x0153f8 thru 0x0192b7 (0x3ec0 bytes) 000242 1786562363.401579 write 0x02c5b0 thru 0x02ce39 (0x88a bytes) 000243 1786562363.406284 fallocate 0x00afe4 thru 0x011314 (0x6331 bytes) 000245 1786562363.415374 mapwrite 0x03b1a3 thru 0x03c3f5 (0x1253 bytes) 000246 1786562363.527181 fallocate 0x03b22c thru 0x03ffff (0x4dd4 bytes) 000247 1786562363.541165 trunc from 0x040000 to 0x00887c (0x37783 bytes) 000248 1786562363.628639 MIRROR_EXTEND to 2 mirrors 000250 1786562364.124328 mapread 0x00684b thru 0x00887c (0x2032 bytes) 000251 1786562364.188123 trunc from 0x00887d to 0x014a4b (0xc1cf bytes) 000252 1786562364.236241 trunc from 0x00f16d to 0x014a4b (0x58df bytes) 000253 1786562364.241928 MIRROR_RESYNC 2 mirrors 000254 1786562364.649764 mapread 0x003b1f thru 0x011928 (0xde0a bytes) 000255 1786562364.674165 mapwrite 0x01ab6b thru 0x0223d1 (0x7867 bytes) 000257 1786562364.735322 punch from 0x0043ce to 0x00a6a4 (0x62d7 bytes) 000258 1786562364.740355 punch from 0x0101c1 to 0x011a44 (0x1884 bytes) 000259 1786562364.745139 mapwrite 0x01f443 thru 0x02af9b (0xbb59 bytes) 000260 1786562364.787042 trunc from 0x01c234 to 0x0210c5 (0x4e92 bytes) 000261 1786562364.792308 mapread 0x002757 thru 0x0115df (0xee89 bytes) 000262 1786562364.806090 mapwrite 0x005ed5 thru 0x00a14f (0x427b bytes) 000263 1786562364.822628 write 0x00fb43 thru 0x01dd87 (0xe245 bytes) 000264 1786562364.829481 fallocate 0x00cf88 thru 0x019964 (0xc9dd bytes) 000265 1786562364.837127 mapread 0x022104 thru 0x023e83 (0x1d80 bytes) 000266 1786562364.838904 mapwrite 0x03e7af thru 0x03ea77 (0x2c9 bytes) 000267 1786562364.874667 mapread 0x02d53f thru 0x02f8b9 (0x237b bytes) 000269 1786562364.882382 mapwrite 0x02c5d3 thru 0x02f81f (0x324d bytes) 000270 1786562364.894600 fallocate 0x011f09 thru 0x01cfe6 (0xb0de bytes) 000271 1786562364.898663 mapwrite 0x030aa0 thru 0x039830 (0x8d91 bytes) 000272 1786562364.916932 fallocate 0x0294c5 thru 0x036e64 (0xd9a0 bytes) 000273 1786562364.921298 mapread 0x03a0c6 thru 0x03ea77 (0x49b2 bytes) 000274 1786562364.922661 mapwrite 0x02d851 thru 0x037873 (0xa023 bytes) 000275 1786562364.940633 read 0x013c4c thru 0x023aca (0xfe7f bytes) 000276 1786562364.946110 trunc from 0x02b223 to 0x036fdb (0xbdb9 bytes) 000277 1786562364.951558 mapwrite 0x01d555 thru 0x01d652 (0xfe bytes) 000278 1786562364.961792 mapwrite 0x027fe0 thru 0x037d64 (0xfd85 bytes) 000279 1786562364.996249 trunc from 0x01932e to 0x026577 (0xd24a bytes) 000280 1786562365.001583 trunc from 0x03ea78 to 0x0082a8 (0x367cf bytes) 000281 1786562365.020198 read 0x007e98 thru 0x0082a8 (0x411 bytes) 000282 1786562365.022081 mapread 0x005cb8 thru 0x0082a8 (0x25f1 bytes) 000283 1786562365.023944 fallocate 0x01fed5 thru 0x02bdd4 (0xbf00 bytes) 000284 1786562365.029252 mapread 0x005793 thru 0x0082a8 (0x2b16 bytes) 000285 1786562365.030544 punch from 0x0062f4 to 0x0082a8 (0x1fb5 bytes) 000286 1786562365.039630 MIRROR_RESYNC 2 mirrors 000287 1786562365.610493 trunc from 0x0082a9 to 0x007e46 (0x462 bytes) 000288 1786562365.688285 read 0x0019e4 thru 0x0027c1 (0xdde bytes) 000289 1786562365.694358 trunc from 0x000896 to 0x003d6b (0x34d6 bytes) 000290 1786562365.700518 read 0x00326d thru 0x007e46 (0x4bda bytes) 000291 1786562365.711023 trunc from 0x007e47 to 0x039046 (0x31200 bytes) 000292 1786562365.725143 punch from 0x002cbf to 0x0103ed (0xd72f bytes) 000293 1786562365.730675 trunc from 0x00a400 to 0x016551 (0xc152 bytes) 000294 1786562365.735533 mapwrite 0x014970 thru 0x021304 (0xc995 bytes) 000296 1786562365.775356 trunc from 0x01c70d to 0x02b437 (0xed2b bytes) 000298 1786562365.781991 trunc from 0x00dd3c to 0x01aaf6 (0xcdbb bytes) 000299 1786562365.787394 punch from 0x00e14d to 0x01c590 (0xe444 bytes) 000300 1786562365.793481 fallocate 0x02a017 thru 0x037303 (0xd2ed bytes) 000301 1786562365.800295 mapread 0x020a3e thru 0x02f95f (0xef22 bytes) 000302 1786562365.841428 write 0x007679 thru 0x00c4f4 (0x4e7c bytes) 000303 1786562365.844951 read 0x014ba0 thru 0x019878 (0x4cd9 bytes) 000305 1786562365.855119 trunc from 0x01e39d to 0x023ab4 (0x5718 bytes) 000307 1786562365.858746 write 0x02d3dc thru 0x03a168 (0xcd8d bytes) 000308 1786562365.860029 punch from 0x016650 to 0x01dc99 (0x764a bytes) 000309 1786562365.863340 punch from 0x01da22 to 0x02d6f6 (0xfcd5 bytes) 000311 1786562365.874105 mapread 0x02e6ac thru 0x02e8a6 (0x1fb bytes) 000312 1786562365.874937 punch from 0x013232 to 0x01a2c9 (0x7098 bytes) 000313 1786562365.878238 fallocate 0x0344ac thru 0x039ffd (0x5b52 bytes) 000314 1786562365.881237 trunc from 0x03a169 to 0x01a87a (0x1f8ee bytes) 000315 1786562365.891472 punch from 0x005950 to 0x0074ed (0x1b9e bytes) 000316 1786562365.895983 write 0x02ed47 thru 0x0320d6 (0x3390 bytes) 000317 1786562365.898394 write 0x020752 thru 0x025e5e (0x570d bytes) 000318 1786562365.908103 mapread 0x01a487 thru 0x029130 (0xecaa bytes) 000319 1786562365.958216 fallocate 0x03ab01 thru 0x03ffff (0x54ff bytes) 000320 1786562365.963285 write 0x005e40 thru 0x015034 (0xf1f5 bytes) 000323 1786562365.980523 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x320d7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x320d7): Mirror 2: 0x6c226cf2 (stale) CRC-32 checksum value for chunk [0, 0x320d7): Mirror 1: 0xda691f9c Mirror 3: 0xda691f9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x320d7: skipped 000324 1786562366.661505 read 0x008b05 thru 0x013ef4 (0xb3f0 bytes) 000325 1786562366.685117 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x320d7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x320d7): Mirror 2: 0x6c226cf2 (stale) CRC-32 checksum value for chunk [0, 0x320d7): Mirror 1: 0xda691f9c Mirror 3: 0xda691f9c Mirror 4: 0xda691f9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x320d7: skipped 000326 1786562367.474226 write 0x03c2ac thru 0x03ffff (0x3d54 bytes) 000327 1786562367.520286 punch from 0x00dcfe to 0x0184a5 (0xa7a8 bytes) 000328 1786562367.525545 trunc from 0x03425a to 0x03ffff (0xbda6 bytes) 000329 1786562367.537262 mapread 0x0370d9 thru 0x03ffff (0x8f27 bytes) 000330 1786562367.557783 punch from 0x038686 to 0x03f593 (0x6f0e bytes) 000332 1786562367.562998 mapwrite 0x033363 thru 0x03bba0 (0x883e bytes) 000333 1786562367.591502 punch from 0x0133cc to 0x01633c (0x2f71 bytes) 000334 1786562367.601467 MIRROR_RESYNC 4 mirrors 000335 1786562368.594408 punch from 0x00f211 to 0x017ac9 (0x88b9 bytes) 000336 1786562368.635753 write 0x017206 thru 0x01ca56 (0x5851 bytes) 000337 1786562368.661193 fallocate 0x01d6b1 thru 0x025ca2 (0x85f2 bytes) 000338 1786562368.669361 fallocate 0x03e17e thru 0x03ffff (0x1e82 bytes) 000339 1786562368.676542 trunc from 0x040000 to 0x0077bd (0x38842 bytes) 000341 1786562368.701109 MIRROR_RESYNC 4 mirrors 000343 1786562369.428130 trunc from 0x0077be to 0x01c702 (0x14f45 bytes) 000344 1786562369.477324 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c703): Mirror 2: 0x5371026d (stale) Mirror 4: 0x5371026d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c703): Mirror 2: 0x5371026d (stale) Mirror 4: 0x5371026d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c703: skipped 000345 1786562369.840345 read 0x0119d5 thru 0x01c702 (0xad2e bytes) 000346 1786562369.853102 fallocate 0x004061 thru 0x00e550 (0xa4f0 bytes) 000347 1786562369.860770 fallocate 0x03f1a4 thru 0x03ffff (0xe5c bytes) 000348 1786562369.868758 mapread 0x0038a8 thru 0x00a607 (0x6d60 bytes) 000350 1786562369.903963 fallocate 0x02a8b2 thru 0x0317d3 (0x6f22 bytes) 000351 1786562369.909914 trunc from 0x0030d9 to 0x00b8fa (0x8822 bytes) 000352 1786562369.918227 mapread 0x02f49e thru 0x037fc2 (0x8b25 bytes) 000353 1786562369.938239 trunc from 0x040000 to 0x01c634 (0x239cb bytes) 000354 1786562369.975635 trunc from 0x01c635 to 0x027fa9 (0xb975 bytes) 000355 1786562369.994093 mapread 0x012738 thru 0x01d737 (0xb000 bytes) 000356 1786562370.015236 mapread 0x00d121 thru 0x015560 (0x8440 bytes) 000357 1786562370.016430 read 0x011712 thru 0x01f2f9 (0xdbe8 bytes) 000359 1786562370.020254 read 0x01970e thru 0x0235b2 (0x9ea5 bytes) 000360 1786562370.034970 punch from 0x0109db to 0x013a85 (0x30ab bytes) 000361 1786562370.041160 read 0x02567c thru 0x027fa9 (0x292e bytes) 000362 1786562370.058989 read 0x01ae05 thru 0x01f19d (0x4399 bytes) 000363 1786562370.062093 mapread 0x00acb1 thru 0x00ec6f (0x3fbf bytes) 000364 1786562370.063688 trunc from 0x008fde to 0x00c66d (0x3690 bytes) 000365 1786562370.077595 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27faa): Mirror 4: 0x2beb898 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27faa): Mirror 4: 0x2beb898 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27faa: skipped 000366 1786562370.284424 mapwrite 0x030396 thru 0x038c1a (0x8885 bytes) 000367 1786562370.375705 read 0x0021c1 thru 0x0042dd (0x211d bytes) 000368 1786562370.379764 trunc from 0x038c1b to 0x0113e3 (0x27837 bytes) 000369 1786562370.391397 read 0x00a3fd thru 0x0113e3 (0x6fe7 bytes) 000370 1786562370.397550 mapread 0x00e8c4 thru 0x010ab3 (0x21f0 bytes) 000371 1786562370.412508 MIRROR_RESYNC 2 mirrors 000373 1786562370.840227 punch from 0x00572e to 0x0091c8 (0x3a9b bytes) 000374 1786562370.880415 fallocate 0x027cb7 thru 0x0360cf (0xe419 bytes) 000375 1786562370.885568 trunc from 0x020b94 to 0x02b6f4 (0xab61 bytes) 000376 1786562370.893629 punch from 0x020c79 to 0x024c54 (0x3fdc bytes) 000377 1786562370.899722 trunc from 0x0360d0 to 0x037c32 (0x1b63 bytes) 000378 1786562370.977627 trunc from 0x02cd66 to 0x037c32 (0xaecd bytes) 000379 1786562371.000558 read 0x023075 thru 0x029da6 (0x6d32 bytes) 000380 1786562371.066740 MIRROR_RESYNC 2 mirrors 000381 1786562371.752473 trunc from 0x01abc4 to 0x01d2ff (0x273c bytes) 000382 1786562371.787048 mapread 0x02e2cb thru 0x037c32 (0x9968 bytes) 000383 1786562371.810856 trunc from 0x02192d to 0x030a3e (0xf112 bytes) 000384 1786562371.815177 read 0x0012f3 thru 0x00bdb6 (0xaac4 bytes) 000385 1786562371.828930 trunc from 0x013c48 to 0x01d0e1 (0x949a bytes) 000386 1786562371.834786 read 0x0046ed thru 0x006bb9 (0x24cd bytes) 000387 1786562371.835600 mapwrite 0x00b324 thru 0x015953 (0xa630 bytes) 000388 1786562371.890638 fallocate 0x00c12b thru 0x0197b6 (0xd68c bytes) 000389 1786562371.895176 mapread 0x015c8d thru 0x01b853 (0x5bc7 bytes) 000390 1786562371.896308 write 0x012270 thru 0x01dd80 (0xbb11 bytes) 000391 1786562371.901776 fallocate 0x03d463 thru 0x03ffff (0x2b9d bytes) 000392 1786562371.914073 read 0x033de4 thru 0x037c32 (0x3e4f bytes) 000393 1786562371.916177 trunc from 0x0179ac to 0x01f83d (0x7e92 bytes) 000394 1786562371.928758 write 0x014a25 thru 0x016ff4 (0x25d0 bytes) 000395 1786562371.932181 fallocate 0x022498 thru 0x02f54a (0xd0b3 bytes) 000396 1786562371.946901 trunc from 0x0132bc to 0x01e2d2 (0xb017 bytes) 000397 1786562371.963477 read 0x02ef2c thru 0x037c32 (0x8d07 bytes) 000398 1786562371.965588 mapread 0x001393 thru 0x010047 (0xecb5 bytes) 000399 1786562371.974738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000400 1786562372.128399 write 0x026666 thru 0x028d0f (0x26aa bytes) 000401 1786562372.142435 mapread 0x034a8f thru 0x037c32 (0x31a4 bytes) 000402 1786562372.158945 fallocate 0x036665 thru 0x03eddd (0x8779 bytes) 000403 1786562372.163479 fallocate 0x01a556 thru 0x023817 (0x92c2 bytes) 000404 1786562372.171618 MIRROR_EXTEND to 2 mirrors 000405 1786562372.796148 fallocate 0x030bdd thru 0x03c79f (0xbbc3 bytes) 000406 1786562372.829140 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000407 1786562372.928767 mapwrite 0x03564d thru 0x038f93 (0x3947 bytes) 000408 1786562372.960602 fallocate 0x02935a thru 0x03011c (0x6dc3 bytes) 000409 1786562372.964453 write 0x0061f3 thru 0x015510 (0xf31e bytes) 000411 1786562372.976769 write 0x01de8e thru 0x02c0e2 (0xe255 bytes) 000412 1786562372.985122 read 0x03dab3 thru 0x03de02 (0x350 bytes) 000415 1786562372.987572 write 0x01862c thru 0x01d4b5 (0x4e8a bytes) 000418 1786562372.994890 mapwrite 0x013143 thru 0x017222 (0x40e0 bytes) 000419 1786562373.019204 fallocate 0x00cca6 thru 0x016031 (0x938c bytes) 000420 1786562373.022733 punch from 0x03d4e6 to 0x03eddd (0x18f8 bytes) 000421 1786562373.030006 MIRROR_EXTEND to 2 mirrors 000422 1786562373.524658 write 0x00a289 thru 0x00af89 (0xd01 bytes) 000423 1786562373.572700 punch from 0x0192b9 to 0x0193d7 (0x11f bytes) 000424 1786562373.577880 write 0x03da27 thru 0x03ffff (0x25d9 bytes) 000425 1786562373.583895 write 0x0280fd thru 0x03251b (0xa41f bytes) 000426 1786562373.595177 trunc from 0x00afa2 to 0x01ae22 (0xfe81 bytes) 000427 1786562374.049771 trunc from 0x00e903 to 0x013e24 (0x5522 bytes) 000428 1786562374.069657 mapwrite 0x01791a thru 0x01dcc7 (0x63ae bytes) 000429 1786562374.130826 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000430 1786562374.338326 trunc from 0x01e687 to 0x027d5b (0x96d5 bytes) 000431 1786562374.348599 fallocate 0x033185 thru 0x035257 (0x20d3 bytes) 000432 1786562374.354740 mapread 0x02e24d thru 0x0331e6 (0x4f9a bytes) 000436 1786562374.394015 MIRROR_EXTEND to 2 mirrors 000438 1786562375.486137 MIRROR_EXTEND to 3 mirrors 000439 1786562376.199499 mapread 0x002de0 thru 0x011743 (0xe964 bytes) 000440 1786562376.237541 mapread 0x022b87 thru 0x0312d5 (0xe74f bytes) 000441 1786562376.260452 trunc from 0x040000 to 0x0146f8 (0x2b907 bytes) 000442 1786562376.297197 mapread 0x00fd66 thru 0x013c6f (0x3f0a bytes) 000443 1786562376.306921 write 0x001833 thru 0x009f5c (0x872a bytes) 000444 1786562376.312118 write 0x03498b thru 0x03ffff (0xb675 bytes) 000446 1786562376.315760 mapwrite 0x00f5d6 thru 0x01044b (0xe76 bytes) 000447 1786562376.336819 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb67cf043 (stale) Mirror 3: 0xb67cf043 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19f8de8a Mirror 4: 0x19f8de8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000448 1786562376.932164 fallocate 0x03f3d6 thru 0x03ffff (0xc2a bytes) 000449 1786562376.965629 trunc from 0x040000 to 0x018270 (0x27d8f bytes) 000450 1786562377.000598 read 0x014bda thru 0x018270 (0x3697 bytes) 000451 1786562377.009713 read 0x00ca7a thru 0x010abd (0x4044 bytes) 000453 1786562377.027036 punch from 0x002bda to 0x009257 (0x667e bytes) 000455 1786562377.040090 trunc from 0x012a63 to 0x018270 (0x580e bytes) 000456 1786562377.044476 write 0x021913 thru 0x02a8b6 (0x8fa4 bytes) 000457 1786562377.052082 MIRROR_RESYNC 4 mirrors 000458 1786562378.263215 fallocate 0x005499 thru 0x0090e0 (0x3c48 bytes) 000459 1786562378.298316 trunc from 0x02a8b7 to 0x036ac1 (0xc20b bytes) 000461 1786562378.323238 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36ac2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ac2): Mirror 2: 0x1a958f5d (stale) Mirror 3: 0x1a958f5d (stale) Mirror 4: 0x1a958f5d (stale) CRC-32 checksum value for chunk [0, 0x36ac2): Mirror 1: 0x1a958f5d Mirror 5: 0x1a958f5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ac2: skipped 000462 1786562379.517979 write 0x03627c thru 0x03d0e8 (0x6e6d bytes) 000464 1786562379.558583 trunc from 0x009b77 to 0x00e6a6 (0x4b30 bytes) 000465 1786562379.562367 trunc from 0x02f134 to 0x031c31 (0x2afe bytes) 000466 1786562379.565513 mapread 0x0181a3 thru 0x01e9d4 (0x6832 bytes) 000467 1786562379.577082 trunc from 0x01f535 to 0x022498 (0x2f64 bytes) 000468 1786562379.580776 read 0x03516d thru 0x03d011 (0x7ea5 bytes) 000469 1786562379.583672 trunc from 0x03c3d6 to 0x03d0e8 (0xd13 bytes) 000470 1786562379.598635 mapread 0x021c4f thru 0x02ec93 (0xd045 bytes) 000471 1786562379.620163 read 0x02393b thru 0x0321bd (0xe883 bytes) 000472 1786562379.621544 trunc from 0x035551 to 0x03d0e8 (0x7b98 bytes) 000473 1786562379.627195 read 0x006c05 thru 0x00f7e3 (0x8bdf bytes) 000474 1786562379.637335 read 0x02f09b thru 0x03a162 (0xb0c8 bytes) 000475 1786562379.638110 trunc from 0x03d0e9 to 0x00b536 (0x31bb2 bytes) 000477 1786562379.660121 mapread 0x00452e thru 0x007d4f (0x3822 bytes) 000478 1786562379.662077 trunc from 0x0054bf to 0x00b536 (0x6078 bytes) 000479 1786562379.667365 fallocate 0x03da38 thru 0x03ffff (0x25c8 bytes) 000480 1786562379.672759 mapwrite 0x0069c4 thru 0x00d794 (0x6dd1 bytes) 000481 1786562379.709524 MIRROR_RESYNC 5 mirrors 000482 1786562380.727221 write 0x02b5bd thru 0x035f5f (0xa9a3 bytes) 000483 1786562380.760750 trunc from 0x02c2fe to 0x0352d3 (0x8fd6 bytes) 000484 1786562380.782652 trunc from 0x022fa5 to 0x028ce5 (0x5d41 bytes) 000485 1786562380.794390 fallocate 0x02ffc6 thru 0x035b38 (0x5b73 bytes) 000486 1786562380.801245 read 0x022f2e thru 0x0265cf (0x36a2 bytes) 000487 1786562380.814749 read 0x018fa3 thru 0x0280b4 (0xf112 bytes) 000489 1786562380.842006 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcc9f6006 (stale) Mirror 4: 0xcc9f6006 (stale) Mirror 5: 0xcc9f6006 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcc9f6006 (stale) Mirror 4: 0xcc9f6006 (stale) Mirror 5: 0xcc9f6006 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1786562381.291345 read 0x02a0e4 thru 0x032f4e (0x8e6b bytes) 000491 1786562381.311719 punch from 0x02dcc5 to 0x0382f9 (0xa635 bytes) 000492 1786562381.337150 mapread 0x03df6b thru 0x03ffff (0x2095 bytes) 000493 1786562381.357925 mapread 0x019a92 thru 0x01d0ae (0x361d bytes) 000494 1786562381.370144 write 0x00f2af thru 0x01be25 (0xcb77 bytes) 000496 1786562381.382105 read 0x0207bd thru 0x021a74 (0x12b8 bytes) 000497 1786562381.383389 trunc from 0x040000 to 0x021f56 (0x1e0a9 bytes) 000498 1786562381.401398 read 0x011490 thru 0x013e62 (0x29d3 bytes) 000499 1786562381.408975 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21f57): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21f57): Mirror 3: 0xa0e58ce3 (stale) Mirror 4: 0xa0e58ce3 (stale) Mirror 5: 0xa0e58ce3 (stale) CRC-32 checksum value for chunk [0, 0x21f57): Mirror 1: 0xed65886b Mirror 6: 0xed65886b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21f57: skipped 000500 1786562382.381043 trunc from 0x021f57 to 0x018df5 (0x9161 bytes) 000501 1786562382.479669 mapwrite 0x02acd9 thru 0x02f202 (0x452a bytes) 000502 1786562382.645030 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f203): Mirror 4: 0x583c7ef2 (stale) Mirror 5: 0x583c7ef2 (stale) Mirror 6: 0xb3dfdfd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f203): Mirror 4: 0x583c7ef2 (stale) Mirror 5: 0x583c7ef2 (stale) Mirror 6: 0xb3dfdfd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f203: skipped 000503 1786562383.104657 trunc from 0x02f203 to 0x03e268 (0xf066 bytes) 000504 1786562383.128824 fallocate 0x034c8e thru 0x039198 (0x450b bytes) 000505 1786562383.134799 trunc from 0x03b2ab to 0x03e268 (0x2fbe bytes) 000506 1786562383.142757 trunc from 0x03e269 to 0x0188fa (0x2596e bytes) 000507 1786562383.157921 write 0x039963 thru 0x03bf69 (0x2607 bytes) 000508 1786562383.160157 punch from 0x032c36 to 0x0372a4 (0x466f bytes) 000509 1786562383.168312 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf6a): Mirror 5: 0x28af2b3 (stale) Mirror 6: 0x2c51b457 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf6a): Mirror 5: 0x28af2b3 (stale) Mirror 6: 0x2c51b457 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf6a: skipped 000510 1786562383.416944 trunc from 0x037c3a to 0x03bf69 (0x4330 bytes) 000511 1786562383.422361 trunc from 0x0235e4 to 0x024838 (0x1255 bytes) 000512 1786562383.426193 mapwrite 0x003f2e thru 0x007891 (0x3964 bytes) 000513 1786562383.460001 write 0x01c128 thru 0x0288f3 (0xc7cc bytes) 000514 1786562383.474392 mapread 0x018b0c thru 0x023a5c (0xaf51 bytes) 000515 1786562383.485969 trunc from 0x02a8da to 0x034286 (0x99ad bytes) 000516 1786562383.492316 mapread 0x0355a3 thru 0x03bf69 (0x69c7 bytes) 000517 1786562383.517576 fallocate 0x00785a thru 0x01624d (0xe9f4 bytes) 000518 1786562383.526160 read 0x015ab3 thru 0x01bbed (0x613b bytes) 000519 1786562383.528422 trunc from 0x00a5cf to 0x00b1e0 (0xc12 bytes) 000520 1786562383.534087 mapwrite 0x010383 thru 0x01179c (0x141a bytes) 000521 1786562383.554161 read 0x013b41 thru 0x01e3e3 (0xa8a3 bytes) 000522 1786562383.555058 write 0x01a6d1 thru 0x021d39 (0x7669 bytes) 000524 1786562383.558325 trunc from 0x03bf6a to 0x0310e3 (0xae86 bytes) 000525 1786562383.583016 write 0x02931f thru 0x02bcfe (0x29e0 bytes) 000526 1786562383.591529 mapread 0x00563a thru 0x01163f (0xc006 bytes) 000527 1786562383.593631 mapread 0x00239f thru 0x007277 (0x4ed9 bytes) 000528 1786562383.595020 punch from 0x01540f to 0x0224d2 (0xd0c4 bytes) 000530 1786562383.611012 read 0x012577 thru 0x018a98 (0x6522 bytes) 000531 1786562383.624163 write 0x02f3f3 thru 0x03b1b4 (0xbdc2 bytes) 000533 1786562383.628422 mapread 0x00c905 thru 0x0102af (0x39ab bytes) 000534 1786562383.630704 read 0x0181c0 thru 0x01a550 (0x2391 bytes) 000535 1786562383.631910 mapwrite 0x023380 thru 0x03296d (0xf5ee bytes) 000536 1786562383.669284 mapread 0x02af3e thru 0x02caa9 (0x1b6c bytes) 000537 1786562383.670795 read 0x037ad1 thru 0x03b1b4 (0x36e4 bytes) 000538 1786562383.672445 fallocate 0x008b13 thru 0x014a72 (0xbf60 bytes) 000541 1786562383.681481 mapwrite 0x032ea6 thru 0x03ffff (0xd15a bytes) 000542 1786562383.730747 trunc from 0x040000 to 0x019a5f (0x265a0 bytes) 000543 1786562383.749377 punch from 0x008463 to 0x0097b5 (0x1353 bytes) 000544 1786562383.756114 read 0x00b2a9 thru 0x0151a7 (0x9eff bytes) 000545 1786562383.757124 write 0x00844b thru 0x014652 (0xc208 bytes) 000546 1786562383.764705 punch from 0x00bc02 to 0x00ffcf (0x43ce bytes) 000547 1786562383.787524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19a60): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a60): Mirror 5: 0x12dadfde (stale) Mirror 6: 0x4df5471b (stale) CRC-32 checksum value for chunk [0, 0x19a60): Mirror 1: 0xeecee286 Mirror 7: 0xeecee286 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a60: skipped 000548 1786562384.797981 mapwrite 0x004b3a thru 0x00aa57 (0x5f1e bytes) 000549 1786562384.910713 write 0x03b495 thru 0x03ffff (0x4b6b bytes) 000550 1786562384.918094 trunc from 0x032370 to 0x03ffff (0xdc90 bytes) 000551 1786562384.938025 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xcc9f6006 (stale) Mirror 6: 0x81973663 (stale) Mirror 7: 0xc279a21d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a4ab44e Mirror 8: 0x1a4ab44e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000552 1786562385.939035 MIRROR_RESYNC 5 mirrors 000553 1786562386.827482 mapread 0x00747a thru 0x0143e4 (0xcf6b bytes) 000556 1786562386.858447 mapwrite 0x005e4d thru 0x00a19a (0x434e bytes) 000558 1786562386.936788 mapwrite 0x02661f thru 0x029dc7 (0x37a9 bytes) 000559 1786562386.965076 trunc from 0x040000 to 0x02bab7 (0x14548 bytes) 000560 1786562387.018716 fallocate 0x00178f thru 0x002415 (0xc87 bytes) 000561 1786562387.023300 trunc from 0x016a5e to 0x01f8ad (0x8e50 bytes) 000562 1786562387.028694 mapread 0x024748 thru 0x02bab7 (0x7370 bytes) 000563 1786562387.030776 trunc from 0x02bab8 to 0x03783d (0xbd86 bytes) 000564 1786562387.046467 trunc from 0x03783e to 0x033c6d (0x3bd0 bytes) 000565 1786562387.056556 trunc from 0x033c6e to 0x01dfc2 (0x15cab bytes) 000566 1786562387.065740 trunc from 0x00cb2c to 0x01a380 (0xd855 bytes) 000567 1786562387.069436 punch from 0x01317f to 0x01a7be (0x7640 bytes) 000569 1786562387.076457 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1dfc3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dfc3): Mirror 5: 0xbd1ba3b9 (stale) Mirror 6: 0xbd1ba3b9 (stale) Mirror 7: 0xbd1ba3b9 (stale) Mirror 8: 0xbd1ba3b9 (stale) CRC-32 checksum value for chunk [0, 0x1dfc3): Mirror 1: 0x5f6854c5 Mirror 9: 0x5f6854c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dfc3: skipped 000570 1786562387.845960 mapwrite 0x035bcf thru 0x03ffff (0xa431 bytes) 000571 1786562387.946816 punch from 0x01d6eb to 0x01d781 (0x97 bytes) 000572 1786562387.955977 punch from 0x016f2e to 0x0231ec (0xc2bf bytes) 000573 1786562387.961469 punch from 0x0180e2 to 0x01c76d (0x468c bytes) 000574 1786562387.966346 punch from 0x0012eb to 0x002332 (0x1048 bytes) 000575 1786562387.972144 mapwrite 0x0066d1 thru 0x015f03 (0xf833 bytes) 000576 1786562388.026347 mapread 0x02717d thru 0x02ebb9 (0x7a3d bytes) 000577 1786562388.039168 trunc from 0x040000 to 0x02bc83 (0x1437c bytes) 000579 1786562388.071460 read 0x0064da thru 0x01234b (0xbe72 bytes) 000580 1786562388.077974 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 5: 0x4bf70aed (stale) Mirror 6: 0x4bf70aed (stale) Mirror 7: 0x4bf70aed (stale) Mirror 8: 0x4bf70aed (stale) Mirror 9: 0xe0742e8d (stale) CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 1: 0xff7ac6f1 Mirror 10: 0xff7ac6f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc84: skipped 000581 1786562388.953795 write 0x01005a thru 0x0117d4 (0x177b bytes) 000582 1786562389.041428 trunc from 0x01fef0 to 0x029c41 (0x9d52 bytes) 000583 1786562389.049944 mapread 0x015680 thru 0x024a84 (0xf405 bytes) 000584 1786562389.090652 fallocate 0x027ccc thru 0x029cca (0x1fff bytes) 000585 1786562389.097951 mapwrite 0x011245 thru 0x013c84 (0x2a40 bytes) 000586 1786562389.127960 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 6: 0x4bf70aed (stale) Mirror 7: 0x4bf70aed (stale) Mirror 8: 0x4bf70aed (stale) Mirror 9: 0xe0742e8d (stale) Mirror 10: 0xff7ac6f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 6: 0x4bf70aed (stale) Mirror 7: 0x4bf70aed (stale) Mirror 8: 0x4bf70aed (stale) Mirror 9: 0xe0742e8d (stale) Mirror 10: 0xff7ac6f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc84: skipped 000587 1786562389.866545 trunc from 0x0237c5 to 0x027911 (0x414d bytes) 000588 1786562389.873056 read 0x0257eb thru 0x02bc83 (0x6499 bytes) 000589 1786562389.894396 trunc from 0x00ec75 to 0x01db41 (0xeecd bytes) 000590 1786562389.900711 mapread 0x014a36 thru 0x02112b (0xc6f6 bytes) 000592 1786562389.941877 MIRROR_RESYNC 6 mirrors 000593 1786562391.171461 write 0x0235ce thru 0x027ea4 (0x48d7 bytes) 000594 1786562391.233274 fallocate 0x0047e5 thru 0x0105f6 (0xbe12 bytes) 000595 1786562391.240361 mapread 0x01671f thru 0x0241cf (0xdab1 bytes) 000596 1786562391.273626 punch from 0x0023ba to 0x0049db (0x2622 bytes) 000598 1786562391.280602 punch from 0x0069a9 to 0x00a7f9 (0x3e51 bytes) 000599 1786562391.286621 punch from 0x02a05c to 0x02bc83 (0x1c28 bytes) 000600 1786562391.295823 punch from 0x021a69 to 0x022525 (0xabd bytes) 000601 1786562391.301404 punch from 0x01d0f3 to 0x02bc83 (0xeb91 bytes) 000602 1786562391.332041 punch from 0x028db9 to 0x02bc83 (0x2ecb bytes) 000604 1786562391.347233 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 6: 0x423be87 (stale) Mirror 7: 0x423be87 (stale) Mirror 8: 0x423be87 (stale) Mirror 9: 0x423be87 (stale) Mirror 10: 0x423be87 (stale) CRC-32 checksum value for chunk [0, 0x2bc84): Mirror 1: 0x52e65869 Mirror 11: 0x52e65869 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc84: skipped 000605 1786562392.148817 trunc from 0x02bc84 to 0x02c2ae (0x62b bytes) 000606 1786562392.228919 punch from 0x01a4fd to 0x023cc7 (0x97cb bytes) 000607 1786562392.241981 read 0x028873 thru 0x02c2ae (0x3a3c bytes) 000608 1786562392.258928 read 0x01af38 thru 0x0266a8 (0xb771 bytes) 000609 1786562392.288015 MIRROR_RESYNC 7 mirrors 000611 1786562394.382101 write 0x009f7e thru 0x01854a (0xe5cd bytes) 000612 1786562394.462367 mapwrite 0x004e14 thru 0x0131a3 (0xe390 bytes) 000614 1786562394.513247 mapwrite 0x003185 thru 0x0084c7 (0x5343 bytes) 000616 1786562394.530655 trunc from 0x02c2af to 0x00e9ea (0x1d8c4 bytes) 000617 1786562394.570232 fallocate 0x00ac9b thru 0x01abd2 (0xff38 bytes) 000619 1786562394.576460 mapread 0x006f3a thru 0x00e9ea (0x7ab1 bytes) 000621 1786562394.582011 mapwrite 0x0392ad thru 0x03ebf0 (0x5944 bytes) 000622 1786562394.643363 mapread 0x02787f thru 0x02f3b5 (0x7b37 bytes) 000623 1786562394.666777 read 0x0376fb thru 0x03ebf0 (0x74f6 bytes) 000625 1786562394.668115 write 0x00cbe0 thru 0x00de3f (0x1260 bytes) 000626 1786562394.671472 trunc from 0x01266b to 0x01c201 (0x9b97 bytes) 000627 1786562394.677045 trunc from 0x03ebf1 to 0x02e59b (0x10655 bytes) 000628 1786562394.691965 trunc from 0x02e59c to 0x005377 (0x29224 bytes) 000629 1786562394.710184 write 0x01e011 thru 0x028e66 (0xae56 bytes) 000630 1786562394.713276 punch from 0x012456 to 0x013d83 (0x192e bytes) 000631 1786562394.717740 read 0x025434 thru 0x028e66 (0x3a33 bytes) 000632 1786562394.718723 write 0x03b89b thru 0x03ffff (0x4765 bytes) 000633 1786562394.720449 mapread 0x028874 thru 0x02ebaa (0x6337 bytes) 000634 1786562394.730071 fallocate 0x030f93 thru 0x0314c6 (0x534 bytes) 000635 1786562394.734826 punch from 0x03bfbc to 0x03e3fb (0x2440 bytes) 000636 1786562394.756652 punch from 0x023763 to 0x0306e6 (0xcf84 bytes) 000637 1786562394.763000 read 0x02e843 thru 0x03b019 (0xc7d7 bytes) 000638 1786562394.777988 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdfa83646 (stale) Mirror 7: 0xdfa83646 (stale) Mirror 8: 0xdfa83646 (stale) Mirror 9: 0xdfa83646 (stale) Mirror 11: 0xdfa83646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdfa83646 (stale) Mirror 7: 0xdfa83646 (stale) Mirror 8: 0xdfa83646 (stale) Mirror 9: 0xdfa83646 (stale) Mirror 11: 0xdfa83646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1786562395.144720 trunc from 0x025e2d to 0x03281a (0xc9ee bytes) 000640 1786562395.149683 write 0x038a95 thru 0x03ffff (0x756b bytes) 000641 1786562395.160265 fallocate 0x004c75 thru 0x00fa59 (0xade5 bytes) 000642 1786562395.168323 mapwrite 0x034d07 thru 0x03be9e (0x7198 bytes) 000643 1786562395.202646 write 0x039fb2 thru 0x03ba53 (0x1aa2 bytes) 000644 1786562395.203779 write 0x0024d7 thru 0x0102f2 (0xde1c bytes) 000645 1786562395.216773 punch from 0x03ff09 to 0x03ffff (0xf7 bytes) 000646 1786562395.222072 write 0x03fd22 thru 0x03ffff (0x2de bytes) 000647 1786562395.226731 punch from 0x0243da to 0x0331fd (0xee24 bytes) 000649 1786562395.237493 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdfa83646 (stale) Mirror 8: 0xdfa83646 (stale) Mirror 9: 0xdfa83646 (stale) Mirror 11: 0xdfa83646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xdfa83646 (stale) Mirror 8: 0xdfa83646 (stale) Mirror 9: 0xdfa83646 (stale) Mirror 11: 0xdfa83646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000650 1786562395.567184 mapread 0x0337cb thru 0x035d42 (0x2578 bytes) 000651 1786562395.584669 write 0x030273 thru 0x037818 (0x75a6 bytes) 000652 1786562395.591076 punch from 0x03bb51 to 0x03ca8f (0xf3f bytes) 000653 1786562395.596795 trunc from 0x026e06 to 0x02e792 (0x798d bytes) 000654 1786562395.601624 punch from 0x01c08d to 0x02b89e (0xf812 bytes) 000655 1786562395.607195 trunc from 0x033f21 to 0x036ce5 (0x2dc5 bytes) 000656 1786562395.619730 fallocate 0x02b1c3 thru 0x03776f (0xc5ad bytes) 000657 1786562395.625370 read 0x00e63b thru 0x0181e0 (0x9ba6 bytes) 000658 1786562395.633694 read 0x0124d3 thru 0x018e89 (0x69b7 bytes) 000659 1786562395.634463 read 0x032f61 thru 0x03d838 (0xa8d8 bytes) 000660 1786562395.642023 write 0x00323b thru 0x0125a2 (0xf368 bytes) 000661 1786562395.645043 trunc from 0x02d712 to 0x038c41 (0xb530 bytes) 000662 1786562395.650683 write 0x00949f thru 0x012d30 (0x9892 bytes) 000663 1786562395.652387 trunc from 0x011f2a to 0x021d26 (0xfdfd bytes) 000664 1786562395.666898 punch from 0x027f0e to 0x0354f1 (0xd5e4 bytes) 000665 1786562395.671552 mapwrite 0x00ed9e thru 0x0179d0 (0x8c33 bytes) 000666 1786562395.731882 punch from 0x010e6e to 0x0112ee (0x481 bytes) 000667 1786562395.738007 trunc from 0x040000 to 0x008232 (0x37dcd bytes) 000668 1786562395.771721 MIRROR_RESYNC 5 mirrors 000669 1786562396.542032 write 0x037782 thru 0x03ffff (0x887e bytes) 000670 1786562396.573366 write 0x0348fb thru 0x03ffff (0xb705 bytes) 000671 1786562396.580400 mapread 0x013ab1 thru 0x018b0f (0x505f bytes) 000672 1786562396.601891 punch from 0x03df16 to 0x03ffff (0x20ea bytes) 000673 1786562396.629671 mapwrite 0x026e74 thru 0x02ae67 (0x3ff4 bytes) 000674 1786562396.668607 mapwrite 0x002c78 thru 0x0088c1 (0x5c4a bytes) 000675 1786562396.702874 read 0x00e0f2 thru 0x012a8f (0x499e bytes) 000677 1786562396.705575 fallocate 0x0276a2 thru 0x03264a (0xafa9 bytes) 000678 1786562396.711566 trunc from 0x040000 to 0x037f94 (0x806b bytes) 000679 1786562396.739372 punch from 0x02d194 to 0x035679 (0x84e6 bytes) 000680 1786562396.746286 write 0x031a66 thru 0x031beb (0x186 bytes) 000681 1786562396.752169 write 0x00e84a thru 0x00fd1d (0x14d4 bytes) 000682 1786562396.753692 mapwrite 0x03776f thru 0x03ffff (0x8891 bytes) 000683 1786562396.815386 read 0x0236d1 thru 0x0246e2 (0x1012 bytes) 000684 1786562396.821074 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xebeff43c (stale) Mirror 8: 0xebeff43c (stale) Mirror 9: 0xebeff43c (stale) Mirror 11: 0xebeff43c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x995aa6cd Mirror 12: 0x995aa6cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000685 1786562398.085422 MIRROR_RESYNC 6 mirrors 000686 1786562400.420301 fallocate 0x036d51 thru 0x03ffff (0x92af bytes) 000687 1786562400.461961 read 0x03a413 thru 0x03da98 (0x3686 bytes) 000688 1786562400.473298 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x995aa6cd (stale) Mirror 8: 0x995aa6cd (stale) Mirror 9: 0x995aa6cd (stale) Mirror 11: 0x995aa6cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x995aa6cd (stale) Mirror 8: 0x995aa6cd (stale) Mirror 9: 0x995aa6cd (stale) Mirror 11: 0x995aa6cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000690 1786562400.908852 trunc from 0x023b7d to 0x03215f (0xe5e3 bytes) 000691 1786562400.915338 write 0x011c72 thru 0x015a24 (0x3db3 bytes) 000692 1786562400.935091 mapread 0x010a71 thru 0x020616 (0xfba6 bytes) 000693 1786562400.995822 punch from 0x021d7e to 0x030383 (0xe606 bytes) 000694 1786562401.015011 MIRROR_RESYNC 5 mirrors 000696 1786562402.598792 fallocate 0x0378b9 thru 0x03cee7 (0x562f bytes) 000697 1786562402.633617 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x905daa51 (stale) Mirror 8: 0x905daa51 (stale) Mirror 9: 0x905daa51 (stale) Mirror 11: 0x905daa51 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x905daa51 Mirror 12: 0x905daa51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1786562404.016456 mapwrite 0x016e81 thru 0x01b87e (0x49fe bytes) 000699 1786562404.250280 trunc from 0x0388a7 to 0x03ffff (0x7759 bytes) 000700 1786562404.259432 trunc from 0x0390b5 to 0x03f2c3 (0x620f bytes) 000701 1786562404.265629 fallocate 0x022ebb thru 0x0263b4 (0x34fa bytes) 000702 1786562404.272601 trunc from 0x00245a to 0x00bff8 (0x9b9f bytes) 000704 1786562404.278082 fallocate 0x01035e thru 0x01a2a0 (0x9f43 bytes) 000705 1786562404.288634 trunc from 0x01dd76 to 0x027175 (0x9400 bytes) 000706 1786562404.293932 mapwrite 0x03fc8e thru 0x03ffff (0x372 bytes) 000707 1786562404.348410 punch from 0x022b66 to 0x030126 (0xd5c1 bytes) 000708 1786562404.361704 trunc from 0x034be5 to 0x034cd6 (0xf2 bytes) 000709 1786562404.367967 mapread 0x0269b3 thru 0x02a3f5 (0x3a43 bytes) 000710 1786562404.405375 write 0x00c829 thru 0x01231f (0x5af7 bytes) 000711 1786562404.425689 write 0x03d3f0 thru 0x03ffff (0x2c10 bytes) 000712 1786562404.432664 MIRROR_RESYNC 6 mirrors 000713 1786562406.291685 read 0x0095ee thru 0x00a539 (0xf4c bytes) 000714 1786562406.298312 trunc from 0x02a4e7 to 0x03112c (0x6c46 bytes) 000715 1786562406.347947 trunc from 0x038cea to 0x03cdae (0x40c5 bytes) 000716 1786562406.362403 read 0x000d4e thru 0x004ef3 (0x41a6 bytes) 000717 1786562406.408628 fallocate 0x033be4 thru 0x03f026 (0xb443 bytes) 000718 1786562406.416140 mapwrite 0x03cb46 thru 0x03dbc1 (0x107c bytes) 000719 1786562406.447983 trunc from 0x040000 to 0x016812 (0x297ed bytes) 000720 1786562406.480559 read 0x00819e thru 0x016812 (0xe675 bytes) 000721 1786562406.486684 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16813): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16813): Mirror 6: 0x132ba2f3 (stale) Mirror 8: 0x132ba2f3 (stale) Mirror 9: 0x132ba2f3 (stale) Mirror 11: 0x132ba2f3 (stale) Mirror 12: 0x132ba2f3 (stale) CRC-32 checksum value for chunk [0, 0x16813): Mirror 1: 0x132ba2f3 Mirror 13: 0x132ba2f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16813: skipped 000723 1786562407.280420 mapwrite 0x02c0d2 thru 0x02f0e2 (0x3011 bytes) 000725 1786562407.377692 mapread 0x006bed thru 0x014aba (0xdece bytes) 000726 1786562407.397266 read 0x015a24 thru 0x018a91 (0x306e bytes) 000727 1786562407.398750 trunc from 0x02f0e3 to 0x02aa15 (0x46cd bytes) 000728 1786562407.416810 write 0x039a97 thru 0x03ffff (0x6569 bytes) 000729 1786562407.418975 trunc from 0x02ac00 to 0x02db33 (0x2f34 bytes) 000730 1786562407.426265 fallocate 0x01636c thru 0x01e9c4 (0x8659 bytes) 000731 1786562407.432880 punch from 0x01279e to 0x018df9 (0x665c bytes) 000732 1786562407.453385 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x67924fc8 (stale) Mirror 8: 0x67924fc8 (stale) Mirror 9: 0x67924fc8 (stale) Mirror 11: 0x67924fc8 (stale) Mirror 12: 0x67924fc8 (stale) Mirror 13: 0x1c0044dd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe088a871 Mirror 14: 0xe088a871 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000733 1786562408.486883 trunc from 0x040000 to 0x01f052 (0x20fad bytes) 000734 1786562408.529811 read 0x009b41 thru 0x0179ab (0xde6b bytes) 000735 1786562408.538470 fallocate 0x0033bc thru 0x008779 (0x53be bytes) 000736 1786562408.541308 trunc from 0x01f053 to 0x017f82 (0x70d0 bytes) 000737 1786562408.559702 MIRROR_RESYNC 8 mirrors 000738 1786562410.232568 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) 000739 1786562410.719812 mapwrite 0x00299e thru 0x005ae2 (0x3145 bytes) 000741 1786562410.817821 mapread 0x00985d thru 0x00ecd2 (0x5476 bytes) 000742 1786562410.820447 trunc from 0x017f83 to 0x025bed (0xdc6b bytes) 000744 1786562410.844889 trunc from 0x025bee to 0x032672 (0xca85 bytes) 000747 1786562410.865168 write 0x0321ae thru 0x03a034 (0x7e87 bytes) 000748 1786562410.879561 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a035): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a035): Mirror 6: 0xd103d082 (stale) Mirror 8: 0xd103d082 (stale) Mirror 9: 0xd103d082 (stale) Mirror 11: 0xd103d082 (stale) Mirror 12: 0xd103d082 (stale) Mirror 13: 0xd103d082 (stale) CRC-32 checksum value for chunk [0, 0x3a035): Mirror 1: 0x899ce9ca Mirror 14: 0x899ce9ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a035: skipped 000749 1786562411.725578 write 0x01ac06 thru 0x027499 (0xc894 bytes) 000750 1786562411.775138 mapread 0x00bf15 thru 0x0130eb (0x71d7 bytes) 000751 1786562411.795734 mapread 0x022f88 thru 0x02506d (0x20e6 bytes) 000752 1786562411.796817 mapread 0x00a7c7 thru 0x012e3b (0x8675 bytes) 000754 1786562411.804333 write 0x01ae49 thru 0x01f47a (0x4632 bytes) 000755 1786562411.805546 write 0x03c0ee thru 0x03ffff (0x3f12 bytes) 000756 1786562411.810292 trunc from 0x040000 to 0x0077d8 (0x38827 bytes) 000757 1786562411.836760 trunc from 0x0077d9 to 0x034fdf (0x2d807 bytes) 000758 1786562411.847480 mapwrite 0x0243bc thru 0x02a3f7 (0x603c bytes) 000759 1786562411.875437 trunc from 0x034fe0 to 0x025171 (0xfe6e bytes) 000760 1786562411.907223 trunc from 0x025172 to 0x01987c (0xb8f5 bytes) 000761 1786562411.934468 fallocate 0x03e29c thru 0x03ffff (0x1d64 bytes) 000762 1786562411.942171 mapread 0x019699 thru 0x01987c (0x1e4 bytes) 000763 1786562411.957771 mapwrite 0x0339bb thru 0x038266 (0x48ac bytes) 000765 1786562412.069749 MIRROR_RESYNC 8 mirrors 000766 1786562413.576466 MIRROR_RESYNC 8 mirrors 000768 1786562414.102099 fallocate 0x00a515 thru 0x010335 (0x5e21 bytes) 000769 1786562414.133738 trunc from 0x02ac9f to 0x02b390 (0x6f2 bytes) 000770 1786562414.140047 mapread 0x02ad62 thru 0x0340c7 (0x9366 bytes) 000771 1786562414.157777 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38267): Mirror 6: 0x49a887df (stale) Mirror 8: 0x49a887df (stale) Mirror 9: 0x49a887df (stale) Mirror 11: 0x49a887df (stale) Mirror 13: 0x49a887df (stale) Mirror 14: 0x49a887df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38267): Mirror 6: 0x49a887df (stale) Mirror 8: 0x49a887df (stale) Mirror 9: 0x49a887df (stale) Mirror 11: 0x49a887df (stale) Mirror 13: 0x49a887df (stale) Mirror 14: 0x49a887df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38267: skipped 000773 1786562414.548074 fallocate 0x01bc15 thru 0x01fe3d (0x4229 bytes) 000774 1786562414.565129 MIRROR_RESYNC 7 mirrors 000775 1786562415.630118 read 0x0040e8 thru 0x0100af (0xbfc8 bytes) 000776 1786562415.641212 trunc from 0x038267 to 0x01ad63 (0x1d503 bytes) 000777 1786562415.683938 mapread 0x014082 thru 0x01ad63 (0x6ce2 bytes) 000778 1786562415.698513 mapwrite 0x03fac4 thru 0x03ffff (0x53c bytes) 000779 1786562415.730259 read 0x002bdf thru 0x00b04c (0x846e bytes) 000780 1786562415.748451 MIRROR_RESYNC 7 mirrors 000781 1786562416.792507 mapwrite 0x00a8f7 thru 0x015b9a (0xb2a4 bytes) 000782 1786562416.904758 trunc from 0x040000 to 0x019edf (0x26120 bytes) 000783 1786562416.919247 mapread 0x00194b thru 0x006cdd (0x5393 bytes) 000784 1786562416.925716 write 0x03f5fc thru 0x03ffff (0xa04 bytes) 000785 1786562416.926786 mapread 0x028660 thru 0x02dd49 (0x56ea bytes) 000786 1786562416.933070 read 0x02e92d thru 0x0368c3 (0x7f97 bytes) 000787 1786562416.941158 mapwrite 0x015758 thru 0x01913b (0x39e4 bytes) 000788 1786562416.962169 read 0x012a1e thru 0x022856 (0xfe39 bytes) 000789 1786562416.970368 punch from 0x008541 to 0x009c11 (0x16d1 bytes) 000790 1786562416.976363 trunc from 0x040000 to 0x01d64b (0x229b4 bytes) 000792 1786562416.989463 trunc from 0x00d36e to 0x01770a (0xa39d bytes) 000793 1786562416.994245 trunc from 0x003767 to 0x00bf87 (0x8821 bytes) 000794 1786562416.998463 read 0x015c98 thru 0x01d64b (0x79b4 bytes) 000795 1786562417.004190 punch from 0x007eab to 0x00a4ca (0x2620 bytes) 000796 1786562417.012960 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1d64c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d64c): Mirror 6: 0xdd42f80d (stale) Mirror 8: 0xdd42f80d (stale) Mirror 9: 0xdd42f80d (stale) Mirror 11: 0xdd42f80d (stale) Mirror 13: 0xdd42f80d (stale) Mirror 14: 0xdd42f80d (stale) CRC-32 checksum value for chunk [0, 0x1d64c): Mirror 1: 0x2b4f7303 Mirror 15: 0x2b4f7303 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d64c: skipped 000797 1786562417.786686 mapwrite 0x00b256 thru 0x01300d (0x7db8 bytes) 000798 1786562417.882785 mapread 0x013d6e thru 0x01d64b (0x98de bytes) 000799 1786562417.895491 trunc from 0x01d64c to 0x026861 (0x9216 bytes) 000800 1786562417.919720 trunc from 0x005644 to 0x006f2b (0x18e8 bytes) 000801 1786562417.924442 trunc from 0x001454 to 0x00a748 (0x92f5 bytes) 000802 1786562417.929568 mapwrite 0x0059da thru 0x0107d8 (0xadff bytes) 000803 1786562417.955635 write 0x002c57 thru 0x004cd5 (0x207f bytes) 000804 1786562417.959118 read 0x0068f3 thru 0x00e11a (0x7828 bytes) 000805 1786562417.960302 write 0x009d39 thru 0x00a134 (0x3fc bytes) 000806 1786562417.961938 mapread 0x01af5a thru 0x01f5de (0x4685 bytes) 000807 1786562417.972935 write 0x03eefa thru 0x03ffff (0x1106 bytes) 000808 1786562417.974715 read 0x035e0f thru 0x036000 (0x1f2 bytes) 000810 1786562417.989698 write 0x01bb98 thru 0x02ac34 (0xf09d bytes) 000811 1786562417.992119 punch from 0x01cc3e to 0x02373b (0x6afe bytes) 000812 1786562418.014364 trunc from 0x027b84 to 0x02f1d2 (0x764f bytes) 000813 1786562418.018395 trunc from 0x022601 to 0x031462 (0xee62 bytes) 000814 1786562418.024790 punch from 0x0393d9 to 0x03ffff (0x6c27 bytes) 000815 1786562418.030611 mapwrite 0x03ad5e thru 0x03be22 (0x10c5 bytes) 000816 1786562418.051750 write 0x012ce7 thru 0x01b20e (0x8528 bytes) 000817 1786562418.058645 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x64dc55d9 (stale) Mirror 8: 0x64dc55d9 (stale) Mirror 9: 0x64dc55d9 (stale) Mirror 11: 0x64dc55d9 (stale) Mirror 14: 0x64dc55d9 (stale) Mirror 15: 0x7019a97c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x64dc55d9 (stale) Mirror 8: 0x64dc55d9 (stale) Mirror 9: 0x64dc55d9 (stale) Mirror 11: 0x64dc55d9 (stale) Mirror 14: 0x64dc55d9 (stale) Mirror 15: 0x7019a97c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000819 1786562418.533379 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x64dc55d9 (stale) Mirror 8: 0x64dc55d9 (stale) Mirror 9: 0x64dc55d9 (stale) Mirror 11: 0x64dc55d9 (stale) Mirror 14: 0x64dc55d9 (stale) Mirror 15: 0x7019a97c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae165533 Mirror 16: 0xae165533 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1786562419.439203 read 0x02c994 thru 0x031b0e (0x517b bytes) 000821 1786562419.450676 fallocate 0x0223eb thru 0x025b15 (0x372b bytes) 000823 1786562419.482230 mapwrite 0x0240d8 thru 0x024c7d (0xba6 bytes) 000824 1786562419.497608 write 0x01ab52 thru 0x026dc7 (0xc276 bytes) 000825 1786562419.514325 fallocate 0x02b0c9 thru 0x038b3d (0xda75 bytes) 000826 1786562419.519431 write 0x036599 thru 0x03ac59 (0x46c1 bytes) 000827 1786562419.527490 trunc from 0x040000 to 0x00323b (0x3cdc4 bytes) 000828 1786562419.559348 write 0x0229db thru 0x02d60e (0xac34 bytes) 000829 1786562419.565645 trunc from 0x0285ef to 0x02d60e (0x5020 bytes) 000830 1786562419.588726 write 0x0338ca thru 0x03ddc7 (0xa4fe bytes) 000832 1786562419.594334 trunc from 0x03ddc8 to 0x024700 (0x196c7 bytes) 000833 1786562419.616277 write 0x03dbff thru 0x03ffff (0x2401 bytes) 000834 1786562419.624030 trunc from 0x021ebe to 0x030a00 (0xeb43 bytes) 000835 1786562419.638637 write 0x030ddb thru 0x03cc60 (0xbe86 bytes) 000836 1786562419.651951 fallocate 0x0288f2 thru 0x032f0f (0xa61e bytes) 000837 1786562419.662948 MIRROR_RESYNC 8 mirrors 000838 1786562420.938956 trunc from 0x040000 to 0x003b97 (0x3c468 bytes) 000839 1786562420.961531 mapwrite 0x03fb42 thru 0x03ffff (0x4be bytes) 000840 1786562420.987290 mapread 0x02c42c thru 0x033b03 (0x76d8 bytes) 000841 1786562421.001622 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe4154972 (stale) Mirror 8: 0xe4154972 (stale) Mirror 9: 0xe4154972 (stale) Mirror 11: 0xe4154972 (stale) Mirror 14: 0xe4154972 (stale) Mirror 15: 0xe4154972 (stale) Mirror 16: 0xe4154972 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa3df2fb5 Mirror 17: 0xa3df2fb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1786562421.750010 trunc from 0x002fce to 0x0085bf (0x55f2 bytes) 000844 1786562421.772758 read 0x039e8e thru 0x03ffff (0x6172 bytes) 000845 1786562421.777592 punch from 0x030975 to 0x03fb6d (0xf1f9 bytes) 000846 1786562421.787576 mapwrite 0x017677 thru 0x01c9d1 (0x535b bytes) 000847 1786562421.829679 write 0x01c05e thru 0x01df76 (0x1f19 bytes) 000848 1786562421.834263 trunc from 0x040000 to 0x024f54 (0x1b0ab bytes) 000849 1786562421.851070 mapwrite 0x011647 thru 0x0152ab (0x3c65 bytes) 000850 1786562421.869423 punch from 0x01b247 to 0x024f54 (0x9d0e bytes) 000851 1786562421.877038 fallocate 0x02106a thru 0x027952 (0x68e9 bytes) 000853 1786562421.898660 mapwrite 0x01249e thru 0x01ea56 (0xc5b9 bytes) 000854 1786562421.941530 trunc from 0x024f55 to 0x01c4e7 (0x8a6d bytes) 000855 1786562421.950767 mapread 0x0182bd thru 0x01c34a (0x408e bytes) 000856 1786562421.952127 mapread 0x01b07d thru 0x01b08e (0x12 bytes) 000857 1786562421.953083 write 0x033e0d thru 0x03a560 (0x6754 bytes) 000858 1786562421.954567 trunc from 0x03a561 to 0x0331bf (0x73a1 bytes) 000859 1786562421.963956 trunc from 0x0331c0 to 0x01becd (0x172f2 bytes) 000860 1786562421.983334 write 0x014ddc thru 0x01907c (0x42a1 bytes) 000861 1786562421.984711 mapwrite 0x0265df thru 0x02710b (0xb2d bytes) 000862 1786562422.050609 mapread 0x016874 thru 0x01aca0 (0x442d bytes) 000865 1786562422.052990 trunc from 0x01ec2c to 0x0236b2 (0x4a87 bytes) 000867 1786562422.063578 trunc from 0x023a0b to 0x02710b (0x3701 bytes) 000868 1786562422.073563 mapwrite 0x021a6e thru 0x022d52 (0x12e5 bytes) 000869 1786562422.108445 mapread 0x00279f thru 0x009b7e (0x73e0 bytes) 000870 1786562422.127382 trunc from 0x02710c to 0x029bbf (0x2ab4 bytes) 000872 1786562422.148637 fallocate 0x03dd75 thru 0x03ffff (0x228b bytes) 000873 1786562422.161910 mapread 0x02740e thru 0x029bbf (0x27b2 bytes) 000874 1786562422.169245 fallocate 0x01cf42 thru 0x01f653 (0x2712 bytes) 000875 1786562422.174141 punch from 0x0274eb to 0x029bbf (0x26d5 bytes) 000876 1786562422.181005 fallocate 0x02ea68 thru 0x03762c (0x8bc5 bytes) 000877 1786562422.193151 fallocate 0x027875 thru 0x02f7f6 (0x7f82 bytes) 000878 1786562422.201701 read 0x00a565 thru 0x00d0d1 (0x2b6d bytes) 000879 1786562422.205279 read 0x016839 thru 0x0200cb (0x9893 bytes) 000880 1786562422.209606 mapwrite 0x00808c thru 0x0148e4 (0xc859 bytes) 000881 1786562422.249720 trunc from 0x019ad9 to 0x01b95c (0x1e84 bytes) 000882 1786562422.258225 read 0x014571 thru 0x017c5b (0x36eb bytes) 000883 1786562422.268953 MIRROR_RESYNC 9 mirrors 000884 1786562424.924394 mapread 0x01862d thru 0x019861 (0x1235 bytes) 000885 1786562424.935680 mapread 0x02d56e thru 0x03630a (0x8d9d bytes) 000886 1786562424.953652 trunc from 0x00f505 to 0x010fda (0x1ad6 bytes) 000887 1786562424.994613 punch from 0x0048d3 to 0x007b2e (0x325c bytes) 000888 1786562425.002876 punch from 0x034139 to 0x03762c (0x34f4 bytes) 000889 1786562425.009427 trunc from 0x0004fc to 0x000a86 (0x58b bytes) 000890 1786562425.017239 mapwrite 0x03a095 thru 0x03ffff (0x5f6b bytes) 000891 1786562425.061231 trunc from 0x030a33 to 0x03e2b7 (0xd885 bytes) 000892 1786562425.066185 punch from 0x02cd65 to 0x031d9c (0x5038 bytes) 000893 1786562425.070649 write 0x011041 thru 0x01d447 (0xc407 bytes) 000894 1786562425.086358 punch from 0x02843c to 0x02cae7 (0x46ac bytes) 000895 1786562425.095656 trunc from 0x040000 to 0x0153bd (0x2ac42 bytes) 000896 1786562425.135381 MIRROR_RESYNC 9 mirrors 000897 1786562426.679913 trunc from 0x0153be to 0x01ca50 (0x7693 bytes) 000898 1786562426.730037 fallocate 0x03f221 thru 0x03ffff (0xddf bytes) 000899 1786562426.734686 fallocate 0x034303 thru 0x038ae6 (0x47e4 bytes) 000900 1786562426.740395 trunc from 0x0224c0 to 0x02bd95 (0x98d6 bytes) 000901 1786562426.744447 mapread 0x00c463 thru 0x01044a (0x3fe8 bytes) 000902 1786562426.762700 trunc from 0x03450c to 0x03aa8d (0x6582 bytes) 000903 1786562426.768259 fallocate 0x02ad06 thru 0x0345a0 (0x989b bytes) 000904 1786562426.774036 trunc from 0x040000 to 0x02ecc7 (0x11338 bytes) 000906 1786562426.793709 write 0x0355e8 thru 0x03ffff (0xaa18 bytes) 000907 1786562426.795539 trunc from 0x022c12 to 0x02dc53 (0xb042 bytes) 000908 1786562426.813790 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fec936b (stale) Mirror 8: 0x1fec936b (stale) Mirror 9: 0x1fec936b (stale) Mirror 11: 0x1fec936b (stale) Mirror 14: 0x1fec936b (stale) Mirror 15: 0x1fec936b (stale) Mirror 16: 0x1fec936b (stale) Mirror 17: 0x1fec936b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf8aeef18 Mirror 18: 0xf8aeef18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1786562427.649347 mapwrite 0x038764 thru 0x03a606 (0x1ea3 bytes) 000910 1786562427.780766 write 0x02cf13 thru 0x03325f (0x634d bytes) 000911 1786562427.791774 write 0x007d7a thru 0x013f09 (0xc190 bytes) 000913 1786562427.803204 trunc from 0x040000 to 0x01ab74 (0x2548b bytes) 000914 1786562427.825124 punch from 0x016d90 to 0x01ab74 (0x3de5 bytes) 000915 1786562427.829538 punch from 0x00beb6 to 0x01440e (0x8559 bytes) 000916 1786562427.845586 punch from 0x008f47 to 0x012e17 (0x9ed1 bytes) 000917 1786562427.850325 punch from 0x00bba5 to 0x01aa83 (0xeedf bytes) 000918 1786562427.855488 punch from 0x006ebc to 0x0080d1 (0x1216 bytes) 000919 1786562427.867079 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1ab75): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab75): Mirror 6: 0x1c52e761 (stale) Mirror 8: 0x1c52e761 (stale) Mirror 9: 0x1c52e761 (stale) Mirror 11: 0x1c52e761 (stale) Mirror 14: 0x1c52e761 (stale) Mirror 15: 0x1c52e761 (stale) Mirror 16: 0x1c52e761 (stale) Mirror 17: 0x1c52e761 (stale) Mirror 18: 0x1c52e761 (stale) CRC-32 checksum value for chunk [0, 0x1ab75): Mirror 1: 0x84140506 Mirror 19: 0x84140506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab75: skipped 000920 1786562428.790585 trunc from 0x000662 to 0x0058a9 (0x5248 bytes) 000922 1786562428.836967 trunc from 0x01204c to 0x019103 (0x70b8 bytes) 000923 1786562428.841186 trunc from 0x01ab75 to 0x03675a (0x1bbe6 bytes) 000924 1786562428.862204 write 0x03ac75 thru 0x03ffff (0x538b bytes) 000925 1786562428.864993 mapwrite 0x0049ee thru 0x0144c1 (0xfad4 bytes) 000926 1786562428.912594 read 0x018d42 thru 0x01b0d8 (0x2397 bytes) 000927 1786562428.918246 trunc from 0x040000 to 0x00e738 (0x318c7 bytes) 000928 1786562428.937393 mapread 0x001204 thru 0x005f42 (0x4d3f bytes) 000929 1786562428.939059 trunc from 0x00c983 to 0x00e738 (0x1db6 bytes) 000930 1786562428.944635 punch from 0x006d86 to 0x009675 (0x28f0 bytes) 000931 1786562428.951546 trunc from 0x00783f to 0x00c248 (0x4a0a bytes) 000932 1786562428.959013 mapwrite 0x02b5de thru 0x033d2a (0x874d bytes) 000933 1786562429.003317 mapread 0x01bd7a thru 0x01f293 (0x351a bytes) 000934 1786562429.011596 fallocate 0x000c58 thru 0x00ea0f (0xddb8 bytes) 000935 1786562429.016646 write 0x00983a thru 0x00e039 (0x4800 bytes) 000937 1786562429.029343 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x33d2b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d2b): Mirror 6: 0xed8af4c0 (stale) Mirror 8: 0xed8af4c0 (stale) Mirror 9: 0xed8af4c0 (stale) Mirror 11: 0xed8af4c0 (stale) Mirror 14: 0xed8af4c0 (stale) Mirror 15: 0xed8af4c0 (stale) Mirror 16: 0xed8af4c0 (stale) Mirror 17: 0xed8af4c0 (stale) Mirror 18: 0xed8af4c0 (stale) Mirror 19: 0xead7567e (stale) CRC-32 checksum value for chunk [0, 0x33d2b): Mirror 1: 0x3494b4d3 Mirror 20: 0x3494b4d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d2b: skipped 000939 1786562429.845490 trunc from 0x01d975 to 0x023c91 (0x631d bytes) 000940 1786562429.884190 write 0x006c57 thru 0x016717 (0xfac1 bytes) 000942 1786562429.913334 mapwrite 0x021710 thru 0x0267c5 (0x50b6 bytes) 000943 1786562429.965707 write 0x003698 thru 0x011503 (0xde6c bytes) 000944 1786562429.974580 write 0x01e6d0 thru 0x01e8e4 (0x215 bytes) 000945 1786562429.977189 fallocate 0x023ae3 thru 0x027186 (0x36a4 bytes) 000946 1786562429.983999 trunc from 0x033d2b to 0x00a8f6 (0x29434 bytes) 000948 1786562430.020344 trunc from 0x00a8f7 to 0x01795f (0xd069 bytes) 000949 1786562430.035117 write 0x020ede thru 0x02a844 (0x9967 bytes) 000951 1786562430.036898 read 0x01e122 thru 0x01ea0f (0x8ee bytes) 000952 1786562430.041560 fallocate 0x03fd87 thru 0x03ffff (0x279 bytes) 000953 1786562430.048266 write 0x0268fd thru 0x03483e (0xdf42 bytes) 000954 1786562430.050877 read 0x024a8e thru 0x025ff3 (0x1566 bytes) 000955 1786562430.051550 mapwrite 0x01b579 thru 0x023494 (0x7f1c bytes) 000956 1786562430.074541 read 0x01d868 thru 0x02d245 (0xf9de bytes) 000957 1786562430.075383 read 0x03403d thru 0x03483e (0x802 bytes) 000958 1786562430.076030 mapwrite 0x0235d7 thru 0x0290f5 (0x5b1f bytes) 000959 1786562430.092546 mapwrite 0x01f667 thru 0x028bbe (0x9558 bytes) 000960 1786562430.107679 trunc from 0x02be16 to 0x03483e (0x8a29 bytes) 000962 1786562430.111939 punch from 0x018537 to 0x01a850 (0x231a bytes) 000963 1786562430.115461 write 0x00412d thru 0x00472b (0x5ff bytes) 000964 1786562430.116234 trunc from 0x03483f to 0x01c13b (0x18703 bytes) 000965 1786562430.126610 trunc from 0x01c0c7 to 0x01c13b (0x75 bytes) 000967 1786562430.131112 write 0x02be47 thru 0x039541 (0xd6fb bytes) 000968 1786562430.133378 trunc from 0x039542 to 0x01659b (0x22fa6 bytes) 000969 1786562430.151407 MIRROR_RESYNC 12 mirrors 000970 1786562431.899890 MIRROR_RESYNC 12 mirrors 000971 1786562432.332592 fallocate 0x03e395 thru 0x03ffff (0x1c6b bytes) 000972 1786562432.374682 write 0x0160fc thru 0x01d9b3 (0x78b8 bytes) 000973 1786562432.384072 trunc from 0x040000 to 0x036901 (0x96fe bytes) 000974 1786562432.400024 fallocate 0x00c825 thru 0x01a10d (0xd8e9 bytes) 000975 1786562432.412471 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x36902): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36902): Mirror 6: 0x320baa57 (stale) Mirror 8: 0x320baa57 (stale) Mirror 9: 0x320baa57 (stale) Mirror 11: 0x320baa57 (stale) Mirror 14: 0x320baa57 (stale) Mirror 15: 0x320baa57 (stale) Mirror 16: 0x320baa57 (stale) Mirror 17: 0x320baa57 (stale) Mirror 18: 0x320baa57 (stale) Mirror 19: 0x320baa57 (stale) Mirror 20: 0x320baa57 (stale) CRC-32 checksum value for chunk [0, 0x36902): Mirror 1: 0xd55629b8 Mirror 21: 0xd55629b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36902: skipped 000976 1786562433.161880 MIRROR_RESYNC 13 mirrors 000977 1786562434.951268 trunc from 0x036902 to 0x02efe9 (0x7918 bytes) 000978 1786562435.016018 trunc from 0x00be26 to 0x00db7c (0x1d57 bytes) 000979 1786562435.024608 trunc from 0x02efea to 0x023465 (0xbb84 bytes) 000980 1786562435.061937 mapwrite 0x009d90 thru 0x013043 (0x92b4 bytes) 000981 1786562435.117915 mapread 0x016214 thru 0x020115 (0x9f02 bytes) 000982 1786562435.173966 write 0x004ba4 thru 0x012979 (0xddd6 bytes) 000983 1786562435.176675 mapread 0x00ca0a thru 0x00fc73 (0x326a bytes) 000984 1786562435.178293 read 0x00b544 thru 0x01904e (0xdb0b bytes) 000985 1786562435.179092 fallocate 0x01dec2 thru 0x022c08 (0x4d47 bytes) 000986 1786562435.184325 trunc from 0x002928 to 0x00510c (0x27e5 bytes) 000987 1786562435.198428 fallocate 0x018bb2 thru 0x01f3ea (0x6839 bytes) 000988 1786562435.203568 punch from 0x01e0af to 0x023465 (0x53b7 bytes) 000989 1786562435.211130 write 0x02d98d thru 0x030d2b (0x339f bytes) 000991 1786562435.213198 punch from 0x0127dd to 0x01ad19 (0x853d bytes) 000992 1786562435.220109 trunc from 0x030d2c to 0x0360cb (0x53a0 bytes) 000994 1786562435.232665 trunc from 0x0360cc to 0x03bce5 (0x5c1a bytes) 000995 1786562435.258676 punch from 0x029ee7 to 0x0329a5 (0x8abf bytes) 000996 1786562435.270878 mapread 0x0276ce thru 0x0298a0 (0x21d3 bytes) 000997 1786562435.281383 trunc from 0x025f77 to 0x035559 (0xf5e3 bytes) 000999 1786562435.287797 write 0x0036ad thru 0x010aea (0xd43e bytes) 001000 1786562435.296758 read 0x0049ee thru 0x00a52e (0x5b41 bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 15:20:39 (1786562439) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 15:20:47 (1786562447) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5dd:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c1:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237799 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5dd:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5de:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c1:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12309 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5dd:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5de:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c2:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c1:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5dd:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5de:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c2:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c1:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5df:0x0] } lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 15:20:54 (1786562454) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre2 Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3804352 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6358016 .. done Extending file size to 8941152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8239776 .. done Extending file size to 8344064 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8428128 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8700832 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64950 64951 64953 64954 64955 done 192.168.202.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre2 (opts:) 192.168.202.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:22:19 (1786562539) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre2 Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2090240 .. Extending file size to 6794272 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7074112 .. failed Extending file size to 5475808 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9104384 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6247744 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6278912 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7520160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8512960 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807072 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 67021 67022 67023 67025 67026 failed 192.168.202.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre2 (opts:) 192.168.202.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:23:48 (1786562628) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2936 1268368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 54304 3552716 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 34100 3572920 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 88404 7125636 2% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2936 1268368 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 54304 3552716 2% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 34100 3572920 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 88404 7125636 2% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300598 s, 7.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294848 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x94226803 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:24:11 (1786562651) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0900089 s, 23.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 15:24:19 (1786562659) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x411:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5d8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5d8:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5d9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x5f4:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:24:28 (1786562668) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790518 s, 26.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:24:37 (1786562677) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142099 s, 14.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:24:45 (1786562685) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349623 s, 6.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5da:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f5:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5da:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f5:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f6:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:25:21 (1786562721) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg202-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167142 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5db:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f8:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f7:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5db:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f8:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f7:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5dc:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:26:18 (1786562778) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg202-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156202 s, 13.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:27:13 (1786562833) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg202-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg202-server: Pool lustre.test_204f created Adding targets to pool oleg202-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg202-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149741 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f9:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5dd:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5f9:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5dd:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg202-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg202-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg202-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:28:32 (1786562912) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:28:43 (1786562923) lfs mirror mirror: Invalid layout: Wrong flags error: lfs extend: /mnt/lustre/f205b.sanity-flr: cannot create volatile file: Invalid argument sanity-flr test_205b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4398:test_205b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4415:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_205b.*.1786562927.log rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.debug_log.oleg202-server.1786562927.log.JDwraG" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.dmesg.oleg202-server.1786562927.log.4HQFnM" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 205b (12s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:28:54 (1786562934) oleg202-server: Pool lustre.test_206 created Adding targets to pool oleg202-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg202-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg202-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg202-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg202-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:29:24 (1786562964) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105938 s, 9.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719926 s, 14.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x53b94ca5 Mirror 3: 0x53b94ca5 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:29:36 (1786562976) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:29:41 (1786562981) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (6s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:29:47 (1786562987) Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0632589 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2140000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xdf0000 Read this (4 bytes): BBBB 192.168.202.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg202-client.virtnet /mnt/lustre2 (opts:) PASS 209a (52s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:30:40 (1786563040) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0473142 s, 173 kB/s the very first read read with warm cache PASS 209b (14s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:30:54 (1786563054) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887593 s, 11.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x42e lmm_fid: [0x200000405:0x42e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x42e lmm_fid: [0x200000405:0x42e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5e6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612206 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 15:30:57.000000000 -0400 Modify: 2026-08-12 15:30:57.000000000 -0400 Change: 2026-08-12 15:30:59.000000000 -0400 Birth: 2026-08-12 15:30:58.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:31:06 (1786563066) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530715 s, 19.8 MB/s oleg202-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg202-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg202-server: O/280000401/d3/603: File not found by ext2_lookup PASS 210b (34s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:31:40 (1786563100) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.533927 s, 76.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0332298 s, 123 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 15:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x433:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 15:31 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:31:54 (1786563114) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (12s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:32:06 (1786563126) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (17s) == sanity-flr test complete, duration 1414 sec =========== 15:32:23 (1786563143) sanity-flr: FAIL: test_0m failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr sanity-flr: FAIL: test_205b /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed === sanity-flr: start cleanup 15:32:26 (1786563146) === === sanity-flr: finish cleanup 15:32:47 (1786563167) ===