-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 15:56:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg221-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:57:13 (1786564633) === oleg221-client.virtnet: executing check_config_client /mnt/lustre oleg221-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg221-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94c507115800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94c507115800.idle_timeout=debug disable quota as required oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:57:57 (1786564677) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 15:57:59 (1786564679) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:58:18 (1786564698) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:58:21 (1786564701) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:58:24 (1786564704) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:58:44 (1786564724) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:58:49 (1786564729) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (6s) == sanity-flr test 0g: lfs mirror create flags support === 15:58:55 (1786564735) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94c507115800.stats=clear osc.lustre-OST0001-osc-ffff94c507115800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:59:04 (1786564744) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:59:15 (1786564755) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:59:24 (1786564764) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:59:28 (1786564768) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created error: lfs extend: layout is NULL /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000402 lmm_object_id: 0x6 lmm_fid: [0x240000402:0x6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xf:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000402 lmm_object_id: 0x6 lmm_fid: [0x240000402:0x6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x10:0x0] } sanity-flr test_0m: @@@@@@ FAIL: failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:899:test_0m() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:909:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0m.*.1786564773.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.debug_log.oleg221-server.1786564773.log.XYzt4i" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0m.dmesg.oleg221-server.1786564773.log.Jgfl4d" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:59:38 (1786564778) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x11:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 15:59:49 (1786564789) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:59:57 (1786564797) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:00:06 (1786564806) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:00:15 (1786564815) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:00:25 (1786564825) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:00:32 (1786564832) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.375687 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12066 s, 8.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:00:41 (1786564841) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.376072 s, 27.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:00:51 (1786564851) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:01:04 (1786564864) Stopping /mnt/lustre-ost1 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg221-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg221-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (85s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:02:29 (1786564949) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.693878 s, 13.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg221-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:03:16 (1786564996) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg221-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg221-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:04:41 (1786565081) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.21972 s, 16.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 16:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.416s user 0m0.000s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.089s user 0m0.001s sys 0m0.172s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg221-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg221-server oleg221-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.420s user 0m0.001s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg221-server: oleg221-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg221-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg221-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c507115800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:06:16 (1786565176) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:06:19 (1786565179) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:06:23 (1786565183) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:06:26 (1786565186) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 16:06:35 (1786565195) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.435826 s, 28.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:06:46 (1786565206) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254322 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429788 s, 24.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:06:54 (1786565214) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:07:03 (1786565223) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00187732 s, 545 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125111 s, 818 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149123 s, 687 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000946417 s, 1.1 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 16:07:15 (1786565235) create test files with size 15730296 .. 079694dfc0cdf450656ec927f3cff92d - 9a44f41bf6903c2f6b8aecc44f6a9a0c - 41560169510474aa5af61246ae8ebc23 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg221-client.virtnet /mnt/lustre (opts:) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 16:07:43 (1786565263) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:07:57 (1786565277) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:08:05 (1786565285) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773052 s, 13.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025596 s, 41.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 16:08:15 (1786565295) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0768571 s, 27.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.385337 s, 10.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236488 s, 8.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 16:08:34 (1786565314) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 16:08:38 (1786565318) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:08:42 (1786565322) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280447 s, 37.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 16:08:52 (1786565332) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.383168 s, 8.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:09:02 (1786565342) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:09:14 (1786565354) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.483725 s, 21.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg221-client.virtnet /mnt/lustre (opts:) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:09:28 (1786565368) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff94c509298000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.315288 s, 33.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg221-client.virtnet /mnt/lustre (opts:) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff94c514a88000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:09:47 (1786565387) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:10:35 (1786565435) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:11:18 (1786565478) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:11:22 (1786565482) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286438 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209517 s, 15.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd59597d5 (nosync,stale) Mirror 2: 0xd59597d5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2b11e69790ca6ef79382871f5a2c0539 - mirror 1 checksum: 39a752e30018cb87e3e19ed36f3f2357 - mirror 2 checksum: 2b11e69790ca6ef79382871f5a2c0539 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T16:11:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T16:11:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2b11e69790ca6ef79382871f5a2c0539 - mirror 1 checksum: 39a752e30018cb87e3e19ed36f3f2357 - mirror 2 checksum: 39a752e30018cb87e3e19ed36f3f2357 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T16:11:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T16:11:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:11:35 (1786565495) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385404 s, 27.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:11:44 (1786565504) Failing mds1 on oleg221-server Stopping /mnt/lustre-mds1 (opts:) on oleg221-server 16:11:58 (1786565518) shut down facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server Failover mds1 to oleg221-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 16:12:35 (1786565555) targets are mounted 16:12:35 (1786565555) facet_failover done oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263307 s, 39.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (70s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:12:54 (1786565574) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0583292 s, 351 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0269288 s, 761 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:13:04 (1786565584) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0785489 s, 261 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0101758 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:13:16 (1786565596) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00730375 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:13:25 (1786565605) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0557009 s, 368 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:13:34 (1786565614) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00952719 s, 430 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0237397 s, 173 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:13:43 (1786565623) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:14:07 (1786565647) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:14:36 (1786565676) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:15:02 (1786565702) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:15:26 (1786565726) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 16:15:34 (1786565734) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57585 mirror split pid 57586 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57585 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57586 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 16:16:42 (1786565802) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5804 fallocating to largest ever: 0x11bcc 000001 1786565804.943568 fallocate 0x006c6e thru 0x011bcb (0xaf5e bytes) fallocating to largest ever: 0x3cf10 000002 1786565804.958976 fallocate 0x03af4b thru 0x03cf0f (0x1fc5 bytes) 000003 1786565804.967778 write 0x00ee02 thru 0x011f04 (0x3103 bytes) 000004 1786565804.986686 MIRROR_EXTEND to 2 mirrors 000005 1786565805.670085 trunc from 0x0228e1 to 0x02c2e3 (0x9a03 bytes) 000006 1786565805.709075 mapread 0x00a874 thru 0x01541b (0xaba8 bytes) 000007 1786565805.734944 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000008 1786565805.926998 mapread 0x03a1c0 thru 0x03cf0f (0x2d50 bytes) 000010 1786565805.950045 read 0x01a810 thru 0x028545 (0xdd36 bytes) 000011 1786565805.971586 trunc from 0x03cf10 to 0x000f62 (0x3bfad bytes) 000013 1786565806.002592 read 0x000c9b thru 0x000f62 (0x2c8 bytes) 000015 1786565806.024021 fallocate 0x010d60 thru 0x01df70 (0xd211 bytes) 000016 1786565806.030791 punch from 0x015a6c to 0x01df70 (0x8505 bytes) 000017 1786565806.038021 read 0x009a99 thru 0x00c899 (0x2e01 bytes) 000018 1786565806.051818 MIRROR_EXTEND to 2 mirrors 000019 1786565806.609786 read 0x01d624 thru 0x01df70 (0x94d bytes) 000020 1786565806.624572 punch from 0x018bf3 to 0x01cb26 (0x3f34 bytes) 000021 1786565806.670978 punch from 0x00767d to 0x00b16e (0x3af2 bytes) fallocating to largest ever: 0x40000 000022 1786565806.677754 fallocate 0x03eeea thru 0x03ffff (0x1116 bytes) 000023 1786565806.687686 MIRROR_RESYNC 2 mirrors 000024 1786565807.409154 trunc from 0x00dc0a to 0x013b1c (0x5f13 bytes) 000025 1786565807.451592 mapread 0x008a14 thru 0x015281 (0xc86e bytes) 000026 1786565807.483058 fallocate 0x029194 thru 0x038115 (0xef82 bytes) 000027 1786565807.488882 read 0x02d7f2 thru 0x03122d (0x3a3c bytes) 000028 1786565807.498964 trunc from 0x038116 to 0x0023e3 (0x35d32 bytes) 000029 1786565807.523411 fallocate 0x0225a3 thru 0x0243e5 (0x1e43 bytes) 000030 1786565807.530929 mapread 0x002290 thru 0x0023e3 (0x154 bytes) 000031 1786565807.538962 read 0x0018f6 thru 0x0023e3 (0xaee bytes) 000032 1786565807.539827 read 0x001fa4 thru 0x0023e3 (0x440 bytes) 000034 1786565807.542187 mapread 0x001afe thru 0x001d10 (0x213 bytes) 000035 1786565807.543952 trunc from 0x0023e4 to 0x006b73 (0x4790 bytes) 000036 1786565807.566014 trunc from 0x00072b to 0x003c12 (0x34e8 bytes) 000037 1786565807.572602 mapread 0x001154 thru 0x005a18 (0x48c5 bytes) 000038 1786565807.583951 write 0x01c879 thru 0x027d15 (0xb49d bytes) 000039 1786565807.589950 write 0x02c323 thru 0x02dd4d (0x1a2b bytes) 000040 1786565807.596277 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000041 1786565807.749337 read 0x024b9b thru 0x027fcf (0x3435 bytes) 000042 1786565807.758767 trunc from 0x02dd4e to 0x013971 (0x1a3dc bytes) 000043 1786565807.777526 mapread 0x012eec thru 0x013971 (0xa86 bytes) 000044 1786565807.791129 trunc from 0x013972 to 0x00aec4 (0x8aad bytes) 000045 1786565807.805892 mapwrite 0x03a939 thru 0x03f14c (0x4814 bytes) 000046 1786565807.865278 punch from 0x01df60 to 0x027f98 (0xa039 bytes) 000048 1786565807.872119 read 0x0056b2 thru 0x00c478 (0x6dc7 bytes) 000049 1786565807.886982 punch from 0x013035 to 0x01461e (0x15ea bytes) 000050 1786565807.891590 mapread 0x01cfb5 thru 0x02b4ad (0xe4f9 bytes) 000052 1786565807.897324 read 0x0022d3 thru 0x007d41 (0x5a6f bytes) 000053 1786565807.898665 fallocate 0x0019c3 thru 0x0118d7 (0xff15 bytes) 000054 1786565807.904932 trunc from 0x00060a to 0x00f170 (0xeb67 bytes) 000055 1786565807.913045 read 0x017c72 thru 0x0188df (0xc6e bytes) 000056 1786565807.914579 mapwrite 0x001448 thru 0x002a61 (0x161a bytes) 000057 1786565807.935230 mapwrite 0x03f4e6 thru 0x03ffff (0xb1a bytes) 000058 1786565807.960176 trunc from 0x018ba7 to 0x023e79 (0xb2d3 bytes) 000059 1786565807.966817 trunc from 0x040000 to 0x002b96 (0x3d469 bytes) 000060 1786565807.987393 trunc from 0x002b97 to 0x03ac9e (0x38108 bytes) 000061 1786565808.009103 read 0x01b27a thru 0x02b17c (0xff03 bytes) 000063 1786565808.020829 trunc from 0x03ac9f to 0x02bd28 (0xef76 bytes) 000064 1786565808.034677 read 0x003801 thru 0x007a32 (0x4232 bytes) 000065 1786565808.043105 MIRROR_EXTEND to 2 mirrors 000066 1786565808.771729 read 0x00febf thru 0x01a98c (0xaace bytes) 000067 1786565808.792142 fallocate 0x034042 thru 0x03919f (0x515e bytes) 000068 1786565808.822594 trunc from 0x0391a0 to 0x0176ef (0x21ab0 bytes) 000069 1786565808.867581 write 0x01b152 thru 0x01def1 (0x2da0 bytes) 000070 1786565808.868984 mapread 0x019ddb thru 0x01b597 (0x17bd bytes) 000071 1786565808.884295 read 0x00ed38 thru 0x0113f9 (0x26c2 bytes) 000072 1786565808.895090 read 0x013dea thru 0x01def1 (0xa108 bytes) 000073 1786565808.897605 read 0x00d1a6 thru 0x01a68c (0xd4e7 bytes) 000074 1786565808.898960 mapwrite 0x01e759 thru 0x026610 (0x7eb8 bytes) 000075 1786565808.961132 trunc from 0x026611 to 0x01ed90 (0x7880 bytes) 000076 1786565808.976094 punch from 0x01cc54 to 0x01ed90 (0x213d bytes) 000077 1786565808.983497 trunc from 0x015f4d to 0x01cc7c (0x6d30 bytes) 000078 1786565808.993596 MIRROR_RESYNC 2 mirrors 000079 1786565809.610789 mapwrite 0x00fc72 thru 0x014190 (0x451f bytes) 000080 1786565809.699666 write 0x02a3d7 thru 0x039217 (0xee41 bytes) 000081 1786565809.703659 mapwrite 0x00f5ab thru 0x019b9e (0xa5f4 bytes) 000082 1786565809.734226 mapread 0x0011d0 thru 0x0079f0 (0x6821 bytes) 000085 1786565809.750467 fallocate 0x01a53f thru 0x021891 (0x7353 bytes) 000087 1786565809.757774 punch from 0x00c99e to 0x0120d9 (0x573c bytes) 000088 1786565809.763703 mapwrite 0x01b9a4 thru 0x01ede5 (0x3442 bytes) 000091 1786565809.780624 fallocate 0x010614 thru 0x019f4f (0x993c bytes) 000092 1786565809.786042 fallocate 0x02b59b thru 0x039f8a (0xe9f0 bytes) 000093 1786565809.791074 fallocate 0x02bf06 thru 0x036988 (0xaa83 bytes) 000094 1786565809.802896 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39f8b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f8b): Mirror 2: 0xb8d9ee45 (stale) CRC-32 checksum value for chunk [0, 0x39f8b): Mirror 1: 0xcddc2f01 Mirror 3: 0xcddc2f01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f8b: skipped 000095 1786565810.808443 trunc from 0x00247e to 0x00d1e3 (0xad66 bytes) 000096 1786565810.871022 fallocate 0x034a1c thru 0x03e96f (0x9f54 bytes) 000098 1786565810.882781 trunc from 0x039f8b to 0x014412 (0x25b78 bytes) 000099 1786565810.929323 mapwrite 0x023cc8 thru 0x032234 (0xe56d bytes) 000100 1786565811.087671 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32235): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32235): Mirror 2: 0x935b71e8 (stale) Mirror 3: 0xcc5d04a1 (stale) CRC-32 checksum value for chunk [0, 0x32235): Mirror 1: 0xd043c56a Mirror 4: 0xd043c56a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32235: skipped 000101 1786565812.008212 write 0x00ccbc thru 0x0154ea (0x882f bytes) 000102 1786565812.072968 trunc from 0x0319f5 to 0x032234 (0x840 bytes) 000103 1786565812.079463 write 0x027bf8 thru 0x033e46 (0xc24f bytes) 000104 1786565812.086290 write 0x01b74f thru 0x02ad7d (0xf62f bytes) 000105 1786565812.092895 mapwrite 0x034ea9 thru 0x03ccba (0x7e12 bytes) 000106 1786565812.164593 MIRROR_RESYNC 4 mirrors 000107 1786565813.545785 mapwrite 0x01a590 thru 0x028692 (0xe103 bytes) 000108 1786565813.673762 write 0x0198fc thru 0x01e6af (0x4db4 bytes) 000109 1786565813.677909 mapread 0x033074 thru 0x03724e (0x41db bytes) 000110 1786565813.694521 fallocate 0x0263df thru 0x036193 (0xfdb5 bytes) 000111 1786565813.700631 mapread 0x001ad5 thru 0x00cdd4 (0xb300 bytes) 000112 1786565813.712762 trunc from 0x03ccbb to 0x03dc0c (0xf52 bytes) 000113 1786565813.740965 fallocate 0x03dbe2 thru 0x03ffff (0x241e bytes) 000114 1786565813.746059 write 0x01fca0 thru 0x02c71f (0xca80 bytes) 000115 1786565813.747291 read 0x01d91f thru 0x01e98c (0x106e bytes) 000116 1786565813.747769 mapwrite 0x003907 thru 0x008193 (0x488d bytes) 000117 1786565813.770272 mapread 0x02e282 thru 0x02e672 (0x3f1 bytes) 000119 1786565813.772131 punch from 0x01ec40 to 0x02a6f2 (0xbab3 bytes) 000120 1786565813.783215 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa1b4d94a (stale) Mirror 3: 0xa1b4d94a (stale) Mirror 4: 0xa1b4d94a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d98b78b Mirror 5: 0x4d98b78b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000121 1786565815.136509 MIRROR_RESYNC 5 mirrors 000122 1786565816.690468 fallocate 0x03f2a3 thru 0x03ffff (0xd5d bytes) 000123 1786565816.730313 punch from 0x025759 to 0x02a909 (0x51b1 bytes) 000124 1786565816.738435 fallocate 0x01d0e1 thru 0x0276dd (0xa5fd bytes) 000125 1786565816.757498 trunc from 0x03adfa to 0x03ffff (0x5206 bytes) 000126 1786565816.765740 mapread 0x00c831 thru 0x011ff7 (0x57c7 bytes) 000127 1786565816.790424 trunc from 0x004c9d to 0x00e49f (0x9803 bytes) 000128 1786565816.801955 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4d98b78b (stale) Mirror 3: 0x4d98b78b (stale) Mirror 4: 0x4d98b78b (stale) Mirror 5: 0x4d98b78b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8fc2d20 Mirror 6: 0xb8fc2d20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000129 1786565817.866703 punch from 0x005fef to 0x00abf1 (0x4c03 bytes) 000130 1786565817.908593 trunc from 0x040000 to 0x03094d (0xf6b2 bytes) 000131 1786565817.937190 trunc from 0x021d69 to 0x024d70 (0x3008 bytes) 000132 1786565817.943507 write 0x03a06f thru 0x03ffff (0x5f91 bytes) 000133 1786565817.945302 write 0x03a812 thru 0x03ffff (0x57ee bytes) 000134 1786565817.948008 mapwrite 0x022838 thru 0x029257 (0x6a20 bytes) 000135 1786565817.992520 mapread 0x01cf66 thru 0x02a9eb (0xda86 bytes) 000136 1786565818.003227 mapwrite 0x03994f thru 0x03efb2 (0x5664 bytes) 000137 1786565818.032519 write 0x02a7f5 thru 0x02f951 (0x515d bytes) 000138 1786565818.039285 read 0x009dd9 thru 0x01985c (0xfa84 bytes) 000139 1786565818.050759 write 0x007efd thru 0x011258 (0x935c bytes) 000140 1786565818.056923 write 0x01448f thru 0x01b19e (0x6d10 bytes) 000141 1786565818.060005 trunc from 0x017d1a to 0x024d76 (0xd05d bytes) 000142 1786565818.109935 fallocate 0x000a76 thru 0x00c8ca (0xbe55 bytes) 000143 1786565818.119439 punch from 0x03bb02 to 0x03ffff (0x44fe bytes) 000144 1786565818.128446 read 0x02563c thru 0x02ce31 (0x77f6 bytes) 000145 1786565818.132472 read 0x019d45 thru 0x022b3c (0x8df8 bytes) 000147 1786565818.153203 trunc from 0x000b08 to 0x0061e9 (0x56e2 bytes) 000148 1786565818.159689 mapwrite 0x029c25 thru 0x0347b3 (0xab8f bytes) 000149 1786565818.204796 trunc from 0x00fe1f to 0x010129 (0x30b bytes) 000151 1786565818.224298 trunc from 0x02c529 to 0x031844 (0x531c bytes) 000153 1786565818.232255 read 0x0159df thru 0x017c16 (0x2238 bytes) 000154 1786565818.234993 write 0x011999 thru 0x01e199 (0xc801 bytes) 000156 1786565818.256734 MIRROR_RESYNC 6 mirrors 000157 1786565820.314739 punch from 0x0310aa to 0x03384f (0x27a6 bytes) 000158 1786565820.374313 read 0x015461 thru 0x024fd9 (0xfb79 bytes) 000159 1786565820.391684 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9d9756ac (stale) Mirror 4: 0x9d9756ac (stale) Mirror 5: 0x9d9756ac (stale) Mirror 6: 0x9d9756ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9d9756ac (stale) Mirror 4: 0x9d9756ac (stale) Mirror 5: 0x9d9756ac (stale) Mirror 6: 0x9d9756ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1786565820.878073 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9d9756ac (stale) Mirror 5: 0x9d9756ac (stale) Mirror 6: 0x9d9756ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9d9756ac (stale) Mirror 5: 0x9d9756ac (stale) Mirror 6: 0x9d9756ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000161 1786565821.333500 trunc from 0x030715 to 0x03997e (0x926a bytes) 000162 1786565821.338763 mapread 0x00d507 thru 0x00ed6e (0x1868 bytes) 000163 1786565821.358746 punch from 0x03e413 to 0x03ffff (0x1bed bytes) 000164 1786565821.364663 mapwrite 0x01d3c1 thru 0x027ad5 (0xa715 bytes) 000165 1786565821.404832 fallocate 0x01d535 thru 0x02a7e6 (0xd2b2 bytes) 000166 1786565821.410669 mapwrite 0x017819 thru 0x01d916 (0x60fe bytes) 000167 1786565821.431169 mapwrite 0x01dcbe thru 0x01ee3c (0x117f bytes) 000168 1786565821.443758 read 0x033aa4 thru 0x03af82 (0x74df bytes) 000169 1786565821.454256 punch from 0x0251db to 0x031616 (0xc43c bytes) 000172 1786565821.469487 mapwrite 0x03e7b9 thru 0x03ffff (0x1847 bytes) 000173 1786565821.492031 trunc from 0x040000 to 0x017e3c (0x281c3 bytes) 000174 1786565821.512283 trunc from 0x017e3d to 0x005737 (0x12705 bytes) 000175 1786565821.535963 punch from 0x0037be to 0x005737 (0x1f7a bytes) 000176 1786565821.541742 fallocate 0x031f5c thru 0x0362f4 (0x4399 bytes) 000177 1786565821.554402 write 0x017a4b thru 0x0189a5 (0xf5b bytes) 000179 1786565821.561596 fallocate 0x03a36c thru 0x03ffff (0x5c94 bytes) 000180 1786565821.570563 trunc from 0x00d446 to 0x0186b7 (0xb272 bytes) 000181 1786565821.608205 mapwrite 0x010cd8 thru 0x012982 (0x1cab bytes) 000182 1786565821.640213 write 0x00a973 thru 0x00e56f (0x3bfd bytes) 000183 1786565821.644891 trunc from 0x0189a6 to 0x028d2e (0x10389 bytes) 000184 1786565821.663560 fallocate 0x0095ce thru 0x011434 (0x7e67 bytes) 000185 1786565821.675252 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28d2f): Mirror 5: 0x388cd984 (stale) Mirror 6: 0x388cd984 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d2f): Mirror 5: 0x388cd984 (stale) Mirror 6: 0x388cd984 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d2f: skipped 000186 1786565822.001000 write 0x0280d5 thru 0x0362f7 (0xe223 bytes) 000187 1786565822.013637 fallocate 0x01f755 thru 0x029be3 (0xa48f bytes) 000188 1786565822.022286 write 0x0091af thru 0x00e188 (0x4fda bytes) 000189 1786565822.033182 punch from 0x00ba72 to 0x018821 (0xcdb0 bytes) 000190 1786565822.053828 write 0x02d9cb thru 0x03d1b4 (0xf7ea bytes) 000191 1786565822.057416 fallocate 0x0314dd thru 0x0344ea (0x300e bytes) 000192 1786565822.062041 fallocate 0x00c638 thru 0x01779b (0xb164 bytes) 000193 1786565822.067556 fallocate 0x02c6dd thru 0x02d360 (0xc84 bytes) 000194 1786565822.072601 read 0x02da30 thru 0x034c8b (0x725c bytes) 000195 1786565822.073322 mapread 0x02b081 thru 0x038a8b (0xda0b bytes) 000197 1786565822.078427 mapwrite 0x01e494 thru 0x028dcd (0xa93a bytes) 000198 1786565822.143699 trunc from 0x03d1b5 to 0x03c029 (0x118b bytes) 000199 1786565822.168469 fallocate 0x0104d3 thru 0x01099f (0x4cd bytes) 000200 1786565822.177036 trunc from 0x01744a to 0x02097e (0x9535 bytes) 000202 1786565822.186412 trunc from 0x02812a to 0x02e754 (0x662b bytes) 000203 1786565822.192957 read 0x00486e thru 0x006c0a (0x239d bytes) 000204 1786565822.206982 fallocate 0x0364f4 thru 0x03ffff (0x9b0c bytes) 000205 1786565822.214650 trunc from 0x03c02a to 0x00721a (0x34e0f bytes) 000206 1786565822.237037 trunc from 0x00721b to 0x0252ee (0x1e0d4 bytes) 000207 1786565822.261467 punch from 0x01f36c to 0x01f826 (0x4bb bytes) 000208 1786565822.267499 punch from 0x010a51 to 0x013ec8 (0x3478 bytes) 000209 1786565822.273511 punch from 0x00c520 to 0x00f37d (0x2e5e bytes) 000211 1786565822.281197 punch from 0x002f90 to 0x00ce57 (0x9ec8 bytes) 000212 1786565822.292043 mapwrite 0x0045ac thru 0x010492 (0xbee7 bytes) 000213 1786565822.404542 MIRROR_RESYNC 3 mirrors 000214 1786565823.306234 mapwrite 0x010b32 thru 0x01df8a (0xd459 bytes) 000215 1786565823.487984 write 0x0183f2 thru 0x01b594 (0x31a3 bytes) 000216 1786565823.497672 write 0x02164e thru 0x022b99 (0x154c bytes) 000218 1786565823.509598 read 0x013513 thru 0x01a8b4 (0x73a2 bytes) 000219 1786565823.520145 mapread 0x00fac0 thru 0x01722e (0x776f bytes) 000220 1786565823.531413 punch from 0x00d524 to 0x00d867 (0x344 bytes) 000221 1786565823.554960 MIRROR_RESYNC 3 mirrors 000222 1786565824.681371 trunc from 0x00d1eb to 0x00e608 (0x141e bytes) 000224 1786565824.756879 fallocate 0x0390ac thru 0x03e09a (0x4fef bytes) 000225 1786565824.778993 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e09b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e09b): Mirror 5: 0xdaf36977 (stale) Mirror 6: 0xdaf36977 (stale) CRC-32 checksum value for chunk [0, 0x3e09b): Mirror 1: 0x47362fb2 Mirror 7: 0x47362fb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e09b: skipped 000226 1786565825.917692 MIRROR_RESYNC 4 mirrors 000227 1786565826.738824 read 0x0208c8 thru 0x02457c (0x3cb5 bytes) 000228 1786565826.751471 mapwrite 0x001bd6 thru 0x0116aa (0xfad5 bytes) 000229 1786565826.883787 MIRROR_RESYNC 4 mirrors 000230 1786565827.992643 fallocate 0x01736f thru 0x01b090 (0x3d22 bytes) 000231 1786565828.024554 punch from 0x031f77 to 0x03e09a (0xc124 bytes) 000232 1786565828.030486 punch from 0x0195e1 to 0x01dbf6 (0x4616 bytes) 000233 1786565828.036985 punch from 0x006743 to 0x00eafd (0x83bb bytes) 000234 1786565828.043393 fallocate 0x017e87 thru 0x0219fd (0x9b77 bytes) 000235 1786565828.049827 read 0x024f57 thru 0x025229 (0x2d3 bytes) 000236 1786565828.057419 punch from 0x038fd0 to 0x03e09a (0x50cb bytes) 000237 1786565828.062987 mapwrite 0x00dc19 thru 0x01a17b (0xc563 bytes) 000238 1786565828.123700 mapwrite 0x00b945 thru 0x017146 (0xb802 bytes) 000240 1786565828.155652 trunc from 0x03e09b to 0x00c88b (0x3180f bytes) 000241 1786565828.186748 read 0x00c5b5 thru 0x00c88b (0x2d7 bytes) 000242 1786565828.187537 trunc from 0x00bda2 to 0x00bdc6 (0x25 bytes) 000243 1786565828.194748 write 0x0290c7 thru 0x02a0de (0x1018 bytes) 000244 1786565828.197151 fallocate 0x00f3db thru 0x01bbed (0xc813 bytes) 000246 1786565828.208182 mapread 0x0029bb thru 0x002d5a (0x3a0 bytes) 000247 1786565828.226377 write 0x0142dd thru 0x01ba54 (0x7778 bytes) 000248 1786565828.228908 read 0x02345b thru 0x02a0de (0x6c84 bytes) 000249 1786565828.230505 read 0x026ec3 thru 0x02a0de (0x321c bytes) 000251 1786565828.232743 mapwrite 0x0156e1 thru 0x022cc8 (0xd5e8 bytes) 000252 1786565828.268056 trunc from 0x02a0df to 0x03b742 (0x11664 bytes) 000253 1786565828.289673 write 0x025f54 thru 0x033cec (0xdd99 bytes) 000254 1786565828.298266 write 0x0349a8 thru 0x035f0e (0x1567 bytes) 000255 1786565828.315517 trunc from 0x03b743 to 0x00741c (0x34326 bytes) 000256 1786565828.344875 write 0x030d78 thru 0x03c7f1 (0xba7a bytes) 000257 1786565828.355807 read 0x002d05 thru 0x007c8d (0x4f89 bytes) 000258 1786565828.361980 punch from 0x03933e to 0x03c7f1 (0x34b4 bytes) 000259 1786565828.407800 mapread 0x024fec thru 0x03159d (0xc5b2 bytes) 000260 1786565828.449640 mapread 0x015e6a thru 0x01b4f8 (0x568f bytes) 000261 1786565828.456973 write 0x004d4e thru 0x00a139 (0x53ec bytes) 000262 1786565828.471703 read 0x0148c9 thru 0x015023 (0x75b bytes) 000264 1786565828.474761 mapwrite 0x01ea4c thru 0x029ad1 (0xb086 bytes) 000265 1786565828.554824 trunc from 0x03c7f2 to 0x034c47 (0x7baa bytes) 000266 1786565828.576555 read 0x01a7a2 thru 0x023606 (0x8e65 bytes) 000267 1786565828.577358 mapread 0x001605 thru 0x0048b4 (0x32b0 bytes) 000268 1786565828.578438 trunc from 0x034c48 to 0x014502 (0x20745 bytes) 000269 1786565828.601292 mapwrite 0x003672 thru 0x00d5c0 (0x9f4f bytes) 000270 1786565828.626405 mapwrite 0x03bde1 thru 0x03ffff (0x421f bytes) 000272 1786565828.676518 trunc from 0x008db2 to 0x009ce7 (0xf36 bytes) 000273 1786565828.683693 mapwrite 0x02cb79 thru 0x02dc59 (0x10e1 bytes) 000274 1786565828.695521 punch from 0x03d650 to 0x03ffff (0x29b0 bytes) 000275 1786565828.702523 mapread 0x022450 thru 0x02b0a2 (0x8c53 bytes) 000276 1786565828.704067 read 0x02879e thru 0x02a6d2 (0x1f35 bytes) 000277 1786565828.707407 trunc from 0x040000 to 0x03adff (0x5200 bytes) 000278 1786565828.728780 read 0x01d7b7 thru 0x01fc1d (0x2467 bytes) 000279 1786565828.731870 trunc from 0x0398de to 0x03adff (0x1522 bytes) 000280 1786565828.736947 mapwrite 0x00cfd8 thru 0x01c7d6 (0xf7ff bytes) 000281 1786565828.759340 fallocate 0x039c89 thru 0x03ffff (0x6377 bytes) 000282 1786565828.764043 trunc from 0x037027 to 0x03adff (0x3dd9 bytes) 000283 1786565828.773123 MIRROR_RESYNC 4 mirrors 000285 1786565829.732481 trunc from 0x0222e9 to 0x02d507 (0xb21f bytes) 000286 1786565829.792742 fallocate 0x0251fb thru 0x034da3 (0xfba9 bytes) 000287 1786565829.803309 trunc from 0x03ae00 to 0x018da2 (0x2205d bytes) 000289 1786565829.824991 fallocate 0x03ed82 thru 0x03ffff (0x127e bytes) 000290 1786565829.831581 fallocate 0x014bd3 thru 0x01ce55 (0x8283 bytes) 000291 1786565829.838305 mapwrite 0x02e747 thru 0x035a2d (0x72e7 bytes) 000292 1786565829.877556 trunc from 0x040000 to 0x005ae7 (0x3a518 bytes) 000293 1786565829.896618 mapread 0x003bb4 thru 0x005ae7 (0x1f34 bytes) 000294 1786565829.906504 punch from 0x00054e to 0x005ae7 (0x559a bytes) 000295 1786565829.911625 mapread 0x000c08 thru 0x005ae7 (0x4ee0 bytes) 000296 1786565829.923791 fallocate 0x03b36b thru 0x03ffff (0x4c95 bytes) 000298 1786565829.930563 trunc from 0x040000 to 0x00191c (0x3e6e3 bytes) 000299 1786565829.954084 read 0x00104e thru 0x00191c (0x8cf bytes) 000300 1786565829.954889 punch from 0x001054 to 0x00191c (0x8c9 bytes) 000301 1786565829.962555 trunc from 0x00191d to 0x016fb8 (0x1569c bytes) 000302 1786565829.975306 read 0x0126e4 thru 0x016fb8 (0x48d5 bytes) 000303 1786565829.985985 trunc from 0x016fb9 to 0x02908f (0x120d7 bytes) 000304 1786565830.006058 fallocate 0x028342 thru 0x02aecd (0x2b8c bytes) 000305 1786565830.014397 punch from 0x01d343 to 0x02aecd (0xdb8b bytes) 000306 1786565830.019363 mapwrite 0x00c0d3 thru 0x011554 (0x5482 bytes) 000307 1786565830.037549 trunc from 0x02aece to 0x02a8a9 (0x624 bytes) 000308 1786565830.051932 mapwrite 0x0028b2 thru 0x007351 (0x4aa0 bytes) 000309 1786565830.065370 trunc from 0x002826 to 0x008693 (0x5e6e bytes) 000310 1786565830.070282 mapwrite 0x01712d thru 0x021987 (0xa85b bytes) 000311 1786565830.095007 trunc from 0x00b84b to 0x01204d (0x6803 bytes) 000312 1786565830.100731 write 0x034d0d thru 0x03fe97 (0xb18b bytes) 000313 1786565830.104627 trunc from 0x03fe98 to 0x01c1b9 (0x23cde bytes) 000314 1786565830.124902 write 0x02eaa6 thru 0x03bb0d (0xd068 bytes) 000315 1786565830.129149 trunc from 0x00bc0e to 0x014977 (0x8d6a bytes) 000316 1786565830.141684 fallocate 0x01a01b thru 0x01b527 (0x150d bytes) 000317 1786565830.157432 MIRROR_RESYNC 4 mirrors 000318 1786565831.424767 trunc from 0x01039d to 0x010ac2 (0x726 bytes) 000319 1786565831.467896 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb0e): Mirror 5: 0xe5b89f3b (stale) Mirror 7: 0xe5b89f3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb0e): Mirror 5: 0xe5b89f3b (stale) Mirror 7: 0xe5b89f3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb0e: skipped 000320 1786565831.737831 mapread 0x01c6e0 thru 0x01d2d8 (0xbf9 bytes) 000321 1786565831.752020 mapwrite 0x0315f0 thru 0x0350d4 (0x3ae5 bytes) 000322 1786565831.787672 fallocate 0x030b37 thru 0x031a39 (0xf03 bytes) 000323 1786565831.793148 read 0x03a437 thru 0x03bb0d (0x16d7 bytes) 000324 1786565831.798310 trunc from 0x03bb0e to 0x016743 (0x253ca bytes) 000326 1786565831.821599 read 0x0152b7 thru 0x016743 (0x148d bytes) 000327 1786565831.831866 trunc from 0x004734 to 0x009fde (0x58ab bytes) 000328 1786565831.837255 punch from 0x003d71 to 0x00a135 (0x63c5 bytes) 000330 1786565831.843725 read 0x009976 thru 0x00f8bc (0x5f47 bytes) 000331 1786565831.853493 trunc from 0x016744 to 0x0205fb (0x9eb8 bytes) 000333 1786565831.881198 write 0x009112 thru 0x01895a (0xf849 bytes) 000334 1786565831.895734 read 0x005f7d thru 0x0105ae (0xa632 bytes) 000335 1786565831.896743 read 0x00e147 thru 0x019255 (0xb10f bytes) 000336 1786565831.898282 trunc from 0x00b594 to 0x0133ea (0x7e57 bytes) 000337 1786565831.913128 read 0x009164 thru 0x012841 (0x96de bytes) 000338 1786565831.929473 MIRROR_RESYNC 3 mirrors 000340 1786565832.801498 mapread 0x0063f2 thru 0x00da38 (0x7647 bytes) 000341 1786565832.820599 trunc from 0x0205fc to 0x02cdf8 (0xc7fd bytes) 000342 1786565832.864669 mapread 0x026069 thru 0x02cdf8 (0x6d90 bytes) 000343 1786565832.878758 mapwrite 0x00acbc thru 0x00c58a (0x18cf bytes) 000345 1786565832.908068 read 0x021935 thru 0x023b7c (0x2248 bytes) 000346 1786565832.908837 punch from 0x025536 to 0x02cdf8 (0x78c3 bytes) 000347 1786565832.918605 MIRROR_RESYNC 3 mirrors 000348 1786565834.154500 mapread 0x00df1b thru 0x013cef (0x5dd5 bytes) 000349 1786565834.205463 write 0x031c0d thru 0x037653 (0x5a47 bytes) 000350 1786565834.260452 trunc from 0x037654 to 0x0328c7 (0x4d8c bytes) 000351 1786565834.286314 punch from 0x02c3d6 to 0x0328c7 (0x64f2 bytes) 000352 1786565834.298514 trunc from 0x004758 to 0x004c8e (0x537 bytes) 000353 1786565834.313559 MIRROR_RESYNC 3 mirrors 000354 1786565835.595579 write 0x01d9dc thru 0x0228b6 (0x4edb bytes) 000355 1786565835.623199 trunc from 0x0328c8 to 0x03fa3c (0xd175 bytes) 000356 1786565835.643409 trunc from 0x00d7dc to 0x0162dc (0x8b01 bytes) 000357 1786565835.649498 mapread 0x03ccf7 thru 0x03fa3c (0x2d46 bytes) 000358 1786565835.671533 mapwrite 0x01e9b0 thru 0x0231cc (0x481d bytes) 000359 1786565835.706533 trunc from 0x0171bf to 0x02483c (0xd67e bytes) 000360 1786565835.712392 punch from 0x002129 to 0x0024b1 (0x389 bytes) 000361 1786565835.717703 write 0x019340 thru 0x01a53a (0x11fb bytes) 000362 1786565835.736709 write 0x025627 thru 0x026275 (0xc4f bytes) 000363 1786565835.738235 trunc from 0x020931 to 0x026cc2 (0x6392 bytes) 000364 1786565835.770455 mapread 0x03963d thru 0x03fa3c (0x6400 bytes) 000365 1786565835.772377 mapwrite 0x014e9b thru 0x01dec1 (0x9027 bytes) 000366 1786565835.809212 mapread 0x025a3b thru 0x02916e (0x3734 bytes) 000367 1786565835.821740 mapwrite 0x0396ab thru 0x03d306 (0x3c5c bytes) 000368 1786565835.846604 mapread 0x006a6f thru 0x00b55c (0x4aee bytes) 000369 1786565835.861559 mapwrite 0x03f883 thru 0x03ffff (0x77d bytes) 000370 1786565835.893557 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x18b554ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x18b554ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000371 1786565836.158479 read 0x001137 thru 0x0097ff (0x86c9 bytes) 000372 1786565836.169221 trunc from 0x040000 to 0x033706 (0xc8f9 bytes) 000373 1786565836.192819 write 0x018c2b thru 0x019bec (0xfc2 bytes) 000374 1786565836.207420 trunc from 0x033707 to 0x03f2f8 (0xbbf2 bytes) 000375 1786565836.239602 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f2f9): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2f9): Mirror 7: 0xc0a9b784 (stale) CRC-32 checksum value for chunk [0, 0x3f2f9): Mirror 1: 0x9f0f1467 Mirror 8: 0x9f0f1467 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2f9: skipped 000376 1786565837.238931 trunc from 0x03f2f9 to 0x00d773 (0x31b85 bytes) 000377 1786565837.372689 mapwrite 0x024ee4 thru 0x0260e3 (0x1200 bytes) 000378 1786565837.498499 read 0x0040ec thru 0x01289d (0xe7b2 bytes) 000379 1786565837.512622 punch from 0x000d28 to 0x002d87 (0x2060 bytes) 000380 1786565837.519690 trunc from 0x0260e4 to 0x018224 (0xdebf bytes) 000381 1786565837.553399 mapwrite 0x00b18e thru 0x015a87 (0xa8fa bytes) 000382 1786565837.631644 MIRROR_RESYNC 3 mirrors 000383 1786565839.295186 trunc from 0x00f619 to 0x018224 (0x8c0c bytes) 000384 1786565839.388651 mapwrite 0x00214c thru 0x006c70 (0x4b25 bytes) 000385 1786565839.483643 write 0x0339e7 thru 0x0342ca (0x8e4 bytes) 000386 1786565839.499259 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: 0xd67d7e5c (stale) Mirror 8: 0xd67d7e5c (stale) CRC-32 checksum value for chunk [0, 0x342cb): Mirror 1: 0x4e4bd3f1 Mirror 9: 0x4e4bd3f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342cb: skipped 000387 1786565840.613008 mapread 0x02335b thru 0x03032a (0xcfd0 bytes) 000388 1786565840.743297 mapread 0x000b22 thru 0x008b21 (0x8000 bytes) 000389 1786565840.780194 read 0x02186f thru 0x0262ac (0x4a3e bytes) 000390 1786565840.795461 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: 0xd67d7e5c (stale) Mirror 8: 0xd67d7e5c (stale) CRC-32 checksum value for chunk [0, 0x342cb): Mirror 1: 0x4e4bd3f1 Mirror 9: 0x4e4bd3f1 Mirror 10: 0x4e4bd3f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342cb: skipped 000391 1786565842.167367 read 0x01125d thru 0x017a05 (0x67a9 bytes) 000393 1786565842.213381 trunc from 0x02d5c8 to 0x0342ca (0x6d03 bytes) 000394 1786565842.298627 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: 0xd67d7e5c (stale) Mirror 8: 0xd67d7e5c (stale) Mirror 9: 0x4e4bd3f1 (stale) Mirror 10: 0x4e4bd3f1 (stale) CRC-32 checksum value for chunk [0, 0x342cb): Mirror 1: 0x3cac3a2b Mirror 11: 0x3cac3a2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342cb: skipped 000395 1786565843.575116 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342cb): Mirror 7: 0xd67d7e5c (stale) Mirror 8: 0xd67d7e5c (stale) Mirror 9: 0x4e4bd3f1 (stale) Mirror 10: 0x4e4bd3f1 (stale) CRC-32 checksum value for chunk [0, 0x342cb): Mirror 1: 0x3cac3a2b Mirror 11: 0x3cac3a2b Mirror 12: 0x3cac3a2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342cb: skipped 000396 1786565844.790333 MIRROR_RESYNC 7 mirrors 000397 1786565846.200770 mapwrite 0x002407 thru 0x007969 (0x5563 bytes) 000398 1786565846.303299 write 0x02d068 thru 0x036618 (0x95b1 bytes) 000399 1786565846.312556 punch from 0x0076d8 to 0x00e8f9 (0x7222 bytes) 000400 1786565846.327061 trunc from 0x036619 to 0x024ad3 (0x11b45 bytes) 000401 1786565846.380731 mapread 0x013c69 thru 0x01d82b (0x9bc3 bytes) 000402 1786565846.403184 write 0x00284e thru 0x00fbc0 (0xd373 bytes) 000403 1786565846.404981 write 0x02ba2d thru 0x02c99f (0xf73 bytes) 000404 1786565846.406216 read 0x00bcf5 thru 0x0161d2 (0xa4de bytes) 000406 1786565846.407171 punch from 0x01d31c to 0x02be08 (0xeaed bytes) 000407 1786565846.426711 mapread 0x029cd0 thru 0x02c99f (0x2cd0 bytes) 000408 1786565846.437991 mapread 0x010c51 thru 0x01761a (0x69ca bytes) 000409 1786565846.440557 trunc from 0x025ba3 to 0x02c4c9 (0x6927 bytes) 000410 1786565846.450316 punch from 0x0267f8 to 0x0294dd (0x2ce6 bytes) 000411 1786565846.464271 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c9a0): Mirror 7: 0x5d7a532b (stale) Mirror 9: 0x5d7a532b (stale) Mirror 10: 0x5d7a532b (stale) Mirror 11: 0x5d7a532b (stale) Mirror 12: 0x5d7a532b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c9a0): Mirror 7: 0x5d7a532b (stale) Mirror 9: 0x5d7a532b (stale) Mirror 10: 0x5d7a532b (stale) Mirror 11: 0x5d7a532b (stale) Mirror 12: 0x5d7a532b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c9a0: skipped 000412 1786565847.091823 punch from 0x00b923 to 0x00f999 (0x4077 bytes) 000413 1786565847.097577 mapread 0x005dd0 thru 0x013ad8 (0xdd09 bytes) 000414 1786565847.130778 trunc from 0x019c21 to 0x019eb1 (0x291 bytes) 000415 1786565847.136470 mapread 0x024b78 thru 0x02c99f (0x7e28 bytes) 000416 1786565847.148453 mapread 0x00fc05 thru 0x01c943 (0xcd3f bytes) 000417 1786565847.156157 read 0x01af1c thru 0x027b57 (0xcc3c bytes) 000418 1786565847.161391 MIRROR_RESYNC 6 mirrors 000419 1786565849.434318 punch from 0x009790 to 0x00b487 (0x1cf8 bytes) 000420 1786565849.482774 mapwrite 0x02bc55 thru 0x03519b (0x9547 bytes) 000421 1786565849.593719 mapread 0x02d6bd thru 0x02e972 (0x12b6 bytes) 000422 1786565849.594940 trunc from 0x03519c to 0x008097 (0x2d104 bytes) 000423 1786565849.625178 read 0x00333d thru 0x008097 (0x4d5b bytes) 000424 1786565849.649298 fallocate 0x019691 thru 0x022064 (0x89d4 bytes) 000425 1786565849.656694 mapread 0x00cf46 thru 0x011575 (0x4630 bytes) 000426 1786565849.671493 fallocate 0x032683 thru 0x03b583 (0x8f01 bytes) 000427 1786565849.679331 trunc from 0x01e46b to 0x021e9d (0x3a33 bytes) 000429 1786565849.688480 read 0x004209 thru 0x00ea11 (0xa809 bytes) 000430 1786565849.694973 MIRROR_RESYNC 6 mirrors 000431 1786565851.336349 read 0x00d2e3 thru 0x017151 (0x9e6f bytes) 000432 1786565851.350983 fallocate 0x0201b9 thru 0x026c24 (0x6a6c bytes) 000433 1786565851.419947 trunc from 0x006b0e to 0x00f782 (0x8c75 bytes) 000434 1786565851.429398 read 0x013421 thru 0x016e7d (0x3a5d bytes) 000435 1786565851.447658 mapwrite 0x006097 thru 0x00fc73 (0x9bdd bytes) 000436 1786565851.496252 mapread 0x00057b thru 0x004a7f (0x4505 bytes) 000438 1786565851.506220 trunc from 0x0095a1 to 0x00eb50 (0x55b0 bytes) 000439 1786565851.520299 mapread 0x00c97c thru 0x00f751 (0x2dd6 bytes) 000441 1786565851.554226 read 0x006782 thru 0x00d6d0 (0x6f4f bytes) 000442 1786565851.566519 read 0x023388 thru 0x0253e1 (0x205a bytes) 000443 1786565851.568512 mapwrite 0x02bef0 thru 0x02e31d (0x242e bytes) 000445 1786565851.637035 fallocate 0x01e68e thru 0x020d7e (0x26f1 bytes) 000446 1786565851.647455 write 0x003350 thru 0x00ae7a (0x7b2b bytes) 000447 1786565851.655464 write 0x000475 thru 0x00b00b (0xab97 bytes) 000448 1786565851.657194 fallocate 0x01e367 thru 0x0214cd (0x3167 bytes) 000449 1786565851.668442 mapwrite 0x026275 thru 0x0296c7 (0x3453 bytes) 000450 1786565851.695227 read 0x00aa3c thru 0x00fd8f (0x5354 bytes) 000451 1786565851.696100 write 0x003b52 thru 0x007073 (0x3522 bytes) 000452 1786565851.701555 trunc from 0x01c561 to 0x020463 (0x3f03 bytes) 000454 1786565851.710554 write 0x034d75 thru 0x03ffff (0xb28b bytes) 000455 1786565851.727942 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb3af6172 (stale) Mirror 10: 0xb3af6172 (stale) Mirror 11: 0xb3af6172 (stale) Mirror 12: 0xb3af6172 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb3af6172 (stale) Mirror 10: 0xb3af6172 (stale) Mirror 11: 0xb3af6172 (stale) Mirror 12: 0xb3af6172 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1786565852.217994 mapread 0x0271bd thru 0x0339b9 (0xc7fd bytes) 000457 1786565852.250756 read 0x007f83 thru 0x00cae8 (0x4b66 bytes) 000458 1786565852.264446 punch from 0x004cdc to 0x00aa05 (0x5d2a bytes) 000459 1786565852.270608 fallocate 0x037ca5 thru 0x03eee1 (0x723d bytes) 000460 1786565852.276187 mapwrite 0x004fe8 thru 0x011dce (0xcde7 bytes) 000461 1786565852.333038 punch from 0x00cf2f to 0x01b1ff (0xe2d1 bytes) 000462 1786565852.338972 mapwrite 0x03e8d4 thru 0x03ffff (0x172c bytes) 000463 1786565852.373070 fallocate 0x0095c2 thru 0x00df6b (0x49aa bytes) 000464 1786565852.382735 trunc from 0x03c6d6 to 0x03ffff (0x392a bytes) 000466 1786565852.391089 mapwrite 0x03ba4d thru 0x03ca86 (0x103a bytes) 000467 1786565852.436036 mapwrite 0x02ab01 thru 0x02ce59 (0x2359 bytes) 000468 1786565852.453370 trunc from 0x02bdbd to 0x037e52 (0xc096 bytes) 000469 1786565852.464240 read 0x010039 thru 0x01ed40 (0xed08 bytes) 000470 1786565852.515149 write 0x03ac53 thru 0x03ffff (0x53ad bytes) 000471 1786565852.522618 mapwrite 0x029d19 thru 0x02ba1c (0x1d04 bytes) 000472 1786565852.572926 punch from 0x02aa9b to 0x03827e (0xd7e4 bytes) 000473 1786565852.582573 read 0x02e6dd thru 0x034fc7 (0x68eb bytes) 000474 1786565852.612947 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb3af6172 (stale) Mirror 10: 0xb3af6172 (stale) Mirror 11: 0xb3af6172 (stale) Mirror 12: 0xb3af6172 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5550d95 Mirror 13: 0x5550d95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1786565853.816617 trunc from 0x001a69 to 0x004abd (0x3055 bytes) 000476 1786565853.868309 read 0x018961 thru 0x02132a (0x89ca bytes) 000477 1786565853.881864 fallocate 0x0001c9 thru 0x004afc (0x4934 bytes) 000478 1786565853.891720 mapwrite 0x038778 thru 0x03ffff (0x7888 bytes) 000479 1786565853.952909 fallocate 0x007479 thru 0x017430 (0xffb8 bytes) 000480 1786565853.968339 write 0x02167d thru 0x024a36 (0x33ba bytes) 000481 1786565853.991439 fallocate 0x01a43d thru 0x01d7a5 (0x3369 bytes) 000482 1786565853.999184 punch from 0x0206eb to 0x0238d5 (0x31eb bytes) 000483 1786565854.018595 mapwrite 0x00135e thru 0x00dcde (0xc981 bytes) 000484 1786565854.067630 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb3af6172 (stale) Mirror 10: 0xb3af6172 (stale) Mirror 11: 0xb3af6172 (stale) Mirror 12: 0xb3af6172 (stale) Mirror 13: 0x5550d95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfe5228e Mirror 14: 0xcfe5228e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000486 1786565855.111533 mapwrite 0x01927d thru 0x0270b0 (0xde34 bytes) 000488 1786565855.247098 punch from 0x008a44 to 0x011a12 (0x8fcf bytes) 000489 1786565855.255116 fallocate 0x02d9d2 thru 0x032951 (0x4f80 bytes) 000490 1786565855.271976 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb3af6172 (stale) Mirror 10: 0xb3af6172 (stale) Mirror 11: 0xb3af6172 (stale) Mirror 12: 0xb3af6172 (stale) Mirror 13: 0x5550d95 (stale) Mirror 14: 0xcfe5228e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6075d0f2 Mirror 15: 0x6075d0f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000491 1786565856.398977 punch from 0x016eec to 0x0255be (0xe6d3 bytes) 000492 1786565856.458046 MIRROR_RESYNC 8 mirrors 000493 1786565858.088475 read 0x016249 thru 0x01bf5e (0x5d16 bytes) 000494 1786565858.098308 punch from 0x01d4d8 to 0x02a33f (0xce68 bytes) 000495 1786565858.132681 read 0x03410c thru 0x03e98a (0xa87f bytes) 000496 1786565858.175798 MIRROR_RESYNC 8 mirrors 000497 1786565859.648356 mapread 0x003c99 thru 0x006be6 (0x2f4e bytes) 000498 1786565859.671397 MIRROR_RESYNC 8 mirrors 000499 1786565860.349040 trunc from 0x018530 to 0x020668 (0x8139 bytes) 000500 1786565860.396315 mapwrite 0x0208da thru 0x02a4d9 (0x9c00 bytes) 000501 1786565860.466497 mapread 0x02042a thru 0x02dcdc (0xd8b3 bytes) 000502 1786565860.511081 trunc from 0x00e9ce to 0x015ffa (0x762d bytes) 000503 1786565860.534577 mapread 0x02e6b4 thru 0x030b9b (0x24e8 bytes) 000504 1786565860.542579 fallocate 0x0395f5 thru 0x03ee19 (0x5825 bytes) 000505 1786565860.559180 mapread 0x0201ee thru 0x023ba4 (0x39b7 bytes) 000506 1786565860.560314 trunc from 0x040000 to 0x006c96 (0x39369 bytes) 000507 1786565860.655024 read 0x00293b thru 0x006c96 (0x435c bytes) 000508 1786565860.685716 write 0x027728 thru 0x035568 (0xde41 bytes) 000509 1786565860.703872 trunc from 0x035569 to 0x01c0a2 (0x194c6 bytes) 000510 1786565860.743315 write 0x014f4d thru 0x01eba6 (0x9c5a bytes) 000511 1786565860.757008 mapread 0x01c4af thru 0x01eba6 (0x26f8 bytes) 000512 1786565860.758949 write 0x012f5a thru 0x01512c (0x21d3 bytes) 000514 1786565860.776162 trunc from 0x007802 to 0x00ce5e (0x565d bytes) 000515 1786565860.795400 trunc from 0x0179cb to 0x01951d (0x1b53 bytes) 000516 1786565860.841887 trunc from 0x00cffa to 0x0138a4 (0x68ab bytes) 000517 1786565860.862778 mapread 0x005a25 thru 0x010360 (0xa93c bytes) 000518 1786565860.911456 punch from 0x00bdf3 to 0x017371 (0xb57f bytes) 000519 1786565860.935729 punch from 0x00f4a9 to 0x016988 (0x74e0 bytes) 000520 1786565860.949495 fallocate 0x02a7e9 thru 0x03269f (0x7eb7 bytes) 000521 1786565860.960473 write 0x005e75 thru 0x006b4d (0xcd9 bytes) 000522 1786565860.969265 punch from 0x02c67e to 0x03269f (0x6022 bytes) 000523 1786565860.979485 read 0x02521e thru 0x0289ae (0x3791 bytes) 000524 1786565861.001779 mapread 0x013741 thru 0x015b6b (0x242b bytes) 000526 1786565861.012036 mapread 0x006503 thru 0x008d63 (0x2861 bytes) 000527 1786565861.031069 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326a0): Mirror 7: 0x350d5c69 (stale) Mirror 10: 0x350d5c69 (stale) Mirror 11: 0x350d5c69 (stale) Mirror 12: 0x350d5c69 (stale) Mirror 13: 0x350d5c69 (stale) Mirror 15: 0x350d5c69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326a0): Mirror 7: 0x350d5c69 (stale) Mirror 10: 0x350d5c69 (stale) Mirror 11: 0x350d5c69 (stale) Mirror 12: 0x350d5c69 (stale) Mirror 13: 0x350d5c69 (stale) Mirror 15: 0x350d5c69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326a0: skipped 000528 1786565861.840132 mapwrite 0x03a5a8 thru 0x03fe11 (0x586a bytes) 000529 1786565861.926771 trunc from 0x0399fe to 0x03cae4 (0x30e7 bytes) 000530 1786565861.940096 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fe12): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe12): Mirror 7: 0xcebcf9e9 (stale) Mirror 10: 0xcebcf9e9 (stale) Mirror 11: 0xcebcf9e9 (stale) Mirror 12: 0xcebcf9e9 (stale) Mirror 13: 0xcebcf9e9 (stale) Mirror 15: 0xcebcf9e9 (stale) CRC-32 checksum value for chunk [0, 0x3fe12): Mirror 1: 0xa441d209 Mirror 16: 0xa441d209 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe12: skipped 000531 1786565863.193238 mapwrite 0x02bc04 thru 0x03492e (0x8d2b bytes) 000532 1786565863.309589 mapread 0x027aed thru 0x036602 (0xeb16 bytes) 000533 1786565863.327537 read 0x016339 thru 0x01bec8 (0x5b90 bytes) 000534 1786565863.344392 read 0x019861 thru 0x023e5a (0xa5fa bytes) 000535 1786565863.362626 read 0x03fdc3 thru 0x03fe11 (0x4f bytes) 000536 1786565863.363384 fallocate 0x03ea93 thru 0x03ffff (0x156d bytes) 000537 1786565863.369319 punch from 0x02b6fb to 0x034421 (0x8d27 bytes) 000538 1786565863.376330 punch from 0x01ad49 to 0x01e7db (0x3a93 bytes) 000539 1786565863.381600 write 0x018eea thru 0x0283ae (0xf4c5 bytes) 000540 1786565863.389417 mapwrite 0x0353f3 thru 0x03c383 (0x6f91 bytes) 000541 1786565863.411552 trunc from 0x040000 to 0x0330fd (0xcf02 bytes) 000542 1786565863.433785 write 0x01ce51 thru 0x01f1b4 (0x2364 bytes) 000545 1786565863.439648 write 0x031630 thru 0x038979 (0x734a bytes) 000546 1786565863.448961 fallocate 0x01a1f8 thru 0x0203a9 (0x61b2 bytes) 000547 1786565863.454261 trunc from 0x03897a to 0x02594b (0x1302e bytes) 000548 1786565863.473620 trunc from 0x01e13b to 0x01e7de (0x6a4 bytes) 000549 1786565863.484646 write 0x01241c thru 0x015a02 (0x35e7 bytes) 000550 1786565863.486191 mapread 0x018537 thru 0x018fce (0xa98 bytes) 000551 1786565863.489780 trunc from 0x02594c to 0x009c9f (0x1bcac bytes) 000552 1786565863.511029 write 0x030a62 thru 0x03d8f0 (0xce8f bytes) 000553 1786565863.514131 trunc from 0x03c1ec to 0x03d8f0 (0x1705 bytes) 000554 1786565863.527535 write 0x01d09a thru 0x01f94e (0x28b5 bytes) 000557 1786565863.541224 mapread 0x030b47 thru 0x036323 (0x57dd bytes) 000558 1786565863.543200 fallocate 0x00d765 thru 0x01c609 (0xeea5 bytes) 000559 1786565863.548834 mapwrite 0x03321f thru 0x0362f8 (0x30da bytes) 000561 1786565863.566904 trunc from 0x028135 to 0x02c7cc (0x4698 bytes) 000562 1786565863.572260 trunc from 0x02cce4 to 0x02f54f (0x286c bytes) 000563 1786565863.578224 punch from 0x01f701 to 0x023fb4 (0x48b4 bytes) 000564 1786565863.583770 write 0x032b0c thru 0x03e0eb (0xb5e0 bytes) 000565 1786565863.591487 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e0ec): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0ec): Mirror 7: 0x3f1f1356 (stale) Mirror 10: 0x3f1f1356 (stale) Mirror 11: 0x3f1f1356 (stale) Mirror 12: 0x3f1f1356 (stale) Mirror 13: 0x3f1f1356 (stale) Mirror 15: 0x3f1f1356 (stale) Mirror 16: 0xae2173cc (stale) CRC-32 checksum value for chunk [0, 0x3e0ec): Mirror 1: 0x1e1f291c Mirror 17: 0x1e1f291c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0ec: skipped 000566 1786565864.452494 fallocate 0x034c01 thru 0x035992 (0xd92 bytes) 000567 1786565864.492883 fallocate 0x03452e thru 0x03ffff (0xbad2 bytes) 000568 1786565864.501063 fallocate 0x034673 thru 0x03bee5 (0x7873 bytes) 000569 1786565864.506047 trunc from 0x040000 to 0x022882 (0x1d77d bytes) 000570 1786565864.553467 read 0x00b811 thru 0x00c681 (0xe71 bytes) 000571 1786565864.564628 punch from 0x0012e1 to 0x00c9d7 (0xb6f7 bytes) 000572 1786565864.574616 MIRROR_RESYNC 9 mirrors 000573 1786565866.312296 trunc from 0x022883 to 0x023af5 (0x1273 bytes) 000574 1786565866.362802 fallocate 0x029e5f thru 0x032dcf (0x8f71 bytes) 000575 1786565866.368831 fallocate 0x02cdde thru 0x035029 (0x824c bytes) 000576 1786565866.375468 trunc from 0x03502a to 0x0110a5 (0x23f84 bytes) 000577 1786565866.395674 write 0x03677a thru 0x03ffff (0x9886 bytes) 000578 1786565866.397580 mapwrite 0x03093a thru 0x034634 (0x3cfb bytes) 000579 1786565866.440835 trunc from 0x040000 to 0x03f6e9 (0x916 bytes) 000580 1786565866.462423 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f6ea): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6ea): Mirror 7: 0x756d7e76 (stale) Mirror 10: 0x756d7e76 (stale) Mirror 11: 0x756d7e76 (stale) Mirror 12: 0x756d7e76 (stale) Mirror 13: 0x756d7e76 (stale) Mirror 15: 0x756d7e76 (stale) Mirror 16: 0x756d7e76 (stale) Mirror 17: 0x756d7e76 (stale) CRC-32 checksum value for chunk [0, 0x3f6ea): Mirror 1: 0x1b9887c5 Mirror 18: 0x1b9887c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6ea: skipped 000581 1786565867.747123 punch from 0x023451 to 0x02d8b9 (0xa469 bytes) 000582 1786565867.793762 punch from 0x001b37 to 0x008e37 (0x7301 bytes) 000583 1786565867.800117 punch from 0x01a891 to 0x02816c (0xd8dc bytes) 000584 1786565867.805517 read 0x004ccc thru 0x01007a (0xb3af bytes) 000585 1786565867.815785 punch from 0x0036a5 to 0x00d942 (0xa29e bytes) 000586 1786565867.822164 punch from 0x0080db to 0x00a24f (0x2175 bytes) 000587 1786565867.826533 punch from 0x021138 to 0x0292ff (0x81c8 bytes) 000588 1786565867.831442 punch from 0x02d636 to 0x0387b7 (0xb182 bytes) 000589 1786565867.837697 trunc from 0x00fe5c to 0x016e70 (0x7015 bytes) 000590 1786565867.842878 fallocate 0x00146c thru 0x0107a0 (0xf335 bytes) 000591 1786565867.848484 trunc from 0x03f6ea to 0x0371f7 (0x84f2 bytes) 000592 1786565867.880997 punch from 0x02f273 to 0x03101b (0x1da9 bytes) 000593 1786565867.886484 trunc from 0x00082c to 0x00e696 (0xde6b bytes) 000594 1786565867.896578 MIRROR_RESYNC 10 mirrors 000595 1786565869.245464 read 0x012521 thru 0x01aec9 (0x89a9 bytes) 000596 1786565869.257250 mapread 0x00a26f thru 0x0179cd (0xd75f bytes) 000597 1786565869.284724 mapread 0x01f864 thru 0x0285dd (0x8d7a bytes) 000598 1786565869.305237 write 0x02a603 thru 0x02eca5 (0x46a3 bytes) 000599 1786565869.355523 trunc from 0x0371f8 to 0x027bc9 (0xf62e bytes) 000600 1786565869.382409 mapread 0x00a81b thru 0x014dac (0xa592 bytes) 000601 1786565869.412158 mapwrite 0x0192ad thru 0x01ebb0 (0x5904 bytes) 000602 1786565869.445253 read 0x0123cb thru 0x0214cb (0xf101 bytes) 000603 1786565869.446499 read 0x004919 thru 0x011df0 (0xd4d8 bytes) 000604 1786565869.452455 MIRROR_RESYNC 10 mirrors 000605 1786565871.139486 mapread 0x027228 thru 0x027bc9 (0x9a2 bytes) 000606 1786565871.145417 trunc from 0x027bca to 0x03c647 (0x14a7e bytes) 000607 1786565871.186138 fallocate 0x03f74e thru 0x03ffff (0x8b2 bytes) 000608 1786565871.192273 fallocate 0x00f630 thru 0x01a7af (0xb180 bytes) 000609 1786565871.197734 write 0x02e2c8 thru 0x03123b (0x2f74 bytes) 000610 1786565871.207565 mapwrite 0x025e77 thru 0x033da5 (0xdf2f bytes) 000611 1786565871.256134 MIRROR_RESYNC 10 mirrors 000612 1786565873.586489 fallocate 0x02e0f9 thru 0x02f8ec (0x17f4 bytes) 000614 1786565873.640137 mapwrite 0x013751 thru 0x021aed (0xe39d bytes) 000615 1786565873.733570 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x97b62363 (stale) Mirror 10: 0x97b62363 (stale) Mirror 12: 0x97b62363 (stale) Mirror 13: 0x97b62363 (stale) Mirror 15: 0x97b62363 (stale) Mirror 16: 0x97b62363 (stale) Mirror 17: 0x97b62363 (stale) Mirror 18: 0x97b62363 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x97b62363 (stale) Mirror 10: 0x97b62363 (stale) Mirror 12: 0x97b62363 (stale) Mirror 13: 0x97b62363 (stale) Mirror 15: 0x97b62363 (stale) Mirror 16: 0x97b62363 (stale) Mirror 17: 0x97b62363 (stale) Mirror 18: 0x97b62363 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1786565874.691518 trunc from 0x040000 to 0x031ac8 (0xe537 bytes) 000617 1786565874.717267 write 0x03ebdf thru 0x03ffff (0x1421 bytes) 000618 1786565874.718823 mapread 0x01018f thru 0x0101bd (0x2f bytes) 000619 1786565874.726548 trunc from 0x040000 to 0x01aea7 (0x25158 bytes) 000621 1786565874.758637 punch from 0x00b84b to 0x019861 (0xe017 bytes) 000622 1786565874.767479 punch from 0x004ae6 to 0x0130f3 (0xe60e bytes) 000623 1786565874.786441 write 0x021251 thru 0x024b44 (0x38f4 bytes) 000624 1786565874.806509 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x24b45): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b45): Mirror 7: 0x7b70b543 (stale) Mirror 10: 0x7b70b543 (stale) Mirror 12: 0x7b70b543 (stale) Mirror 13: 0x7b70b543 (stale) Mirror 15: 0x7b70b543 (stale) Mirror 16: 0x7b70b543 (stale) Mirror 17: 0x7b70b543 (stale) Mirror 18: 0x7b70b543 (stale) CRC-32 checksum value for chunk [0, 0x24b45): Mirror 1: 0x58dd766a Mirror 19: 0x58dd766a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b45: skipped 000625 1786565876.416483 mapwrite 0x00413a thru 0x009803 (0x56ca bytes) 000626 1786565876.519150 write 0x01d8d6 thru 0x025f21 (0x864c bytes) 000627 1786565876.528570 mapread 0x00c24e thru 0x00dcfa (0x1aad bytes) 000628 1786565876.529735 mapread 0x0188e8 thru 0x01d149 (0x4862 bytes) 000629 1786565876.543706 mapwrite 0x02054f thru 0x0266e9 (0x619b bytes) 000630 1786565876.583641 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x266ea): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266ea): Mirror 7: 0xc36be136 (stale) Mirror 10: 0xc36be136 (stale) Mirror 12: 0xc36be136 (stale) Mirror 13: 0xc36be136 (stale) Mirror 15: 0xc36be136 (stale) Mirror 16: 0xc36be136 (stale) Mirror 17: 0xc36be136 (stale) Mirror 18: 0xc36be136 (stale) Mirror 19: 0x7962abd6 (stale) CRC-32 checksum value for chunk [0, 0x266ea): Mirror 1: 0xb4be615d Mirror 20: 0xb4be615d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266ea: skipped 000631 1786565877.524175 mapwrite 0x02a39f thru 0x02e1f8 (0x3e5a bytes) 000632 1786565877.607726 read 0x00cbb1 thru 0x017053 (0xa4a3 bytes) 000633 1786565877.621593 punch from 0x001761 to 0x010c6f (0xf50f bytes) 000634 1786565877.627327 trunc from 0x02e1f9 to 0x021355 (0xcea3 bytes) 000635 1786565877.651592 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x21356): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21356): Mirror 7: 0x256ae137 (stale) Mirror 10: 0x256ae137 (stale) Mirror 12: 0x256ae137 (stale) Mirror 13: 0x256ae137 (stale) Mirror 15: 0x256ae137 (stale) Mirror 16: 0x256ae137 (stale) Mirror 17: 0x256ae137 (stale) Mirror 18: 0x256ae137 (stale) Mirror 19: 0x63ac29f6 (stale) Mirror 20: 0x3f2543bb (stale) CRC-32 checksum value for chunk [0, 0x21356): Mirror 1: 0xb0e65505 Mirror 21: 0xb0e65505 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21356: skipped 000636 1786565878.575719 mapwrite 0x015fb8 thru 0x0161f1 (0x23a bytes) 000637 1786565878.717145 trunc from 0x01c31f to 0x021355 (0x5037 bytes) 000638 1786565878.723302 write 0x00347a thru 0x007c09 (0x4790 bytes) 000639 1786565878.743800 fallocate 0x01fff1 thru 0x026e18 (0x6e28 bytes) 000640 1786565878.750181 mapread 0x01f953 thru 0x026e18 (0x74c6 bytes) 000641 1786565878.768185 mapread 0x0180bf thru 0x01f2cc (0x720e bytes) 000642 1786565878.771357 punch from 0x000ec3 to 0x0092d0 (0x840e bytes) 000643 1786565878.794706 mapread 0x021ac7 thru 0x025ab7 (0x3ff1 bytes) 000644 1786565878.798707 trunc from 0x026e19 to 0x018e81 (0xdf97 bytes) 000645 1786565878.844363 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x18e82): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18e82): Mirror 7: 0x6e844cc6 (stale) Mirror 10: 0x6e844cc6 (stale) Mirror 12: 0x6e844cc6 (stale) Mirror 13: 0x6e844cc6 (stale) Mirror 15: 0x6e844cc6 (stale) Mirror 16: 0x6e844cc6 (stale) Mirror 17: 0x6e844cc6 (stale) Mirror 18: 0x6e844cc6 (stale) Mirror 19: 0x6e844cc6 (stale) Mirror 20: 0x16565fb5 (stale) Mirror 21: 0x6e844cc6 (stale) CRC-32 checksum value for chunk [0, 0x18e82): Mirror 1: 0xa8ff77e2 Mirror 22: 0xa8ff77e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18e82: skipped 000646 1786565879.864127 MIRROR_RESYNC 13 mirrors 000647 1786565881.860289 write 0x03fafe thru 0x03ffff (0x502 bytes) 000648 1786565881.907170 mapread 0x03ab0b thru 0x03ffff (0x54f5 bytes) 000651 1786565881.927088 mapread 0x015f58 thru 0x025a29 (0xfad2 bytes) 000653 1786565881.973358 read 0x0280da thru 0x02e489 (0x63b0 bytes) 000654 1786565881.989567 trunc from 0x040000 to 0x0076e2 (0x3891d bytes) 000655 1786565882.030565 trunc from 0x0076e3 to 0x022ce2 (0x1b600 bytes) 000656 1786565882.055591 read 0x00ff0a thru 0x017925 (0x7a1c bytes) 000657 1786565882.067804 trunc from 0x022ce3 to 0x01e591 (0x4751 bytes) 000658 1786565882.082324 write 0x0377aa thru 0x03d517 (0x5d6e bytes) 000659 1786565882.090180 MIRROR_RESYNC 13 mirrors 000660 1786565883.841952 MIRROR_EXTEND to 14 mirrors 000661 1786565884.944222 mapwrite 0x02cb4b thru 0x034dd3 (0x8289 bytes) 000662 1786565885.078707 mapread 0x0015aa thru 0x008316 (0x6d6d bytes) 000663 1786565885.089766 write 0x012e29 thru 0x01f1b7 (0xc38f bytes) 000664 1786565885.106567 fallocate 0x01b7b8 thru 0x022c7a (0x74c3 bytes) 000665 1786565885.112647 mapread 0x026feb thru 0x03238d (0xb3a3 bytes) 000666 1786565885.114575 fallocate 0x0317fb thru 0x03df2f (0xc735 bytes) 000667 1786565885.119939 trunc from 0x03d518 to 0x02be14 (0x11703 bytes) 000668 1786565885.143105 punch from 0x02467a to 0x029f21 (0x58a8 bytes) 000670 1786565885.148713 mapwrite 0x02b8b3 thru 0x02c415 (0xb63 bytes) 000671 1786565885.190114 trunc from 0x010e2f to 0x011506 (0x6d8 bytes) 000672 1786565885.195568 write 0x015e37 thru 0x0197e2 (0x39ac bytes) 000673 1786565885.197807 mapread 0x010db9 thru 0x01f09f (0xe2e7 bytes) 000674 1786565885.210129 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c416): Mirror 10: 0xf6c2c3ec (stale) Mirror 12: 0xf6c2c3ec (stale) Mirror 13: 0xf6c2c3ec (stale) Mirror 15: 0xf6c2c3ec (stale) Mirror 16: 0xf6c2c3ec (stale) Mirror 17: 0xf6c2c3ec (stale) Mirror 18: 0xf6c2c3ec (stale) Mirror 19: 0xf6c2c3ec (stale) Mirror 20: 0xf6c2c3ec (stale) Mirror 21: 0xf6c2c3ec (stale) Mirror 22: 0xf6c2c3ec (stale) Mirror 23: 0xf6c2c3ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c416): Mirror 10: 0xf6c2c3ec (stale) Mirror 12: 0xf6c2c3ec (stale) Mirror 13: 0xf6c2c3ec (stale) Mirror 15: 0xf6c2c3ec (stale) Mirror 16: 0xf6c2c3ec (stale) Mirror 17: 0xf6c2c3ec (stale) Mirror 18: 0xf6c2c3ec (stale) Mirror 19: 0xf6c2c3ec (stale) Mirror 20: 0xf6c2c3ec (stale) Mirror 21: 0xf6c2c3ec (stale) Mirror 22: 0xf6c2c3ec (stale) Mirror 23: 0xf6c2c3ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c416: skipped 000675 1786565886.033186 mapwrite 0x002b0b thru 0x009643 (0x6b39 bytes) 000677 1786565886.093826 trunc from 0x01bb1e to 0x01cf86 (0x1469 bytes) 000678 1786565886.104213 punch from 0x0255cb to 0x02c415 (0x6e4b bytes) 000679 1786565886.111996 fallocate 0x007fe1 thru 0x0127f7 (0xa817 bytes) 000680 1786565886.121033 punch from 0x01d92a to 0x0249db (0x70b2 bytes) 000681 1786565886.129222 write 0x0166da thru 0x0190e2 (0x2a09 bytes) 000682 1786565886.151556 fallocate 0x00d308 thru 0x013ad9 (0x67d2 bytes) 000683 1786565886.160615 punch from 0x001522 to 0x005a6e (0x454d bytes) 000684 1786565886.169335 trunc from 0x01fdee to 0x0297a7 (0x99ba bytes) 000685 1786565886.178389 trunc from 0x00bd25 to 0x017287 (0xb563 bytes) 000686 1786565886.196509 punch from 0x02b1a8 to 0x02c415 (0x126e bytes) 000687 1786565886.206581 fallocate 0x001fa0 thru 0x00f754 (0xd7b5 bytes) 000688 1786565886.214435 mapwrite 0x00aec6 thru 0x011cb6 (0x6df1 bytes) 000689 1786565886.252318 read 0x01528d thru 0x01949f (0x4213 bytes) 000690 1786565886.260197 read 0x014a1f thru 0x01f268 (0xa84a bytes) 000691 1786565886.272155 punch from 0x012674 to 0x013d6f (0x16fc bytes) 000692 1786565886.283191 MIRROR_RESYNC 13 mirrors 000693 1786565889.733440 read 0x004c3f thru 0x01236b (0xd72d bytes) 000694 1786565889.750411 fallocate 0x0208fe thru 0x0295d2 (0x8cd5 bytes) 000695 1786565889.846865 trunc from 0x02c416 to 0x007cf0 (0x24725 bytes) 000696 1786565889.876564 write 0x023727 thru 0x0254fb (0x1dd5 bytes) 000697 1786565889.880517 fallocate 0x032062 thru 0x0394fd (0x749c bytes) 000698 1786565889.905242 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: 0x4f263eb9 (stale) Mirror 12: 0x4f263eb9 (stale) Mirror 13: 0x4f263eb9 (stale) Mirror 15: 0x4f263eb9 (stale) Mirror 16: 0x4f263eb9 (stale) Mirror 17: 0x4f263eb9 (stale) Mirror 18: 0x4f263eb9 (stale) Mirror 19: 0x4f263eb9 (stale) Mirror 20: 0x4f263eb9 (stale) Mirror 21: 0x4f263eb9 (stale) Mirror 22: 0x4f263eb9 (stale) Mirror 23: 0x4f263eb9 (stale) CRC-32 checksum value for chunk [0, 0x394fe): Mirror 1: 0xcc88d908 Mirror 24: 0xcc88d908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394fe: skipped 000699 1786565891.105057 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: 0x4f263eb9 (stale) Mirror 12: 0x4f263eb9 (stale) Mirror 13: 0x4f263eb9 (stale) Mirror 15: 0x4f263eb9 (stale) Mirror 16: 0x4f263eb9 (stale) Mirror 17: 0x4f263eb9 (stale) Mirror 18: 0x4f263eb9 (stale) Mirror 19: 0x4f263eb9 (stale) Mirror 20: 0x4f263eb9 (stale) Mirror 21: 0x4f263eb9 (stale) Mirror 22: 0x4f263eb9 (stale) Mirror 23: 0x4f263eb9 (stale) CRC-32 checksum value for chunk [0, 0x394fe): Mirror 1: 0xcc88d908 Mirror 24: 0xcc88d908 Mirror 25: 0xcc88d908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394fe: skipped 000700 1786565892.494381 trunc from 0x0367e9 to 0x0394fd (0x2d15 bytes) 000701 1786565892.535679 trunc from 0x011292 to 0x015e00 (0x4b6f bytes) 000702 1786565892.543781 trunc from 0x033e15 to 0x0394fd (0x56e9 bytes) 000703 1786565892.551475 fallocate 0x02d882 thru 0x02eda0 (0x151f bytes) 000704 1786565892.556814 trunc from 0x0217cb to 0x031729 (0xff5f bytes) 000705 1786565892.564791 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394fe): Mirror 10: 0x4f263eb9 (stale) Mirror 12: 0x4f263eb9 (stale) Mirror 13: 0x4f263eb9 (stale) Mirror 15: 0x4f263eb9 (stale) Mirror 16: 0x4f263eb9 (stale) Mirror 17: 0x4f263eb9 (stale) Mirror 18: 0x4f263eb9 (stale) Mirror 19: 0x4f263eb9 (stale) Mirror 20: 0x4f263eb9 (stale) Mirror 21: 0x4f263eb9 (stale) Mirror 22: 0x4f263eb9 (stale) Mirror 23: 0x4f263eb9 (stale) Mirror 24: 0xcc88d908 (stale) Mirror 25: 0xcc88d908 (stale) CRC-32 checksum value for chunk [0, 0x394fe): Mirror 1: 0xf949ab8b Mirror 26: 0xf949ab8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394fe: skipped 000706 1786565893.937708 mapread 0x021a40 thru 0x02c9ca (0xaf8b bytes) 000708 1786565893.990892 fallocate 0x03edb5 thru 0x03ffff (0x124b bytes) 000709 1786565894.046088 punch from 0x009095 to 0x017c69 (0xebd5 bytes) 000711 1786565894.051866 read 0x025664 thru 0x034b3b (0xf4d8 bytes) 000712 1786565894.062506 read 0x0012e7 thru 0x010c96 (0xf9b0 bytes) 000713 1786565894.077146 trunc from 0x040000 to 0x02da68 (0x12597 bytes) 000714 1786565894.107590 trunc from 0x02da69 to 0x031c9f (0x4237 bytes) 000715 1786565894.128947 fallocate 0x02f659 thru 0x035b40 (0x64e8 bytes) 000716 1786565894.134545 trunc from 0x01707a to 0x0231b9 (0xc140 bytes) 000717 1786565894.140747 write 0x02ce24 thru 0x02d364 (0x541 bytes) 000719 1786565894.149336 MIRROR_RESYNC 16 mirrors 000720 1786565897.638134 read 0x02f393 thru 0x035b40 (0x67ae bytes) 000721 1786565897.646156 mapread 0x002f94 thru 0x004094 (0x1101 bytes) 000722 1786565897.668959 read 0x033161 thru 0x035b40 (0x29e0 bytes) 000723 1786565897.669403 trunc from 0x035b41 to 0x01296e (0x231d2 bytes) 000724 1786565897.717830 trunc from 0x00e3ab to 0x011f83 (0x3bd9 bytes) 000725 1786565897.729489 MIRROR_RESYNC 16 mirrors 000726 1786565899.948173 trunc from 0x01296f to 0x03e011 (0x2b6a3 bytes) 000727 1786565899.994082 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e012): Mirror 10: 0x9b205460 (stale) Mirror 12: 0x9b205460 (stale) Mirror 15: 0x9b205460 (stale) Mirror 16: 0x9b205460 (stale) Mirror 17: 0x9b205460 (stale) Mirror 18: 0x9b205460 (stale) Mirror 19: 0x9b205460 (stale) Mirror 20: 0x9b205460 (stale) Mirror 21: 0x9b205460 (stale) Mirror 22: 0x9b205460 (stale) Mirror 23: 0x9b205460 (stale) Mirror 24: 0x9b205460 (stale) Mirror 25: 0x9b205460 (stale) Mirror 26: 0x9b205460 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e012): Mirror 10: 0x9b205460 (stale) Mirror 12: 0x9b205460 (stale) Mirror 15: 0x9b205460 (stale) Mirror 16: 0x9b205460 (stale) Mirror 17: 0x9b205460 (stale) Mirror 18: 0x9b205460 (stale) Mirror 19: 0x9b205460 (stale) Mirror 20: 0x9b205460 (stale) Mirror 21: 0x9b205460 (stale) Mirror 22: 0x9b205460 (stale) Mirror 23: 0x9b205460 (stale) Mirror 24: 0x9b205460 (stale) Mirror 25: 0x9b205460 (stale) Mirror 26: 0x9b205460 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e012: skipped 000728 1786565900.651720 trunc from 0x006ad5 to 0x015ae1 (0xf00d bytes) 000729 1786565900.658687 read 0x018674 thru 0x0197b0 (0x113d bytes) 000730 1786565900.669472 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3e012): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e012): Mirror 10: 0x9b205460 (stale) Mirror 12: 0x9b205460 (stale) Mirror 15: 0x9b205460 (stale) Mirror 16: 0x9b205460 (stale) Mirror 17: 0x9b205460 (stale) Mirror 18: 0x9b205460 (stale) Mirror 19: 0x9b205460 (stale) Mirror 20: 0x9b205460 (stale) Mirror 21: 0x9b205460 (stale) Mirror 22: 0x9b205460 (stale) Mirror 23: 0x9b205460 (stale) Mirror 24: 0x9b205460 (stale) Mirror 25: 0x9b205460 (stale) Mirror 26: 0x9b205460 (stale) CRC-32 checksum value for chunk [0, 0x3e012): Mirror 1: 0x3db96070 Mirror 27: 0x3db96070 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e012: skipped 000731 1786565901.912568 punch from 0x010fd7 to 0x01ff0e (0xef38 bytes) 000732 1786565901.990964 write 0x014cf2 thru 0x015818 (0xb27 bytes) 000733 1786565902.012380 mapread 0x02a2d6 thru 0x02ae56 (0xb81 bytes) 000734 1786565902.020709 trunc from 0x00493d to 0x008613 (0x3cd7 bytes) 000735 1786565902.027691 read 0x00eea4 thru 0x01eddb (0xff38 bytes) 000736 1786565902.049748 mapread 0x008c48 thru 0x0182f0 (0xf6a9 bytes) 000737 1786565902.070722 punch from 0x03dd3e to 0x03e011 (0x2d4 bytes) 000738 1786565902.087138 mapread 0x0339b1 thru 0x03a5f4 (0x6c44 bytes) 000739 1786565902.099241 mapread 0x018e2f thru 0x0246a3 (0xb875 bytes) 000740 1786565902.110967 punch from 0x01aace to 0x02630e (0xb841 bytes) 000741 1786565902.116451 trunc from 0x03e012 to 0x01902f (0x24fe2 bytes) 000742 1786565902.146594 mapwrite 0x025429 thru 0x032bb8 (0xd790 bytes) 000743 1786565902.216613 mapread 0x013e1e thru 0x014ccd (0xeb0 bytes) 000745 1786565902.219164 write 0x00f8d6 thru 0x01c981 (0xd0ac bytes) 000746 1786565902.228189 punch from 0x017749 to 0x0214dd (0x9d95 bytes) 000747 1786565902.248253 write 0x00d1ce thru 0x018344 (0xb177 bytes) 000748 1786565902.255611 punch from 0x02b707 to 0x032bb8 (0x74b2 bytes) 000750 1786565902.263607 read 0x029716 thru 0x032bb8 (0x94a3 bytes) 000751 1786565902.274267 read 0x0264ac thru 0x032bb8 (0xc70d bytes) 000752 1786565902.275804 trunc from 0x021742 to 0x029752 (0x8011 bytes) 000754 1786565902.282534 trunc from 0x032bb9 to 0x008d1b (0x29e9d bytes) 000755 1786565902.299165 fallocate 0x031b55 thru 0x03474d (0x2bf9 bytes) 000757 1786565902.309050 read 0x0037c0 thru 0x008d1b (0x555c bytes) 000758 1786565902.310754 read 0x0088e6 thru 0x008d1b (0x436 bytes) 000759 1786565902.312069 trunc from 0x00444c to 0x008d1b (0x48d0 bytes) 000760 1786565902.323111 mapwrite 0x03787e thru 0x03ffff (0x8782 bytes) 000761 1786565902.397194 trunc from 0x0273e2 to 0x033bb5 (0xc7d4 bytes) 000762 1786565902.402333 write 0x01d52e thru 0x02d0ae (0xfb81 bytes) 000763 1786565902.411991 mapread 0x02e1d3 thru 0x0361ec (0x801a bytes) 000764 1786565902.454289 trunc from 0x040000 to 0x02ff38 (0x100c7 bytes) 000765 1786565902.476235 mapwrite 0x021cc2 thru 0x0308a4 (0xebe3 bytes) 000766 1786565902.522236 write 0x0145a6 thru 0x01863a (0x4095 bytes) 000767 1786565902.523627 fallocate 0x00ac45 thru 0x010263 (0x561f bytes) 000768 1786565902.530055 mapread 0x02d5b0 thru 0x0308a4 (0x32f5 bytes) 000769 1786565902.531964 read 0x007708 thru 0x0144c0 (0xcdb9 bytes) 000770 1786565902.533111 mapread 0x00106d thru 0x008eaa (0x7e3e bytes) 000771 1786565902.535675 trunc from 0x024178 to 0x02c645 (0x84ce bytes) 000773 1786565902.543126 trunc from 0x02c585 to 0x0308a4 (0x4320 bytes) 000774 1786565902.548816 fallocate 0x02dc22 thru 0x035bb3 (0x7f92 bytes) 000775 1786565902.554624 read 0x01c699 thru 0x021123 (0x4a8b bytes) 000776 1786565902.556617 mapwrite 0x03e907 thru 0x03ffff (0x16f9 bytes) 000777 1786565902.604767 write 0x00b26f thru 0x01a62f (0xf3c1 bytes) 000778 1786565902.608473 punch from 0x02d4b6 to 0x03566d (0x81b8 bytes) 000780 1786565902.615424 read 0x03a2f7 thru 0x03ffff (0x5d09 bytes) 000781 1786565902.616474 read 0x02e8ae thru 0x033991 (0x50e4 bytes) 000782 1786565902.626370 trunc from 0x015683 to 0x02318a (0xdb08 bytes) 000784 1786565902.644364 trunc from 0x040000 to 0x00e570 (0x31a8f bytes) 000785 1786565902.661154 mapread 0x00d107 thru 0x00e570 (0x146a bytes) 000786 1786565902.663653 mapwrite 0x02fc7c thru 0x039593 (0x9918 bytes) 000787 1786565902.777037 trunc from 0x039594 to 0x00417b (0x35418 bytes) 000788 1786565902.796597 write 0x01c122 thru 0x02374e (0x762d bytes) 000790 1786565902.800570 trunc from 0x002b19 to 0x00a2c0 (0x77a8 bytes) 000792 1786565902.808137 write 0x01f1c6 thru 0x02ca8d (0xd8c8 bytes) 000793 1786565902.810232 trunc from 0x02ca8e to 0x0033cd (0x296c0 bytes) 000794 1786565902.831045 write 0x011051 thru 0x0190a2 (0x8052 bytes) 000795 1786565902.834758 punch from 0x013dd4 to 0x014b26 (0xd53 bytes) 000796 1786565902.849637 mapwrite 0x01d52f thru 0x01f97e (0x2450 bytes) 000797 1786565902.900873 mapread 0x0175b0 thru 0x01d43b (0x5e8c bytes) 000798 1786565902.917255 mapread 0x013b6d thru 0x01a20e (0x66a2 bytes) 000800 1786565902.928776 fallocate 0x037ff1 thru 0x03c755 (0x4765 bytes) 000801 1786565902.939481 MIRROR_RESYNC 16 mirrors 000802 1786565906.559758 fallocate 0x0205c8 thru 0x022f6a (0x29a3 bytes) 000803 1786565906.634208 trunc from 0x03c756 to 0x00a442 (0x32313 bytes) 000804 1786565906.682541 fallocate 0x02b60d thru 0x034094 (0x8a88 bytes) 000805 1786565906.701025 read 0x0018ff thru 0x0042ec (0x29ee bytes) 000806 1786565906.715936 punch from 0x0061c8 to 0x00a442 (0x427b bytes) 000807 1786565906.723751 mapwrite 0x035f91 thru 0x03fa8c (0x9afc bytes) 000808 1786565906.797022 punch from 0x012bc8 to 0x0187e3 (0x5c1c bytes) 000809 1786565906.824389 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa8d): Mirror 10: 0x5e055e07 (stale) Mirror 12: 0x5e055e07 (stale) Mirror 15: 0x5e055e07 (stale) Mirror 16: 0x5e055e07 (stale) Mirror 17: 0x5e055e07 (stale) Mirror 18: 0x5e055e07 (stale) Mirror 19: 0x5e055e07 (stale) Mirror 20: 0x5e055e07 (stale) Mirror 21: 0x5e055e07 (stale) Mirror 22: 0x5e055e07 (stale) Mirror 23: 0x5e055e07 (stale) Mirror 24: 0x5e055e07 (stale) Mirror 26: 0x5e055e07 (stale) Mirror 27: 0x5e055e07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa8d): Mirror 10: 0x5e055e07 (stale) Mirror 12: 0x5e055e07 (stale) Mirror 15: 0x5e055e07 (stale) Mirror 16: 0x5e055e07 (stale) Mirror 17: 0x5e055e07 (stale) Mirror 18: 0x5e055e07 (stale) Mirror 19: 0x5e055e07 (stale) Mirror 20: 0x5e055e07 (stale) Mirror 21: 0x5e055e07 (stale) Mirror 22: 0x5e055e07 (stale) Mirror 23: 0x5e055e07 (stale) Mirror 24: 0x5e055e07 (stale) Mirror 26: 0x5e055e07 (stale) Mirror 27: 0x5e055e07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa8d: skipped 000810 1786565907.956798 trunc from 0x02f257 to 0x03a7de (0xb588 bytes) 000811 1786565907.964937 trunc from 0x03fa8d to 0x031a17 (0xe075 bytes) 000812 1786565907.994982 trunc from 0x01d964 to 0x02898b (0xb028 bytes) 000813 1786565907.999991 trunc from 0x031a18 to 0x00a480 (0x27597 bytes) 000814 1786565908.021690 trunc from 0x00517c to 0x008d20 (0x3ba5 bytes) 000815 1786565908.028040 mapwrite 0x00e993 thru 0x01cf37 (0xe5a5 bytes) 000816 1786565908.097804 write 0x01fb58 thru 0x02bb30 (0xbfd9 bytes) 000817 1786565908.100713 write 0x01f69c thru 0x0242bf (0x4c24 bytes) 000818 1786565908.102208 read 0x015a86 thru 0x02480a (0xed85 bytes) 000819 1786565908.108033 fallocate 0x005788 thru 0x009a58 (0x42d1 bytes) 000820 1786565908.113781 write 0x01ddf9 thru 0x027855 (0x9a5d bytes) 000821 1786565908.119636 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bb31): Mirror 10: 0x9426230b (stale) Mirror 12: 0x9426230b (stale) Mirror 16: 0x9426230b (stale) Mirror 17: 0x9426230b (stale) Mirror 18: 0x9426230b (stale) Mirror 19: 0x9426230b (stale) Mirror 20: 0x9426230b (stale) Mirror 21: 0x9426230b (stale) Mirror 22: 0x9426230b (stale) Mirror 23: 0x9426230b (stale) Mirror 24: 0x9426230b (stale) Mirror 26: 0x9426230b (stale) Mirror 27: 0x9426230b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb31): Mirror 10: 0x9426230b (stale) Mirror 12: 0x9426230b (stale) Mirror 16: 0x9426230b (stale) Mirror 17: 0x9426230b (stale) Mirror 18: 0x9426230b (stale) Mirror 19: 0x9426230b (stale) Mirror 20: 0x9426230b (stale) Mirror 21: 0x9426230b (stale) Mirror 22: 0x9426230b (stale) Mirror 23: 0x9426230b (stale) Mirror 24: 0x9426230b (stale) Mirror 26: 0x9426230b (stale) Mirror 27: 0x9426230b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb31: skipped 000822 1786565908.847028 mapread 0x008d34 thru 0x00993e (0xc0b bytes) 000823 1786565908.859463 read 0x015689 thru 0x01971f (0x4097 bytes) 000825 1786565908.874212 write 0x0276b6 thru 0x02dea5 (0x67f0 bytes) 000826 1786565908.884692 fallocate 0x00fbe3 thru 0x0181aa (0x85c8 bytes) 000827 1786565908.889125 write 0x00ade3 thru 0x00ef4b (0x4169 bytes) 000828 1786565908.900548 trunc from 0x02dea6 to 0x03bf45 (0xe0a0 bytes) 000829 1786565908.931836 mapread 0x018788 thru 0x0212fd (0x8b76 bytes) 000830 1786565908.952768 trunc from 0x03bf46 to 0x01ea33 (0x1d512 bytes) 000831 1786565908.985934 fallocate 0x03eb0a thru 0x03ffff (0x14f6 bytes) 000832 1786565908.999813 write 0x00a78a thru 0x0140a7 (0x991e bytes) 000833 1786565909.002812 mapwrite 0x0166dc thru 0x01d042 (0x6967 bytes) 000834 1786565909.057620 punch from 0x01e775 to 0x01ea33 (0x2bf bytes) 000835 1786565909.064538 trunc from 0x01ea34 to 0x020642 (0x1c0f bytes) 000836 1786565909.114547 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x20643): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20643): Mirror 10: 0x61fab9e6 (stale) Mirror 12: 0x61fab9e6 (stale) Mirror 16: 0x61fab9e6 (stale) Mirror 17: 0x61fab9e6 (stale) Mirror 18: 0x61fab9e6 (stale) Mirror 19: 0x61fab9e6 (stale) Mirror 20: 0x61fab9e6 (stale) Mirror 21: 0x61fab9e6 (stale) Mirror 22: 0x61fab9e6 (stale) Mirror 23: 0x61fab9e6 (stale) Mirror 24: 0x61fab9e6 (stale) Mirror 26: 0x61fab9e6 (stale) Mirror 27: 0x61fab9e6 (stale) CRC-32 checksum value for chunk [0, 0x20643): Mirror 1: 0x64cfd9cf Mirror 28: 0x64cfd9cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20643: skipped 000837 1786565910.682131 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x20643): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20643): Mirror 10: 0x61fab9e6 (stale) Mirror 12: 0x61fab9e6 (stale) Mirror 16: 0x61fab9e6 (stale) Mirror 17: 0x61fab9e6 (stale) Mirror 18: 0x61fab9e6 (stale) Mirror 19: 0x61fab9e6 (stale) Mirror 20: 0x61fab9e6 (stale) Mirror 21: 0x61fab9e6 (stale) Mirror 22: 0x61fab9e6 (stale) Mirror 23: 0x61fab9e6 (stale) Mirror 24: 0x61fab9e6 (stale) Mirror 26: 0x61fab9e6 (stale) Mirror 27: 0x61fab9e6 (stale) CRC-32 checksum value for chunk [0, 0x20643): Mirror 1: 0x64cfd9cf Mirror 28: 0x64cfd9cf Mirror 29: 0x64cfd9cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20643: skipped 000838 1786565911.817765 write 0x012fa6 thru 0x013dd7 (0xe32 bytes) 000839 1786565911.934544 MIRROR_RESYNC 16 mirrors 000840 1786565915.510125 write 0x01b5e7 thru 0x01d8ae (0x22c8 bytes) 000841 1786565915.582732 punch from 0x011736 to 0x01b271 (0x9b3c bytes) 000842 1786565915.598009 punch from 0x011643 to 0x01b290 (0x9c4e bytes) 000844 1786565915.605692 write 0x029fe0 thru 0x02c26c (0x228d bytes) 000846 1786565915.612812 trunc from 0x01e397 to 0x0253e2 (0x704c bytes) 000847 1786565915.629257 punch from 0x0081aa to 0x010c98 (0x8aef bytes) 000848 1786565915.638300 write 0x02d1cd thru 0x034366 (0x719a bytes) 000849 1786565915.641085 trunc from 0x002987 to 0x00ac82 (0x82fc bytes) 000850 1786565915.650613 fallocate 0x000740 thru 0x0071b1 (0x6a72 bytes) 000851 1786565915.659242 write 0x019f18 thru 0x021a78 (0x7b61 bytes) 000852 1786565915.677135 fallocate 0x0301d4 thru 0x033df6 (0x3c23 bytes) 000853 1786565915.688778 punch from 0x02b69c to 0x02ffc8 (0x492d bytes) 000854 1786565915.730139 fallocate 0x01884a thru 0x020aa8 (0x825f bytes) 000855 1786565915.744660 mapread 0x01f7e6 thru 0x022235 (0x2a50 bytes) 000856 1786565915.774389 read 0x003420 thru 0x01111b (0xdcfc bytes) 000858 1786565915.817045 trunc from 0x034367 to 0x03c3d7 (0x8071 bytes) 000859 1786565915.853572 write 0x029f35 thru 0x0308c3 (0x698f bytes) 000860 1786565915.856101 trunc from 0x02f598 to 0x02f945 (0x3ae bytes) 000861 1786565915.876694 punch from 0x03815b to 0x039d5c (0x1c02 bytes) 000862 1786565915.882969 fallocate 0x000785 thru 0x002684 (0x1f00 bytes) 000863 1786565915.888654 mapread 0x01525b thru 0x019cad (0x4a53 bytes) 000864 1786565915.907226 write 0x02efc3 thru 0x03a3bb (0xb3f9 bytes) 000867 1786565915.917080 write 0x0051df thru 0x006d0b (0x1b2d bytes) 000868 1786565915.918984 read 0x009728 thru 0x00b649 (0x1f22 bytes) 000869 1786565915.925317 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c3d8): Mirror 12: 0xc850a7cf (stale) Mirror 16: 0xc850a7cf (stale) Mirror 17: 0xc850a7cf (stale) Mirror 18: 0xc850a7cf (stale) Mirror 19: 0xc850a7cf (stale) Mirror 20: 0xc850a7cf (stale) Mirror 21: 0xc850a7cf (stale) Mirror 22: 0xc850a7cf (stale) Mirror 23: 0xc850a7cf (stale) Mirror 24: 0xc850a7cf (stale) Mirror 26: 0xc850a7cf (stale) Mirror 27: 0xc850a7cf (stale) Mirror 28: 0xc850a7cf (stale) Mirror 29: 0xc850a7cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3d8): Mirror 12: 0xc850a7cf (stale) Mirror 16: 0xc850a7cf (stale) Mirror 17: 0xc850a7cf (stale) Mirror 18: 0xc850a7cf (stale) Mirror 19: 0xc850a7cf (stale) Mirror 20: 0xc850a7cf (stale) Mirror 21: 0xc850a7cf (stale) Mirror 22: 0xc850a7cf (stale) Mirror 23: 0xc850a7cf (stale) Mirror 24: 0xc850a7cf (stale) Mirror 26: 0xc850a7cf (stale) Mirror 27: 0xc850a7cf (stale) Mirror 28: 0xc850a7cf (stale) Mirror 29: 0xc850a7cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3d8: skipped 000870 1786565916.797293 punch from 0x0115fa to 0x01eebe (0xd8c5 bytes) 000871 1786565916.805837 mapread 0x015eeb thru 0x01d1ba (0x72d0 bytes) 000872 1786565916.828894 trunc from 0x006013 to 0x006ef4 (0xee2 bytes) 000873 1786565916.838298 punch from 0x033006 to 0x03bf76 (0x8f71 bytes) 000875 1786565916.846700 write 0x02c013 thru 0x03066b (0x4659 bytes) 000876 1786565916.865302 trunc from 0x024476 to 0x031326 (0xceb1 bytes) 000877 1786565916.883696 read 0x00799c thru 0x00c0cb (0x4730 bytes) 000878 1786565916.893148 punch from 0x00ade7 to 0x016edc (0xc0f6 bytes) 000879 1786565916.898434 trunc from 0x03c3d8 to 0x02379a (0x18c3d bytes) 000880 1786565916.920034 write 0x009caa thru 0x0171ee (0xd545 bytes) 000881 1786565916.923209 mapwrite 0x03a7c2 thru 0x03ffff (0x583e bytes) 000882 1786565916.972130 trunc from 0x040000 to 0x030c18 (0xf3e7 bytes) 000883 1786565916.994229 write 0x03626e thru 0x03ffff (0x9d92 bytes) 000884 1786565916.996002 read 0x010aa0 thru 0x01a644 (0x9ba5 bytes) 000885 1786565916.997756 write 0x026e00 thru 0x0305e5 (0x97e6 bytes) 000886 1786565917.010137 punch from 0x02d651 to 0x0369e8 (0x9398 bytes) 000887 1786565917.027071 mapwrite 0x025b43 thru 0x028c41 (0x30ff bytes) 000888 1786565917.050382 MIRROR_RESYNC 15 mirrors 000892 1786565921.553411 fallocate 0x03bc93 thru 0x03ffff (0x436d bytes) 000893 1786565921.603350 trunc from 0x040000 to 0x01515e (0x2aea1 bytes) 000894 1786565921.635787 fallocate 0x021ff6 thru 0x02c2de (0xa2e9 bytes) 000895 1786565921.642016 trunc from 0x010572 to 0x01515e (0x4bed bytes) 000897 1786565921.647716 fallocate 0x0370bc thru 0x03ffff (0x8f44 bytes) 000898 1786565921.657415 trunc from 0x040000 to 0x03637f (0x9c80 bytes) 000900 1786565921.673052 write 0x03c1b9 thru 0x03ebbb (0x2a03 bytes) 000901 1786565921.675484 punch from 0x026490 to 0x029c1b (0x378c bytes) 000902 1786565921.687816 MIRROR_RESYNC 15 mirrors 000903 1786565925.268225 trunc from 0x03ebbc to 0x00b547 (0x33674 bytes) 000904 1786565925.333661 mapread 0x005af4 thru 0x00b547 (0x5a54 bytes) 000905 1786565925.393137 mapwrite 0x01bbb1 thru 0x028e93 (0xd2e3 bytes) 000906 1786565925.490938 trunc from 0x00d110 to 0x01a739 (0xd62a bytes) 000907 1786565925.497174 trunc from 0x028e94 to 0x03e9f5 (0x15b62 bytes) 000908 1786565925.519225 trunc from 0x00553a to 0x00c4a9 (0x6f70 bytes) 000909 1786565925.531958 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9f6): Mirror 12: 0xe5ab0b1b (stale) Mirror 16: 0xe5ab0b1b (stale) Mirror 17: 0xe5ab0b1b (stale) Mirror 18: 0xe5ab0b1b (stale) Mirror 19: 0xe5ab0b1b (stale) Mirror 20: 0xe5ab0b1b (stale) Mirror 21: 0xe5ab0b1b (stale) Mirror 22: 0xe5ab0b1b (stale) Mirror 23: 0xe5ab0b1b (stale) Mirror 24: 0xe5ab0b1b (stale) Mirror 26: 0xe5ab0b1b (stale) Mirror 28: 0xe5ab0b1b (stale) Mirror 29: 0xe5ab0b1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9f6): Mirror 12: 0xe5ab0b1b (stale) Mirror 16: 0xe5ab0b1b (stale) Mirror 17: 0xe5ab0b1b (stale) Mirror 18: 0xe5ab0b1b (stale) Mirror 19: 0xe5ab0b1b (stale) Mirror 20: 0xe5ab0b1b (stale) Mirror 21: 0xe5ab0b1b (stale) Mirror 22: 0xe5ab0b1b (stale) Mirror 23: 0xe5ab0b1b (stale) Mirror 24: 0xe5ab0b1b (stale) Mirror 26: 0xe5ab0b1b (stale) Mirror 28: 0xe5ab0b1b (stale) Mirror 29: 0xe5ab0b1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9f6: skipped 000910 1786565926.582815 trunc from 0x000041 to 0x0014bc (0x147c bytes) 000911 1786565926.591121 trunc from 0x03e9f6 to 0x00ac30 (0x33dc5 bytes) 000912 1786565926.658163 write 0x014550 thru 0x0155ec (0x109d bytes) 000913 1786565926.659511 trunc from 0x0155ed to 0x0178dc (0x22f0 bytes) 000914 1786565926.737711 trunc from 0x01470b to 0x0178dc (0x31d2 bytes) 000915 1786565926.777577 write 0x03dc53 thru 0x03ffff (0x23ad bytes) 000916 1786565926.778766 trunc from 0x03839b to 0x03c534 (0x419a bytes) 000917 1786565926.787588 read 0x01edf3 thru 0x02af2c (0xc13a bytes) 000919 1786565926.808020 mapread 0x02e913 thru 0x0386db (0x9dc9 bytes) 000921 1786565926.832445 trunc from 0x040000 to 0x01e01e (0x21fe1 bytes) 000922 1786565926.866438 write 0x025eff thru 0x030397 (0xa499 bytes) 000923 1786565926.872730 mapread 0x02157e thru 0x02257e (0x1001 bytes) 000924 1786565926.889195 mapread 0x00d561 thru 0x01426c (0x6d0c bytes) 000925 1786565926.905180 mapread 0x017116 thru 0x019f8a (0x2e75 bytes) 000926 1786565926.915129 trunc from 0x030398 to 0x021dd5 (0xe5c2 bytes) 000927 1786565926.939769 write 0x01ff99 thru 0x02d779 (0xd7e1 bytes) 000928 1786565926.944448 trunc from 0x00ffea to 0x013792 (0x37a9 bytes) 000930 1786565926.953433 fallocate 0x02ef70 thru 0x031959 (0x29ea bytes) 000932 1786565926.962585 read 0x0180f5 thru 0x01ba56 (0x3962 bytes) 000933 1786565926.967202 trunc from 0x03195a to 0x014dd2 (0x1cb87 bytes) 000934 1786565926.992522 mapread 0x00e86f thru 0x01323a (0x49cc bytes) 000935 1786565927.008805 mapread 0x00e624 thru 0x00f203 (0xbe0 bytes) 000936 1786565927.011006 read 0x00f8bf thru 0x014dd2 (0x5514 bytes) 000937 1786565927.021084 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x14dd3): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14dd3): Mirror 12: 0xceb7a1e1 (stale) Mirror 16: 0xceb7a1e1 (stale) Mirror 17: 0xceb7a1e1 (stale) Mirror 18: 0xceb7a1e1 (stale) Mirror 19: 0xceb7a1e1 (stale) Mirror 20: 0xceb7a1e1 (stale) Mirror 21: 0xceb7a1e1 (stale) Mirror 22: 0xceb7a1e1 (stale) Mirror 23: 0xceb7a1e1 (stale) Mirror 24: 0xceb7a1e1 (stale) Mirror 26: 0xceb7a1e1 (stale) Mirror 28: 0xceb7a1e1 (stale) Mirror 29: 0xceb7a1e1 (stale) CRC-32 checksum value for chunk [0, 0x14dd3): Mirror 1: 0x5ef55ad7 Mirror 30: 0x5ef55ad7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14dd3: skipped 000938 1786565928.754771 read 0x009ec8 thru 0x014dd2 (0xaf0b bytes) 000939 1786565928.789039 mapwrite 0x009696 thru 0x010445 (0x6db0 bytes) 000941 1786565929.046459 punch from 0x00c6df to 0x014dd2 (0x86f4 bytes) 000942 1786565929.071193 MIRROR_RESYNC 15 mirrors 000943 1786565932.708535 punch from 0x00ced5 to 0x014c2d (0x7d59 bytes) 000944 1786565932.755119 write 0x01f832 thru 0x022c16 (0x33e5 bytes) 000945 1786565932.756422 write 0x0305a9 thru 0x03c9d5 (0xc42d bytes) 000946 1786565932.757597 mapwrite 0x036df2 thru 0x0386cf (0x18de bytes) 000947 1786565932.782069 MIRROR_RESYNC 15 mirrors 000948 1786565937.197557 read 0x005c2a thru 0x0148fc (0xecd3 bytes) 000949 1786565937.229390 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000950 1786565937.890116 write 0x03497e thru 0x03dd02 (0x9385 bytes) 000951 1786565937.977893 mapwrite 0x023372 thru 0x02351b (0x1aa bytes) 000952 1786565938.036012 mapwrite 0x012174 thru 0x012ad6 (0x963 bytes) 000953 1786565938.075387 write 0x012a88 thru 0x019e24 (0x739d bytes) 000954 1786565938.083201 fallocate 0x00854f thru 0x017f4c (0xf9fe bytes) 000955 1786565938.089770 fallocate 0x02d221 thru 0x0385b9 (0xb399 bytes) 000957 1786565938.102478 mapwrite 0x0319c8 thru 0x03cca4 (0xb2dd bytes) 000958 1786565938.154434 trunc from 0x037f01 to 0x03dd02 (0x5e02 bytes) 000959 1786565938.170380 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3dd03): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd03): Mirror 12: 0xba0f828d (stale) Mirror 16: 0xba0f828d (stale) Mirror 17: 0xba0f828d (stale) Mirror 18: 0xba0f828d (stale) Mirror 19: 0xba0f828d (stale) Mirror 20: 0xba0f828d (stale) Mirror 22: 0xba0f828d (stale) Mirror 23: 0xba0f828d (stale) Mirror 24: 0xba0f828d (stale) Mirror 26: 0xba0f828d (stale) Mirror 28: 0xba0f828d (stale) Mirror 29: 0xba0f828d (stale) Mirror 30: 0xba0f828d (stale) CRC-32 checksum value for chunk [0, 0x3dd03): Mirror 1: 0x4b6d76d9 Mirror 31: 0x4b6d76d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd03: skipped 000960 1786565940.147960 trunc from 0x03dd03 to 0x000105 (0x3dbfd bytes) 000961 1786565940.315221 read 0x000018 thru 0x000105 (0xee bytes) 000962 1786565940.326666 mapwrite 0x032fb8 thru 0x033c3c (0xc85 bytes) 000963 1786565940.373612 mapwrite 0x00b9f6 thru 0x015196 (0x97a1 bytes) 000964 1786565940.414746 write 0x0384e8 thru 0x03ffff (0x7b18 bytes) 000965 1786565940.416314 trunc from 0x010d54 to 0x01ddcd (0xd07a bytes) 000966 1786565940.422897 write 0x00099b thru 0x005eac (0x5512 bytes) 000967 1786565940.428450 punch from 0x032dab to 0x038930 (0x5b86 bytes) 000969 1786565940.452092 fallocate 0x003b86 thru 0x00dc44 (0xa0bf bytes) 000970 1786565940.465300 read 0x021369 thru 0x026c2b (0x58c3 bytes) 000971 1786565940.484272 trunc from 0x00d8d6 to 0x01a030 (0xc75b bytes) 000972 1786565940.519025 MIRROR_RESYNC 15 mirrors 000973 1786565944.357463 mapread 0x03ea75 thru 0x03ffff (0x158b bytes) 000974 1786565944.366717 mapwrite 0x02995b thru 0x037646 (0xdcec bytes) 000975 1786565944.461294 write 0x01c688 thru 0x01ef89 (0x2902 bytes) 000976 1786565944.481704 trunc from 0x03c003 to 0x03ffff (0x3ffd bytes) 000977 1786565944.495679 trunc from 0x040000 to 0x00566e (0x3a991 bytes) 000978 1786565944.535588 write 0x02cc2a thru 0x02d457 (0x82e bytes) 000979 1786565944.538778 read 0x027fcb thru 0x02d457 (0x548d bytes) 000980 1786565944.547660 read 0x013615 thru 0x01d8c9 (0xa2b5 bytes) 000981 1786565944.559960 read 0x010a3a thru 0x01ed00 (0xe2c7 bytes) 000982 1786565944.560795 punch from 0x02ba85 to 0x02d457 (0x19d3 bytes) 000983 1786565944.572652 punch from 0x000d36 to 0x00d7c4 (0xca8f bytes) 000984 1786565944.578723 mapwrite 0x03763a thru 0x03b5d9 (0x3fa0 bytes) 000985 1786565944.625798 read 0x0049a1 thru 0x006d3f (0x239f bytes) 000986 1786565944.626596 trunc from 0x03b5da to 0x034419 (0x71c0 bytes) 000987 1786565944.647023 mapread 0x027afc thru 0x031ad2 (0x9fd7 bytes) 000988 1786565944.650295 mapread 0x004fa0 thru 0x0145f5 (0xf656 bytes) 000989 1786565944.652792 mapread 0x023794 thru 0x02ef1b (0xb788 bytes) 000990 1786565944.654764 fallocate 0x019fbc thru 0x01e13b (0x4180 bytes) 000991 1786565944.664356 fallocate 0x0238b7 thru 0x02a4a8 (0x6bf2 bytes) 000992 1786565944.670311 punch from 0x0070d7 to 0x00dc76 (0x6ba0 bytes) 000993 1786565944.675189 mapwrite 0x002330 thru 0x00af84 (0x8c55 bytes) 000995 1786565944.711316 trunc from 0x03441a to 0x015bf2 (0x1e827 bytes) 000996 1786565944.737836 punch from 0x00e9d1 to 0x015bf2 (0x7222 bytes) 000997 1786565944.748847 write 0x0191bd thru 0x028769 (0xf5ad bytes) 000998 1786565944.763235 mapread 0x0241ab thru 0x028769 (0x45bf bytes) 000999 1786565944.779324 mapwrite 0x030db7 thru 0x0376a6 (0x68f0 bytes) 001000 1786565944.881304 fallocate 0x011da6 thru 0x016c0a (0x4e65 bytes) All operations completed A-OK! PASS 70a (149s) == sanity-flr test 71: check mirror extend parent fid ==== 16:19:11 (1786565951) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x180:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 16:19:24 (1786565964) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16999 s, 12.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919984 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 0x10004 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 16:19:34 (1786565974) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre2 Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6710368 .. Extending file size to 3257760 .. Extending file size to 7327232 .. Extending file size to 5680928 .. Extending file size to 8575360 .. Extending file size to 7369056 .. Extending file size to 7988544 .. Extending file size to 8055488 .. Extending file size to 8226656 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8807840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9030912 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 61381 61382 61385 61386 61387 192.168.202.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre2 (opts:) 192.168.202.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:21:01 (1786566061) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre2 Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5633632 .. Extending file size to 4186816 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4397280 .. failed Extending file size to 5834400 .. Extending file size to 6241632 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6216192 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6244288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8490176 .. failed Extending file size to 8185600 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996032 .. Extending file size to 8645184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8972384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9366304 .. Extending file size to 9028352 .. Waiting 63085 63086 63088 63089 63090 done 192.168.202.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre2 (opts:) 192.168.202.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:22:20 (1786566140) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1268784 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26176 3580844 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18004 3589016 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44180 7169860 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1268784 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2056 1269248 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26176 3580844 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18004 3589016 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44180 7169860 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0738384 s, 28.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195874 s, 10.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x613da224 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:22:39 (1786566159) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475065 s, 44.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 16:22:47 (1786566167) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x98:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:22:55 (1786566175) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139159 s, 15.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:23:02 (1786566182) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113176 s, 18.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x200:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:23:10 (1786566190) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0975802 s, 21.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:23:43 (1786566223) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg221-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608659 s, 17.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:24:33 (1786566273) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg221-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10907 s, 19.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x201:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:25:24 (1786566324) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg221-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg221-server: Pool lustre.test_204f created Adding targets to pool oleg221-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg221-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111071 s, 18.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x202:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg221-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg221-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg221-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:26:42 (1786566402) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:26:51 (1786566411) lfs mirror mirror: Invalid layout: Wrong flags error: lfs extend: /mnt/lustre/f205b.sanity-flr: cannot create volatile file: Invalid argument sanity-flr test_205b: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4329:test_205b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:4346:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_205b.*.1786566415.log rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.debug_log.oleg221-server.1786566415.log.aJ6qw4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_205b.dmesg.oleg221-server.1786566415.log.tadsQ5" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:27:01 (1786566421) oleg221-server: Pool lustre.test_206 created Adding targets to pool oleg221-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg221-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg221-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg221-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg221-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:27:27 (1786566447) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730104 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206655 s, 50.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa0e49d68 Mirror 3: 0xa0e49d68 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:27:38 (1786566458) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:27:42 (1786566462) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:27:46 (1786566466) Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0472067 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x8b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x8b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x8b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x720000 Read this (4 bytes): BBBB 192.168.202.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg221-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:28:22 (1786566502) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0191806 s, 427 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:28:29 (1786566509) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395089 s, 26.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb5 lmm_fid: [0x200000405:0xb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb5 lmm_fid: [0x200000405:0xb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611317 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 16:28:30.000000000 -0400 Modify: 2026-08-12 16:28:30.000000000 -0400 Change: 2026-08-12 16:28:31.000000000 -0400 Birth: 2026-08-12 16:28:31.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:28:35 (1786566515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217545 s, 48.2 MB/s oleg221-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg221-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg221-server: O/280000401/d31/15f: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:28:59 (1786566539) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.245011 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0359587 s, 114 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 16:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xba:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 16:29 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:29:07 (1786566547) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:29:14 (1786566554) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1944 sec =========== 16:29:20 (1786566560) sanity-flr: FAIL: test_0m failed to call extend /mnt/lustre/d0m.sanity-flr/f0m.sanity-flr sanity-flr: FAIL: test_205b /home/green/git/lustre-release/lustre/utils/lfs mirror extend -N --flags=nocompr /mnt/lustre/f205b.sanity-flr failed === sanity-flr: start cleanup 16:29:22 (1786566562) === === sanity-flr: finish cleanup 16:29:28 (1786566568) ===