-----============= acceptance-small: sanity-flr ============----- Fri Apr 3 01:15:02 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg329-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:15:21 (1775193321) === oleg329-client.virtnet: executing check_config_client /mnt/lustre oleg329-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg329-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ac147ef5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ac147ef5800.idle_timeout=debug disable quota as required oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:16:00 (1775193360) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8194] == sanity-flr test 0a: lfs mirror create with -N option == 01:16:02 (1775193362) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:16:16 (1775193376) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:16:19 (1775193379) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:16:22 (1775193382) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:16:45 (1775193405) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:16:48 (1775193408) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:16:51 (1775193411) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ac147ef5800.stats=clear osc.lustre-OST0001-osc-ffff9ac147ef5800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:16:59 (1775193419) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:17:11 (1775193431) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:17:20 (1775193440) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:17:24 (1775193444) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:17:34 (1775193454) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:17:41 (1775193461) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:17:49 (1775193469) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:17:58 (1775193478) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:18:11 (1775193491) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:18:18 (1775193498) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.416949 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943184 s, 11.1 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:18:25 (1775193505) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.598425 s, 33.3 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:18:34 (1775193514) Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (60s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:19:34 (1775193574) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.555648 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:20:12 (1775193612) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (64s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:21:16 (1775193676) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.58428 s, 20.3 MB/s -rw-r--r-- 1 root root 52428800 Apr 3 01:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.977s user 0m0.002s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.064s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.005s user 0m0.001s sys 0m0.159s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac147ef5800.ost_server_uuid in FULL state after 0 sec PASS 33b (77s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:22:33 (1775193753) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:22:36 (1775193756) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:22:39 (1775193759) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:22:42 (1775193762) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 01:22:49 (1775193769) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.495448 s, 25.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:23:01 (1775193781) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690553 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049091 s, 21.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:23:08 (1775193788) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:23:16 (1775193796) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151178 s, 677 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0023316 s, 439 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00081883 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0021845 s, 469 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:23:27 (1775193807) create test files with size 15747412 .. ff2b71d02459bc378cd423c17ef79116 - cfd39d502be86cf7ec430b42c2246616 - a96a32f45d5f0d97bb1f50164405d60e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 01:23:48 (1775193828) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:23:59 (1775193839) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:24:06 (1775193846) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486694 s, 21.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498367 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 01:24:13 (1775193853) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0957559 s, 21.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.32216 s, 13.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120924 s, 17.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 01:24:25 (1775193865) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:24:28 (1775193868) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:24:31 (1775193871) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577268 s, 18.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 01:24:38 (1775193878) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.200776 s, 15.7 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:24:45 (1775193885) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:24:55 (1775193895) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.257048 s, 40.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:25:05 (1775193905) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9ac143172800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295946 s, 35.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9ac143fb3800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:25:20 (1775193920) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:26:01 (1775193961) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:26:35 (1775193995) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:26:38 (1775193998) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.351318 s, 9.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.232297 s, 13.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf48c93a6 (stale) Mirror 2: 0xf48c93a6 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 95c7ce91142dd043785d5852b288f6aa - mirror 1 checksum: cd47fbb82b13044263ae911c4628288a - mirror 2 checksum: 95c7ce91142dd043785d5852b288f6aa - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 3 01:26:43 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 3 01:26:43 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 95c7ce91142dd043785d5852b288f6aa - mirror 1 checksum: cd47fbb82b13044263ae911c4628288a - mirror 2 checksum: cd47fbb82b13044263ae911c4628288a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 3 01:26:43 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 3 01:26:43 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:26:48 (1775194008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278009 s, 37.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:26:54 (1775194014) Failing mds1 on oleg329-server Stopping /mnt/lustre-mds1 (opts:) on oleg329-server 01:27:06 (1775194026) shut down facet: mds1 facet_host: oleg329-server facet_failover_host: oleg329-server Failover mds1 to oleg329-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 01:27:40 (1775194060) targets are mounted 01:27:40 (1775194060) facet_failover done oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291239 s, 36.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:27:54 (1775194074) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0314398 s, 651 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0115215 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:28:06 (1775194086) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0759145 s, 270 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0207362 s, 988 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:28:18 (1775194098) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00509154 s, 3.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:28:27 (1775194107) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0450841 s, 454 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:28:36 (1775194116) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0064336 s, 637 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0386945 s, 106 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:28:43 (1775194123) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:29:06 (1775194146) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:29:33 (1775194173) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:29:57 (1775194197) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:30:21 (1775194221) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:30:30 (1775194230) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55909 mirror split pid 55910 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7721: 55910 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55909 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 01:31:38 (1775194298) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4299 skipping zero length zero range skipping zero size read truncating to largest ever: 0x39a59 000003 1775194300.292803 trunc from 00000000 to 0x039a58 (0x39a59 bytes) 000004 1775194300.315704 punch from 0x02a298 to 0x034e2a (0xab93 bytes) 000005 1775194300.320844 write 0x017d3b thru 0x01b5a1 (0x3867 bytes) 000006 1775194300.339101 fallocate 0x00dfd9 thru 0x016665 (0x868d bytes) 000007 1775194300.346587 write 0x00ad74 thru 0x017d2a (0xcfb7 bytes) 000008 1775194300.363891 fallocate 0x022790 thru 0x0229b9 (0x22a bytes) 000009 1775194300.370565 fallocate 0x00f609 thru 0x014a82 (0x547a bytes) 000010 1775194300.381645 trunc from 0x039a59 to 0x039292 (0x7c6 bytes) 000011 1775194300.411383 trunc from 0x039293 to 0x028022 (0x11270 bytes) 000012 1775194300.443023 trunc from 0x002f63 to 0x009b2f (0x6bcd bytes) 000013 1775194300.452386 trunc from 0x00f780 to 0x011264 (0x1ae5 bytes) 000014 1775194300.497322 fallocate 0x013701 thru 0x01c822 (0x9122 bytes) 000015 1775194300.507798 trunc from 0x023f32 to 0x028022 (0x40f1 bytes) 000016 1775194300.517841 fallocate 0x0008d9 thru 0x00ea6d (0xe195 bytes) 000017 1775194300.526007 write 0x015bcc thru 0x022d2b (0xd160 bytes) 000018 1775194300.539337 punch from 0x014611 to 0x01d6e1 (0x90d1 bytes) 000019 1775194300.572243 read 0x027a69 thru 0x028022 (0x5ba bytes) 000020 1775194300.582779 read 0x0134b2 thru 0x01d4e3 (0xa032 bytes) 000021 1775194300.594816 mapread 0x02145e thru 0x027492 (0x6035 bytes) 000022 1775194300.595980 mapwrite 0x001d37 thru 0x00c7e7 (0xaab1 bytes) 000023 1775194300.612745 read 0x0093ea thru 0x00a127 (0xd3e bytes) 000026 1775194300.616764 punch from 0x009df2 to 0x016de5 (0xcff4 bytes) 000027 1775194300.631884 trunc from 0x021a60 to 0x023897 (0x1e38 bytes) 000029 1775194300.645178 trunc from 0x028023 to 0x003b80 (0x244a2 bytes) fallocating to largest ever: 0x3bf9c 000031 1775194300.681311 fallocate 0x036703 thru 0x03bf9b (0x5899 bytes) 000033 1775194300.708176 fallocate 0x01e606 thru 0x01fb16 (0x1511 bytes) 000034 1775194300.721944 mapwrite 0x00787e thru 0x008e08 (0x158b bytes) 000036 1775194300.815071 mapread 0x00541a thru 0x008e08 (0x39ef bytes) 000037 1775194300.817042 read 0x003924 thru 0x008e08 (0x54e5 bytes) 000039 1775194300.822635 trunc from 0x008e09 to 0x014e07 (0xbfff bytes) 000040 1775194300.846879 fallocate 0x018ba5 thru 0x01f7e4 (0x6c40 bytes) 000041 1775194300.859587 mapread 0x012ccb thru 0x01eba2 (0xbed8 bytes) 000042 1775194300.872612 read 0x018a61 thru 0x019e9d (0x143d bytes) 000043 1775194300.873531 mapwrite 0x034635 thru 0x036e0f (0x27db bytes) 000044 1775194300.944322 write 0x008d9f thru 0x00f111 (0x6373 bytes) 000045 1775194300.945848 punch from 0x004d93 to 0x00d700 (0x896e bytes) 000046 1775194300.964922 mapwrite 0x02128a thru 0x0292ca (0x8041 bytes) 000047 1775194300.983347 write 0x03529d thru 0x03ffff (0xad63 bytes) 000048 1775194300.984923 punch from 0x03f690 to 0x03ffff (0x970 bytes) 000049 1775194300.996364 trunc from 0x040000 to 0x018d4e (0x272b1 bytes) 000050 1775194301.023862 mapwrite 0x004f48 thru 0x010290 (0xb349 bytes) 000051 1775194301.113491 read 0x00e00b thru 0x017590 (0x9586 bytes) 000052 1775194301.119469 MIRROR_EXTEND to 2 mirrors 000053 1775194301.780420 read 0x013565 thru 0x017531 (0x3fcd bytes) 000054 1775194301.799839 mapread 0x014a3c thru 0x018d4e (0x4313 bytes) 000055 1775194301.826983 fallocate 0x01a51e thru 0x022949 (0x842c bytes) 000056 1775194301.867661 read 0x002caf thru 0x00a18b (0x74dd bytes) 000058 1775194301.879921 trunc from 0x0020df to 0x002d64 (0xc86 bytes) fallocating to largest ever: 0x40000 000059 1775194301.885374 fallocate 0x0397fa thru 0x03ffff (0x6806 bytes) 000060 1775194301.890901 trunc from 0x02294a to 0x025cca (0x3381 bytes) 000061 1775194301.939011 mapread 0x00a72e thru 0x01a597 (0xfe6a bytes) 000062 1775194301.940659 fallocate 0x037190 thru 0x03ffff (0x8e70 bytes) 000063 1775194301.948303 trunc from 0x01cebb to 0x025cca (0x8e10 bytes) 000065 1775194301.958723 punch from 0x025247 to 0x025cca (0xa84 bytes) 000066 1775194301.963718 trunc from 0x01fb42 to 0x025cca (0x6189 bytes) 000067 1775194301.972536 trunc from 0x015848 to 0x01ca60 (0x7219 bytes) 000069 1775194301.979573 trunc from 0x025ccb to 0x029014 (0x334a bytes) 000070 1775194301.994020 trunc from 0x026829 to 0x029014 (0x27ec bytes) 000071 1775194302.000619 fallocate 0x01d171 thru 0x0281cb (0xb05b bytes) 000072 1775194302.006516 mapread 0x02582a thru 0x02880d (0x2fe4 bytes) 000073 1775194302.022631 mapread 0x027404 thru 0x029014 (0x1c11 bytes) 000074 1775194302.024568 mapread 0x012b88 thru 0x01c789 (0x9c02 bytes) 000075 1775194302.070605 mapread 0x021261 thru 0x02149f (0x23f bytes) 000076 1775194302.077452 mapread 0x0245fd thru 0x029014 (0x4a18 bytes) 000077 1775194302.078911 trunc from 0x029015 to 0x02707f (0x1f95 bytes) 000078 1775194302.122536 punch from 0x005762 to 0x011052 (0xb8f1 bytes) 000079 1775194302.131810 read 0x01e544 thru 0x02707f (0x8b3c bytes) 000080 1775194302.142800 write 0x0180de thru 0x024fcb (0xceee bytes) 000081 1775194302.148779 trunc from 0x027080 to 0x02b1c8 (0x4149 bytes) 000082 1775194302.170424 punch from 0x00b815 to 0x015237 (0x9a23 bytes) 000083 1775194302.175813 punch from 0x002de5 to 0x0030c9 (0x2e5 bytes) 000084 1775194302.180667 write 0x00f35b thru 0x012bfa (0x38a0 bytes) 000085 1775194302.193820 trunc from 0x019934 to 0x021f00 (0x85cd bytes) 000086 1775194302.214847 write 0x00f635 thru 0x01a354 (0xad20 bytes) 000087 1775194302.221015 read 0x025db3 thru 0x028c02 (0x2e50 bytes) 000088 1775194302.222590 fallocate 0x00f144 thru 0x015d03 (0x6bc0 bytes) 000089 1775194302.227385 mapread 0x017282 thru 0x021fdd (0xad5c bytes) 000090 1775194302.237772 trunc from 0x026c30 to 0x02b1c8 (0x4599 bytes) 000091 1775194302.243760 trunc from 0x029ae9 to 0x02b1c8 (0x16e0 bytes) 000092 1775194302.248018 trunc from 0x0205f1 to 0x026c82 (0x6692 bytes) 000093 1775194302.253991 punch from 0x01bece to 0x02173c (0x586f bytes) 000094 1775194302.259333 trunc from 0x02a2e7 to 0x02b1c8 (0xee2 bytes) 000096 1775194302.265101 read 0x0044ea thru 0x010ec7 (0xc9de bytes) 000097 1775194302.266035 trunc from 0x02b1c9 to 0x02464f (0x6b79 bytes) 000098 1775194302.285887 write 0x039a2c thru 0x039e9c (0x471 bytes) 000099 1775194302.287734 mapread 0x01f3ca thru 0x022a75 (0x36ac bytes) 000100 1775194302.299919 trunc from 0x039e9d to 0x036ae8 (0x33b4 bytes) 000102 1775194302.315353 mapwrite 0x012c7c thru 0x01f97d (0xcd02 bytes) 000103 1775194302.339734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36ae9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ae9): Mirror 2: 0x2f6a16f (stale) CRC-32 checksum value for chunk [0, 0x36ae9): Mirror 1: 0xb93d6534 Mirror 3: 0xb93d6534 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ae9: skipped 000104 1775194303.179397 read 0x027a45 thru 0x02d250 (0x580c bytes) 000105 1775194303.191110 punch from 0x01ad65 to 0x020006 (0x52a2 bytes) 000106 1775194303.250952 fallocate 0x0214fc thru 0x02c301 (0xae06 bytes) 000107 1775194303.259091 mapread 0x011d42 thru 0x01345c (0x171b bytes) 000108 1775194303.292878 mapwrite 0x02261b thru 0x026884 (0x426a bytes) 000109 1775194303.352131 mapread 0x02f428 thru 0x036ae8 (0x76c1 bytes) 000110 1775194303.376504 read 0x022788 thru 0x02f401 (0xcc7a bytes) 000111 1775194303.378098 trunc from 0x036ae9 to 0x02746f (0xf679 bytes) 000112 1775194303.470803 fallocate 0x00da71 thru 0x010dbf (0x334f bytes) 000113 1775194303.482992 mapread 0x01938a thru 0x025359 (0xbfd0 bytes) 000114 1775194303.489020 fallocate 0x022d45 thru 0x029223 (0x64df bytes) 000115 1775194303.503322 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27470): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27470): Mirror 2: 0x685c7152 (stale) Mirror 3: 0x2f7ee704 (stale) CRC-32 checksum value for chunk [0, 0x27470): Mirror 1: 0x914fc722 Mirror 4: 0x914fc722 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27470: skipped 000116 1775194304.431574 fallocate 0x0372e7 thru 0x03be12 (0x4b2c bytes) 000117 1775194304.497051 trunc from 0x027470 to 0x00c652 (0x1ae1d bytes) 000118 1775194304.590961 fallocate 0x03c0b5 thru 0x03cab3 (0x9ff bytes) 000119 1775194304.600119 read 0x00111e thru 0x00c652 (0xb535 bytes) 000120 1775194304.620598 fallocate 0x011725 thru 0x0203a6 (0xec82 bytes) 000121 1775194304.634581 fallocate 0x03aa15 thru 0x03e6c9 (0x3cb5 bytes) 000122 1775194304.643687 trunc from 0x002d5f to 0x00a3b4 (0x7656 bytes) 000123 1775194304.664866 fallocate 0x0164d8 thru 0x019624 (0x314d bytes) 000124 1775194304.669705 write 0x01ae47 thru 0x023f25 (0x90df bytes) 000125 1775194304.673000 fallocate 0x010851 thru 0x01ea96 (0xe246 bytes) 000126 1775194304.679909 punch from 0x0227ab to 0x023f25 (0x177b bytes) 000127 1775194304.701736 read 0x006b91 thru 0x015d05 (0xf175 bytes) 000128 1775194304.715916 fallocate 0x019cc1 thru 0x019d6e (0xae bytes) 000129 1775194304.727553 trunc from 0x023f26 to 0x0343d8 (0x104b3 bytes) 000130 1775194304.745704 fallocate 0x01099a thru 0x01d596 (0xcbfd bytes) 000131 1775194304.751813 fallocate 0x035276 thru 0x038fc5 (0x3d50 bytes) 000132 1775194304.760823 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 3: 0x76f09772 (stale) Mirror 4: 0xaf82e607 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 3: 0x76f09772 (stale) Mirror 4: 0xaf82e607 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fc6: skipped 000133 1775194305.227874 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 4: 0xaf82e607 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38fc6): Mirror 4: 0xaf82e607 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38fc6: skipped 000134 1775194305.498467 fallocate 0x02bf1b thru 0x030bdc (0x4cc2 bytes) 000135 1775194305.507387 read 0x007ee0 thru 0x012f6f (0xb090 bytes) 000136 1775194305.529240 write 0x02eb62 thru 0x03e801 (0xfca0 bytes) 000137 1775194305.543092 trunc from 0x019d0c to 0x0230af (0x93a4 bytes) 000138 1775194305.562754 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) 000139 1775194305.793248 mapread 0x00b357 thru 0x01623a (0xaee4 bytes) 000142 1775194305.839753 punch from 0x005da2 to 0x011a6c (0xbccb bytes) 000143 1775194305.846520 write 0x00e88d thru 0x013898 (0x500c bytes) 000144 1775194305.862938 trunc from 0x026cd6 to 0x032bf9 (0xbf24 bytes) 000145 1775194305.867820 mapread 0x025a17 thru 0x028f46 (0x3530 bytes) 000147 1775194305.887431 mapread 0x005944 thru 0x00f3d7 (0x9a94 bytes) 000149 1775194305.902058 read 0x039bbf thru 0x03e801 (0x4c43 bytes) 000150 1775194305.929334 read 0x01e5dc thru 0x029bfc (0xb621 bytes) 000151 1775194305.930970 fallocate 0x03c140 thru 0x03ffff (0x3ec0 bytes) 000152 1775194305.939648 fallocate 0x001266 thru 0x0037de (0x2579 bytes) 000153 1775194305.945079 trunc from 0x02550f to 0x03180f (0xc301 bytes) 000154 1775194305.959311 write 0x01f962 thru 0x021571 (0x1c10 bytes) 000155 1775194305.960387 fallocate 0x0210aa thru 0x0226e6 (0x163d bytes) 000156 1775194305.965793 write 0x035d4f thru 0x03ffff (0xa2b1 bytes) 000157 1775194305.975540 read 0x00f83a thru 0x0122ab (0x2a72 bytes) 000158 1775194305.976199 mapread 0x02396b thru 0x02b271 (0x7907 bytes) 000159 1775194305.977282 mapread 0x031a06 thru 0x033ef5 (0x24f0 bytes) 000160 1775194305.982032 mapwrite 0x031917 thru 0x033761 (0x1e4b bytes) 000161 1775194306.020827 write 0x011a77 thru 0x01bf27 (0xa4b1 bytes) 000162 1775194306.023362 read 0x018c15 thru 0x025008 (0xc3f4 bytes) 000163 1775194306.026662 read 0x024178 thru 0x0267f0 (0x2679 bytes) 000164 1775194306.027815 trunc from 0x02166c to 0x02cc34 (0xb5c9 bytes) 000165 1775194306.033758 read 0x020314 thru 0x02ec72 (0xe95f bytes) 000166 1775194306.043966 mapwrite 0x010578 thru 0x013250 (0x2cd9 bytes) 000167 1775194306.063920 read 0x011323 thru 0x01782b (0x6509 bytes) 000169 1775194306.065548 fallocate 0x013708 thru 0x01b8ab (0x81a4 bytes) 000170 1775194306.071046 write 0x02b8c2 thru 0x03b356 (0xfa95 bytes) 000171 1775194306.075310 punch from 0x039d44 to 0x03ffff (0x62bc bytes) 000172 1775194306.097904 fallocate 0x034794 thru 0x03794e (0x31bb bytes) 000173 1775194306.103592 read 0x02a9b9 thru 0x02e0e8 (0x3730 bytes) 000174 1775194306.104354 fallocate 0x001901 thru 0x00ee86 (0xd586 bytes) 000175 1775194306.109320 punch from 0x00462b to 0x00c6b1 (0x8087 bytes) 000177 1775194306.115341 trunc from 0x02d86d to 0x03a6f8 (0xce8c bytes) 000178 1775194306.120003 fallocate 0x02c4d5 thru 0x0345cb (0x80f7 bytes) 000179 1775194306.126532 write 0x036f92 thru 0x037f18 (0xf87 bytes) 000180 1775194306.148185 trunc from 0x040000 to 0x03b6ef (0x4910 bytes) 000181 1775194306.172076 fallocate 0x018f5d thru 0x0218f6 (0x899a bytes) 000182 1775194306.183798 mapwrite 0x00e90f thru 0x01b261 (0xc953 bytes) 000184 1775194306.220747 trunc from 0x01b53a to 0x01c4d2 (0xf99 bytes) 000185 1775194306.235083 fallocate 0x02cd62 thru 0x032dd9 (0x6078 bytes) 000186 1775194306.240541 read 0x005ce5 thru 0x010001 (0xa31d bytes) 000187 1775194306.242306 trunc from 0x03b6f0 to 0x02aa30 (0x10cbf bytes) 000188 1775194306.258876 trunc from 0x02aa31 to 0x036d7e (0xc34e bytes) 000189 1775194306.275590 mapwrite 0x013a3f thru 0x01a5cb (0x6b8d bytes) 000190 1775194306.299606 read 0x02d2fc thru 0x030446 (0x314b bytes) 000191 1775194306.308810 fallocate 0x03a64a thru 0x03e0ef (0x3aa6 bytes) 000192 1775194306.315060 mapwrite 0x006214 thru 0x00fbd2 (0x99bf bytes) 000193 1775194306.351373 read 0x029445 thru 0x029664 (0x220 bytes) 000194 1775194306.352058 mapread 0x026de0 thru 0x032059 (0xb27a bytes) 000195 1775194306.353266 trunc from 0x03e0f0 to 0x00959a (0x34b55 bytes) 000196 1775194306.379670 punch from 0x002f98 to 0x00959a (0x6603 bytes) 000197 1775194306.390121 fallocate 0x0141cd thru 0x01ca83 (0x88b7 bytes) 000198 1775194306.396928 write 0x03e4ec thru 0x03ffff (0x1b14 bytes) 000199 1775194306.401928 trunc from 0x040000 to 0x03dcba (0x2345 bytes) 000200 1775194306.427187 fallocate 0x03c65b thru 0x03ffff (0x39a5 bytes) 000201 1775194306.434063 mapwrite 0x03e779 thru 0x03ffff (0x1887 bytes) 000202 1775194306.510294 fallocate 0x01b972 thru 0x01f35e (0x39ed bytes) 000203 1775194306.514553 trunc from 0x00d71d to 0x012d41 (0x5625 bytes) 000204 1775194306.518945 trunc from 0x040000 to 0x035813 (0xa7ec bytes) 000205 1775194306.537921 trunc from 0x035814 to 0x02407a (0x11799 bytes) 000207 1775194306.555351 read 0x00877e thru 0x0090cb (0x94e bytes) 000208 1775194306.560593 MIRROR_EXTEND to 2 mirrors 000210 1775194307.261941 write 0x02eac0 thru 0x03b110 (0xc651 bytes) 000211 1775194307.329628 read 0x036551 thru 0x03b110 (0x4bc0 bytes) 000212 1775194307.330355 read 0x01996c thru 0x020a91 (0x7126 bytes) 000213 1775194307.363852 MIRROR_RESYNC 2 mirrors 000214 1775194307.992803 trunc from 0x03b111 to 0x00fd73 (0x2b39d bytes) 000215 1775194308.072593 MIRROR_RESYNC 2 mirrors 000216 1775194308.639856 read 0x004f5e thru 0x0070cb (0x216e bytes) 000217 1775194308.646954 fallocate 0x012744 thru 0x020602 (0xdebf bytes) 000218 1775194308.688600 trunc from 0x006e2e to 0x00b157 (0x432a bytes) 000219 1775194308.706418 MIRROR_RESYNC 2 mirrors 000220 1775194309.450811 mapwrite 0x018164 thru 0x01bc4a (0x3ae7 bytes) 000222 1775194309.527852 mapread 0x014784 thru 0x01bc4a (0x74c7 bytes) 000223 1775194309.529740 read 0x00112e thru 0x002cca (0x1b9d bytes) 000224 1775194309.536624 read 0x00e487 thru 0x00ea80 (0x5fa bytes) 000226 1775194309.548566 MIRROR_RESYNC 2 mirrors 000227 1775194310.191611 fallocate 0x02dcc9 thru 0x039ed9 (0xc211 bytes) 000229 1775194310.237440 trunc from 0x01bc4b to 0x036d26 (0x1b0dc bytes) 000230 1775194310.320721 trunc from 0x02c485 to 0x02db6e (0x16ea bytes) 000231 1775194310.339137 mapwrite 0x01d70a thru 0x01f9a2 (0x2299 bytes) 000232 1775194310.382991 trunc from 0x011801 to 0x013a31 (0x2231 bytes) 000233 1775194310.388259 write 0x00e9be thru 0x017746 (0x8d89 bytes) 000234 1775194310.410673 punch from 0x01c5ee to 0x02408d (0x7aa0 bytes) 000235 1775194310.417674 write 0x0122ae thru 0x02053b (0xe28e bytes) 000236 1775194310.426607 mapwrite 0x024646 thru 0x02c2bc (0x7c77 bytes) 000237 1775194310.464729 trunc from 0x036d27 to 0x03d895 (0x6b6f bytes) 000238 1775194310.477672 write 0x01f576 thru 0x02b214 (0xbc9f bytes) 000239 1775194310.479090 trunc from 0x03d896 to 0x034f4b (0x894a bytes) 000240 1775194310.499704 mapwrite 0x0205cc thru 0x02e531 (0xdf66 bytes) 000241 1775194310.530413 write 0x016643 thru 0x021cfd (0xb6bb bytes) 000242 1775194310.532528 trunc from 0x034f4c to 0x01ec7b (0x162d0 bytes) 000243 1775194310.554200 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) 000244 1775194310.723300 trunc from 0x012d92 to 0x01674b (0x39ba bytes) 000245 1775194310.739911 mapwrite 0x015930 thru 0x017aba (0x218b bytes) 000246 1775194310.771883 write 0x00b97a thru 0x01ad32 (0xf3b9 bytes) 000247 1775194310.793026 mapread 0x00faba thru 0x017b6f (0x80b6 bytes) 000248 1775194310.794803 mapread 0x01cc03 thru 0x01e78d (0x1b8b bytes) 000249 1775194310.797515 mapwrite 0x018a5c thru 0x021968 (0x8f0d bytes) 000250 1775194310.852640 fallocate 0x034fe4 thru 0x03b53d (0x655a bytes) 000251 1775194310.858037 read 0x00e83a thru 0x015c82 (0x7449 bytes) 000252 1775194310.859438 mapread 0x00fd7c thru 0x014624 (0x48a9 bytes) 000253 1775194310.861309 trunc from 0x02a955 to 0x02e848 (0x3ef4 bytes) 000254 1775194310.866938 trunc from 0x03b53e to 0x00ac9e (0x3089f bytes) 000255 1775194310.880979 write 0x0247d6 thru 0x031efa (0xd725 bytes) 000257 1775194310.884234 punch from 0x0130b0 to 0x0132e2 (0x233 bytes) 000258 1775194310.889673 mapwrite 0x00db76 thru 0x017bf9 (0xa084 bytes) 000259 1775194310.927777 trunc from 0x031efb to 0x02ae46 (0x70b4 bytes) 000260 1775194310.944333 fallocate 0x02542b thru 0x02c9eb (0x75c1 bytes) 000261 1775194310.950326 write 0x018ebe thru 0x0276af (0xe7f2 bytes) 000262 1775194310.952629 fallocate 0x00cd4b thru 0x018021 (0xb2d7 bytes) 000264 1775194310.960002 mapwrite 0x02d7ec thru 0x0331cf (0x59e4 bytes) 000265 1775194311.014921 trunc from 0x01eb7d to 0x029b91 (0xb015 bytes) 000266 1775194311.032647 MIRROR_EXTEND to 2 mirrors 000267 1775194311.735583 fallocate 0x01eb32 thru 0x023f46 (0x5415 bytes) 000268 1775194311.764560 trunc from 0x0331d0 to 0x014aed (0x1e6e2 bytes) 000269 1775194311.791803 punch from 0x010368 to 0x010e57 (0xaf0 bytes) 000271 1775194311.797980 trunc from 0x014aee to 0x02489e (0xfdb1 bytes) 000272 1775194311.813496 mapwrite 0x02cad6 thru 0x02f211 (0x273c bytes) 000273 1775194311.862023 fallocate 0x02fd3e thru 0x03bc8b (0xbf4e bytes) 000274 1775194311.879093 mapwrite 0x022f41 thru 0x02acfb (0x7dbb bytes) 000275 1775194311.914506 mapwrite 0x02ebd6 thru 0x030135 (0x1560 bytes) 000276 1775194311.946655 write 0x033313 thru 0x03fd8f (0xca7d bytes) 000277 1775194311.948313 trunc from 0x0015e6 to 0x0060d4 (0x4aef bytes) 000278 1775194311.963902 read 0x03b1d2 thru 0x03fd8f (0x4bbe bytes) 000279 1775194311.964573 trunc from 0x03fd90 to 0x03ad79 (0x5016 bytes) 000280 1775194311.982003 punch from 0x005b45 to 0x00a373 (0x482f bytes) 000281 1775194311.986018 trunc from 0x031455 to 0x036d44 (0x58f0 bytes) 000282 1775194312.005064 write 0x00fe7e thru 0x0120c6 (0x2249 bytes) 000283 1775194312.018906 punch from 0x01fce5 to 0x026f3a (0x7256 bytes) 000284 1775194312.024602 trunc from 0x03ad7a to 0x01aafd (0x2027c bytes) 000285 1775194312.042724 fallocate 0x02061c thru 0x029cca (0x96af bytes) 000286 1775194312.048047 mapread 0x01a7b4 thru 0x01aafd (0x34a bytes) 000287 1775194312.054289 write 0x023929 thru 0x024b64 (0x123c bytes) 000288 1775194312.059798 MIRROR_RESYNC 2 mirrors 000290 1775194312.524964 fallocate 0x0038cf thru 0x01284f (0xef81 bytes) 000291 1775194312.568893 fallocate 0x028068 thru 0x02e9bd (0x6956 bytes) 000292 1775194312.574787 read 0x027687 thru 0x02da0e (0x6388 bytes) 000293 1775194312.585031 mapwrite 0x015e07 thru 0x0231dc (0xd3d6 bytes) 000295 1775194312.655943 write 0x032fa0 thru 0x03f4f7 (0xc558 bytes) 000296 1775194312.659973 trunc from 0x00717e to 0x0127d3 (0xb656 bytes) 000298 1775194312.668728 mapread 0x021b30 thru 0x02ff93 (0xe464 bytes) 000299 1775194312.682926 write 0x002d6d thru 0x00a415 (0x76a9 bytes) 000301 1775194312.698852 fallocate 0x017991 thru 0x021a4d (0xa0bd bytes) 000302 1775194312.714009 MIRROR_RESYNC 2 mirrors 000303 1775194313.539995 MIRROR_EXTEND to 3 mirrors 000304 1775194314.298660 mapread 0x01100b thru 0x019762 (0x8758 bytes) 000305 1775194314.318182 fallocate 0x00f8a7 thru 0x018d67 (0x94c1 bytes) 000306 1775194314.368896 trunc from 0x03f4f8 to 0x00931c (0x361db bytes) 000307 1775194314.406223 punch from 0x002027 to 0x00931c (0x72f6 bytes) 000308 1775194314.413585 punch from 0x008d9b to 0x00931c (0x582 bytes) 000309 1775194314.422798 read 0x006e6f thru 0x00931c (0x24ae bytes) 000311 1775194314.436086 trunc from 0x007b1d to 0x00931c (0x1800 bytes) 000312 1775194314.445434 read 0x00550d thru 0x00931c (0x3e10 bytes) 000313 1775194314.458019 punch from 0x0015d8 to 0x00931c (0x7d45 bytes) 000314 1775194314.473446 trunc from 0x00931d to 0x015cc1 (0xc9a5 bytes) 000315 1775194314.490997 mapread 0x004d82 thru 0x00f9d4 (0xac53 bytes) 000316 1775194314.533103 mapwrite 0x033a0e thru 0x036493 (0x2a86 bytes) 000318 1775194314.603085 read 0x01dd6f thru 0x022252 (0x44e4 bytes) 000319 1775194314.605198 mapread 0x004ed4 thru 0x00df72 (0x909f bytes) 000321 1775194314.606931 mapwrite 0x01aa0e thru 0x01eae7 (0x40da bytes) 000322 1775194314.639691 punch from 0x00724f to 0x00d74f (0x6501 bytes) 000323 1775194314.656717 trunc from 0x036494 to 0x013958 (0x22b3b bytes) 000324 1775194314.681377 mapread 0x003cbf thru 0x006b52 (0x2e94 bytes) 000325 1775194314.700460 write 0x01a7e8 thru 0x028455 (0xdc6e bytes) 000326 1775194314.704042 mapwrite 0x00f6ed thru 0x015905 (0x6219 bytes) 000327 1775194314.761191 trunc from 0x026f25 to 0x028455 (0x1531 bytes) 000328 1775194314.767081 trunc from 0x00a146 to 0x00b3ee (0x12a9 bytes) 000330 1775194314.777191 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28456): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28456): Mirror 2: 0x4e3b3577 (stale) Mirror 3: 0x4e3b3577 (stale) CRC-32 checksum value for chunk [0, 0x28456): Mirror 1: 0x64268e43 Mirror 4: 0x64268e43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28456: skipped 000331 1775194315.810937 read 0x00b824 thru 0x00c383 (0xb60 bytes) 000332 1775194315.822843 trunc from 0x028456 to 0x01a018 (0xe43d bytes) 000333 1775194315.868791 write 0x00f724 thru 0x012a82 (0x335f bytes) 000334 1775194315.878710 trunc from 0x01a019 to 0x0278c7 (0xd8af bytes) 000335 1775194315.905537 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x278c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278c8): Mirror 2: 0xfefd99d2 (stale) Mirror 3: 0xfefd99d2 (stale) Mirror 4: 0xca7862ac (stale) CRC-32 checksum value for chunk [0, 0x278c8): Mirror 1: 0xaf67b2f8 Mirror 5: 0xaf67b2f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278c8: skipped 000336 1775194316.849959 trunc from 0x009b6b to 0x01974b (0xfbe1 bytes) 000337 1775194316.897955 fallocate 0x02cd42 thru 0x03cb4d (0xfe0c bytes) 000338 1775194316.910951 mapwrite 0x036e23 thru 0x037c17 (0xdf5 bytes) 000339 1775194317.002532 punch from 0x007df4 to 0x00b894 (0x3aa1 bytes) 000340 1775194317.016113 trunc from 0x037c18 to 0x0187b7 (0x1f460 bytes) 000341 1775194317.042569 trunc from 0x004751 to 0x005f2a (0x17da bytes) 000342 1775194317.047720 trunc from 0x0085ea to 0x00fdcf (0x77e6 bytes) 000343 1775194317.052997 trunc from 0x00a140 to 0x014bf6 (0xaab7 bytes) 000344 1775194317.060592 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x187b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x187b8): Mirror 2: 0x7a906f76 (stale) Mirror 3: 0x7a906f76 (stale) Mirror 4: 0x6c69124c (stale) Mirror 5: 0xbde31c3d (stale) CRC-32 checksum value for chunk [0, 0x187b8): Mirror 1: 0xceb25df4 Mirror 6: 0xceb25df4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x187b8: skipped 000346 1775194318.070519 mapwrite 0x00106e thru 0x00639e (0x5331 bytes) 000347 1775194318.229631 mapread 0x005529 thru 0x00b6ea (0x61c2 bytes) 000348 1775194318.234975 mapwrite 0x031c28 thru 0x037b62 (0x5f3b bytes) 000349 1775194318.296992 trunc from 0x037b63 to 0x03c1f0 (0x468e bytes) 000350 1775194318.321658 mapwrite 0x01f6da thru 0x028e08 (0x972f bytes) 000351 1775194318.375773 read 0x028977 thru 0x02f712 (0x6d9c bytes) 000352 1775194318.377573 mapread 0x01971b thru 0x023444 (0x9d2a bytes) 000353 1775194318.379570 read 0x0322f6 thru 0x033767 (0x1472 bytes) 000354 1775194318.380659 write 0x0131ba thru 0x01e76c (0xb5b3 bytes) 000355 1775194318.404624 trunc from 0x00c04f to 0x00fbf4 (0x3ba6 bytes) 000356 1775194318.414909 write 0x002338 thru 0x00acc6 (0x898f bytes) 000358 1775194318.418079 fallocate 0x017506 thru 0x023a71 (0xc56c bytes) 000359 1775194318.426372 mapread 0x023d39 thru 0x03164c (0xd914 bytes) 000360 1775194318.429265 write 0x0322ae thru 0x034aa5 (0x27f8 bytes) 000361 1775194318.431434 mapwrite 0x023e13 thru 0x031882 (0xda70 bytes) 000362 1775194318.474991 read 0x016bcc thru 0x022fda (0xc40f bytes) 000363 1775194318.476726 trunc from 0x03c1f1 to 0x02756c (0x14c84 bytes) 000364 1775194318.500991 trunc from 0x02756d to 0x02027b (0x72f1 bytes) 000365 1775194318.552842 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2027c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2027c): Mirror 2: 0x1d980ea2 (stale) Mirror 3: 0x1d980ea2 (stale) Mirror 4: 0xe530ca21 (stale) Mirror 5: 0xcb7b8527 (stale) Mirror 6: 0x1818b168 (stale) CRC-32 checksum value for chunk [0, 0x2027c): Mirror 1: 0xc20eb2ae Mirror 7: 0xc20eb2ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2027c: skipped 000366 1775194319.768791 write 0x03f4a0 thru 0x03ffff (0xb60 bytes) 000367 1775194319.839606 punch from 0x02eee4 to 0x035eb3 (0x6fd0 bytes) 000369 1775194319.851457 MIRROR_RESYNC 7 mirrors 000370 1775194321.680440 mapwrite 0x001853 thru 0x00b4a5 (0x9c53 bytes) 000371 1775194321.868075 mapwrite 0x01471b thru 0x0170f1 (0x29d7 bytes) 000372 1775194321.928045 fallocate 0x02d3d5 thru 0x030374 (0x2fa0 bytes) 000373 1775194321.936486 trunc from 0x011ed4 to 0x01c82e (0xa95b bytes) 000374 1775194321.943138 read 0x02009d thru 0x027613 (0x7577 bytes) 000375 1775194321.954730 trunc from 0x040000 to 0x021d91 (0x1e26e bytes) 000376 1775194322.004769 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x21d92): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d92): Mirror 2: 0x302a6398 (stale) Mirror 3: 0x302a6398 (stale) Mirror 4: 0x302a6398 (stale) Mirror 5: 0x302a6398 (stale) Mirror 6: 0x302a6398 (stale) Mirror 7: 0x302a6398 (stale) CRC-32 checksum value for chunk [0, 0x21d92): Mirror 1: 0x1bb1794d Mirror 8: 0x1bb1794d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d92: skipped 000377 1775194322.978961 fallocate 0x03262b thru 0x035924 (0x32fa bytes) 000378 1775194323.026185 punch from 0x00775a to 0x0089f3 (0x129a bytes) 000379 1775194323.031606 mapread 0x01c93a thru 0x021d91 (0x5458 bytes) 000380 1775194323.058895 mapwrite 0x039353 thru 0x03e22d (0x4edb bytes) 000381 1775194323.116593 read 0x0250fe thru 0x029910 (0x4813 bytes) 000382 1775194323.124549 read 0x00cc0a thru 0x00f595 (0x298c bytes) 000383 1775194323.133938 mapwrite 0x003694 thru 0x007634 (0x3fa1 bytes) 000384 1775194323.155645 fallocate 0x030c2a thru 0x03d09a (0xc471 bytes) 000387 1775194323.162043 mapwrite 0x02ce49 thru 0x039252 (0xc40a bytes) 000388 1775194323.182230 write 0x02aa1e thru 0x03745b (0xca3e bytes) 000390 1775194323.184755 fallocate 0x00f28e thru 0x01db6f (0xe8e2 bytes) 000391 1775194323.189700 punch from 0x0030cc to 0x010ec6 (0xddfb bytes) 000393 1775194323.195766 fallocate 0x01021d thru 0x019248 (0x902c bytes) 000394 1775194323.200342 mapread 0x002889 thru 0x00bc3a (0x93b2 bytes) 000395 1775194323.241679 trunc from 0x03e22e to 0x009339 (0x34ef4 bytes) 000396 1775194323.258663 write 0x027ae2 thru 0x02dfd5 (0x64f4 bytes) 000397 1775194323.261606 mapread 0x02becb thru 0x02d230 (0x1366 bytes) 000398 1775194323.267376 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dfd6): Mirror 2: 0x45f95f1e (stale) Mirror 3: 0x45f95f1e (stale) Mirror 4: 0x45f95f1e (stale) Mirror 5: 0x45f95f1e (stale) Mirror 6: 0x45f95f1e (stale) Mirror 8: 0x645114b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfd6): Mirror 2: 0x45f95f1e (stale) Mirror 3: 0x45f95f1e (stale) Mirror 4: 0x45f95f1e (stale) Mirror 5: 0x45f95f1e (stale) Mirror 6: 0x45f95f1e (stale) Mirror 8: 0x645114b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfd6: skipped 000399 1775194323.644335 punch from 0x02498b to 0x026ba7 (0x221d bytes) 000400 1775194323.649095 fallocate 0x03a307 thru 0x03ffff (0x5cf9 bytes) 000401 1775194323.653917 mapwrite 0x00dc7e thru 0x01d0b1 (0xf434 bytes) 000402 1775194323.703818 read 0x00ea7d thru 0x015dee (0x7372 bytes) 000403 1775194323.704781 mapread 0x01e65d thru 0x02d813 (0xf1b7 bytes) 000404 1775194323.721646 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dfd6): Mirror 2: 0x45f95f1e (stale) Mirror 3: 0x45f95f1e (stale) Mirror 4: 0x45f95f1e (stale) Mirror 5: 0x45f95f1e (stale) Mirror 8: 0x645114b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfd6): Mirror 2: 0x45f95f1e (stale) Mirror 3: 0x45f95f1e (stale) Mirror 4: 0x45f95f1e (stale) Mirror 5: 0x45f95f1e (stale) Mirror 8: 0x645114b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfd6: skipped 000405 1775194324.206695 trunc from 0x02dfd6 to 0x01d82e (0x107a7 bytes) 000406 1775194324.260521 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d82f): Mirror 2: 0x4853bc6b (stale) Mirror 3: 0x4853bc6b (stale) Mirror 4: 0x4853bc6b (stale) Mirror 5: 0x4853bc6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d82f): Mirror 2: 0x4853bc6b (stale) Mirror 3: 0x4853bc6b (stale) Mirror 4: 0x4853bc6b (stale) Mirror 5: 0x4853bc6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d82f: skipped 000407 1775194324.704954 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d82f): Mirror 3: 0x4853bc6b (stale) Mirror 4: 0x4853bc6b (stale) Mirror 5: 0x4853bc6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d82f): Mirror 3: 0x4853bc6b (stale) Mirror 4: 0x4853bc6b (stale) Mirror 5: 0x4853bc6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d82f: skipped 000408 1775194325.081016 write 0x023b34 thru 0x023daf (0x27c bytes) 000409 1775194325.094377 punch from 0x022fd9 to 0x023daf (0xdd7 bytes) 000410 1775194325.104530 fallocate 0x02c89c thru 0x038af1 (0xc256 bytes) 000412 1775194325.109880 trunc from 0x023db0 to 0x02bc22 (0x7e73 bytes) 000413 1775194325.142662 read 0x0256c5 thru 0x02bc22 (0x655e bytes) 000414 1775194325.156028 trunc from 0x02bc23 to 0x011d09 (0x19f19 bytes) 000415 1775194325.183911 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 3: 0x96d1142 (stale) Mirror 4: 0x96d1142 (stale) Mirror 5: 0x96d1142 (stale) CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 1: 0x28ca93a1 Mirror 6: 0x28ca93a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d0a: skipped 000416 1775194325.905756 fallocate 0x02970a thru 0x036baf (0xd4a6 bytes) 000417 1775194325.965554 fallocate 0x001fdb thru 0x004029 (0x204f bytes) 000418 1775194325.970355 punch from 0x0095be to 0x011d09 (0x874c bytes) 000419 1775194325.979045 trunc from 0x008fc6 to 0x00ba1b (0x2a56 bytes) 000420 1775194325.989865 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 3: 0x96d1142 (stale) Mirror 4: 0x96d1142 (stale) Mirror 5: 0x96d1142 (stale) Mirror 6: 0x28ca93a1 (stale) CRC-32 checksum value for chunk [0, 0x11d0a): Mirror 1: 0x994a19d0 Mirror 7: 0x994a19d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d0a: skipped 000421 1775194327.017340 trunc from 0x011d0a to 0x033181 (0x21478 bytes) 000424 1775194327.077527 mapread 0x001ebb thru 0x0097b0 (0x78f6 bytes) 000425 1775194327.095556 read 0x00b9b1 thru 0x012d90 (0x73e0 bytes) 000426 1775194327.101346 punch from 0x028dea to 0x02c782 (0x3999 bytes) 000427 1775194327.105595 trunc from 0x0039c6 to 0x004bdf (0x121a bytes) 000428 1775194327.111403 trunc from 0x01d446 to 0x01dd1e (0x8d9 bytes) 000429 1775194327.115550 write 0x00db93 thru 0x01774f (0x9bbd bytes) 000430 1775194327.122499 punch from 0x01eb3f to 0x0253af (0x6871 bytes) 000431 1775194327.126666 mapwrite 0x0097bc thru 0x019292 (0xfad7 bytes) 000432 1775194327.159040 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33182): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33182): Mirror 3: 0xe854f777 (stale) Mirror 4: 0xe854f777 (stale) Mirror 5: 0xe854f777 (stale) Mirror 6: 0xc2391060 (stale) Mirror 7: 0xab0a108b (stale) CRC-32 checksum value for chunk [0, 0x33182): Mirror 1: 0x702c39a Mirror 8: 0x702c39a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33182: skipped 000434 1775194328.198178 punch from 0x001249 to 0x00614a (0x4f02 bytes) 000435 1775194328.249283 punch from 0x02a817 to 0x02cd6e (0x2558 bytes) 000436 1775194328.261107 punch from 0x00c012 to 0x00fad9 (0x3ac8 bytes) 000437 1775194328.271029 mapwrite 0x02d457 thru 0x038034 (0xabde bytes) 000438 1775194328.465963 write 0x034bc1 thru 0x03bb65 (0x6fa5 bytes) 000440 1775194328.471105 read 0x034905 thru 0x03bb65 (0x7261 bytes) 000441 1775194328.471801 write 0x021a6e thru 0x026654 (0x4be7 bytes) 000442 1775194328.497751 mapread 0x035650 thru 0x03bb65 (0x6516 bytes) 000443 1775194328.499432 punch from 0x02d95b to 0x03ada5 (0xd44b bytes) 000445 1775194328.535854 read 0x0124e8 thru 0x022238 (0xfd51 bytes) 000446 1775194328.572370 read 0x026db4 thru 0x026e55 (0xa2 bytes) 000447 1775194328.573213 fallocate 0x02d5e0 thru 0x035b47 (0x8568 bytes) 000448 1775194328.578660 read 0x01819d thru 0x019bae (0x1a12 bytes) 000449 1775194328.585649 MIRROR_RESYNC 7 mirrors 000450 1775194331.769887 punch from 0x01323d to 0x01fee4 (0xcca8 bytes) 000451 1775194331.814545 trunc from 0x03bb66 to 0x03f55e (0x39f9 bytes) 000452 1775194331.838546 read 0x0277e9 thru 0x028f38 (0x1750 bytes) 000453 1775194331.844865 trunc from 0x03f55f to 0x03568e (0x9ed0 bytes) 000454 1775194331.864616 trunc from 0x00ec0c to 0x01c849 (0xdc3e bytes) 000455 1775194331.874036 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3568f): Mirror 3: 0xce47d26 (stale) Mirror 4: 0xce47d26 (stale) Mirror 5: 0xce47d26 (stale) Mirror 7: 0xce47d26 (stale) Mirror 8: 0xce47d26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3568f): Mirror 3: 0xce47d26 (stale) Mirror 4: 0xce47d26 (stale) Mirror 5: 0xce47d26 (stale) Mirror 7: 0xce47d26 (stale) Mirror 8: 0xce47d26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3568f: skipped 000456 1775194332.308325 read 0x01e0d1 thru 0x0215dd (0x350d bytes) 000457 1775194332.319759 trunc from 0x005e51 to 0x010a9f (0xac4f bytes) 000458 1775194332.326740 mapwrite 0x03365a thru 0x03421c (0xbc3 bytes) 000459 1775194332.352706 mapwrite 0x020e42 thru 0x02c6ce (0xb88d bytes) 000461 1775194332.393563 write 0x008199 thru 0x00ddc7 (0x5c2f bytes) 000462 1775194332.406122 mapwrite 0x0099cf thru 0x014600 (0xac32 bytes) 000463 1775194332.442879 mapread 0x02d439 thru 0x03568e (0x8256 bytes) 000464 1775194332.456642 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3568f): Mirror 4: 0xce47d26 (stale) Mirror 5: 0xce47d26 (stale) Mirror 7: 0xce47d26 (stale) Mirror 8: 0xce47d26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3568f): Mirror 4: 0xce47d26 (stale) Mirror 5: 0xce47d26 (stale) Mirror 7: 0xce47d26 (stale) Mirror 8: 0xce47d26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3568f: skipped 000465 1775194332.834353 fallocate 0x01c96d thru 0x0219d2 (0x5066 bytes) 000466 1775194332.844908 trunc from 0x03568f to 0x014469 (0x21225 bytes) 000467 1775194332.881960 mapread 0x008504 thru 0x00bbed (0x36ea bytes) 000468 1775194332.923723 read 0x00bb3c thru 0x014469 (0x892e bytes) 000470 1775194332.931220 mapwrite 0x003f97 thru 0x008401 (0x446b bytes) 000471 1775194332.959686 trunc from 0x01446a to 0x004889 (0xfbe0 bytes) 000472 1775194332.996787 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x488a): Mirror 4: 0x999bcb6e (stale) Mirror 7: 0x999bcb6e (stale) Mirror 8: 0x999bcb6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x488a): Mirror 4: 0x999bcb6e (stale) Mirror 7: 0x999bcb6e (stale) Mirror 8: 0x999bcb6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x488a: skipped 000473 1775194333.406100 trunc from 0x003855 to 0x004889 (0x1035 bytes) 000474 1775194333.417045 trunc from 0x00488a to 0x02fd99 (0x2b510 bytes) 000475 1775194333.438637 read 0x024e85 thru 0x025f49 (0x10c5 bytes) 000477 1775194333.445644 write 0x033f9b thru 0x0385f3 (0x4659 bytes) 000478 1775194333.447811 punch from 0x036483 to 0x0385f3 (0x2171 bytes) 000479 1775194333.461206 mapread 0x03689f thru 0x0385f3 (0x1d55 bytes) 000480 1775194333.478975 mapread 0x01cb64 thru 0x026193 (0x9630 bytes) 000481 1775194333.497721 write 0x00944c thru 0x00b625 (0x21da bytes) 000482 1775194333.513019 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385f4): Mirror 4: 0x4e86c3cf (stale) Mirror 8: 0x4e86c3cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385f4): Mirror 4: 0x4e86c3cf (stale) Mirror 8: 0x4e86c3cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385f4: skipped 000483 1775194333.851873 punch from 0x02d58d to 0x034a27 (0x749b bytes) 000484 1775194333.858544 trunc from 0x005431 to 0x00e785 (0x9355 bytes) 000485 1775194333.864354 trunc from 0x0385f4 to 0x0070b7 (0x3153c bytes) 000486 1775194333.881811 trunc from 0x0070b8 to 0x00bd1a (0x4c63 bytes) 000487 1775194333.902460 mapwrite 0x02380a thru 0x025d1c (0x2513 bytes) 000488 1775194333.952597 read 0x0162d9 thru 0x01a5f1 (0x4319 bytes) 000489 1775194333.959915 fallocate 0x029192 thru 0x03039b (0x720a bytes) 000490 1775194333.970861 MIRROR_RESYNC 3 mirrors 000491 1775194334.801933 MIRROR_EXTEND to 4 mirrors 000492 1775194335.547430 read 0x020d21 thru 0x024a59 (0x3d39 bytes) 000493 1775194335.558534 punch from 0x023023 to 0x02440e (0x13ec bytes) 000494 1775194335.597770 read 0x006ffe thru 0x010aba (0x9abd bytes) 000495 1775194335.627989 mapread 0x025ae7 thru 0x025d1c (0x236 bytes) 000496 1775194335.633499 mapread 0x001efa thru 0x007679 (0x5780 bytes) 000497 1775194335.643706 write 0x03f1cb thru 0x03ffff (0xe35 bytes) 000498 1775194335.651830 punch from 0x03f248 to 0x03ffff (0xdb8 bytes) 000499 1775194335.669393 mapread 0x01cab5 thru 0x02243b (0x5987 bytes) 000500 1775194335.670489 trunc from 0x040000 to 0x01b60a (0x249f5 bytes) 000501 1775194335.699806 trunc from 0x01b60b to 0x015387 (0x6283 bytes) 000502 1775194335.726984 mapread 0x009903 thru 0x00c7eb (0x2ee9 bytes) 000503 1775194335.731773 punch from 0x012126 to 0x015387 (0x3262 bytes) 000505 1775194335.740610 write 0x03ca8a thru 0x03eda0 (0x2317 bytes) 000507 1775194335.744455 read 0x01d24b thru 0x020474 (0x322a bytes) 000508 1775194335.756084 trunc from 0x03eda1 to 0x014f99 (0x29e07 bytes) 000509 1775194335.780393 trunc from 0x0038e0 to 0x010008 (0xc729 bytes) 000510 1775194335.791026 fallocate 0x03210f thru 0x037909 (0x57fb bytes) 000511 1775194335.796953 punch from 0x00c11c to 0x00f602 (0x34e7 bytes) 000512 1775194335.806434 trunc from 0x00b810 to 0x00e505 (0x2cf6 bytes) 000513 1775194335.832773 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14f9a): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14f9a): Mirror 4: 0xc02de89 (stale) Mirror 8: 0xc02de89 (stale) Mirror 9: 0xc02de89 (stale) CRC-32 checksum value for chunk [0, 0x14f9a): Mirror 1: 0xc02de89 Mirror 10: 0xc02de89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14f9a: skipped 000514 1775194336.934847 fallocate 0x0054ad thru 0x007585 (0x20d9 bytes) 000515 1775194336.972210 fallocate 0x02c673 thru 0x032066 (0x59f4 bytes) 000516 1775194336.977644 write 0x011111 thru 0x01992c (0x881c bytes) 000517 1775194336.986488 read 0x01284f thru 0x01992c (0x70de bytes) 000518 1775194336.991016 fallocate 0x004af8 thru 0x006f5e (0x2467 bytes) 000519 1775194336.996057 write 0x012872 thru 0x0142a5 (0x1a34 bytes) 000520 1775194336.997500 mapwrite 0x03cd88 thru 0x03ffff (0x3278 bytes) 000521 1775194337.046866 punch from 0x03d6ba to 0x03ffff (0x2946 bytes) 000522 1775194337.051614 punch from 0x017474 to 0x01ea73 (0x7600 bytes) 000523 1775194337.058385 fallocate 0x0239b4 thru 0x0337d8 (0xfe25 bytes) 000525 1775194337.063340 trunc from 0x040000 to 0x00e827 (0x317d8 bytes) 000527 1775194337.081308 mapwrite 0x0150fd thru 0x023759 (0xe65d bytes) 000528 1775194337.142445 trunc from 0x02375a to 0x01b089 (0x86d0 bytes) 000529 1775194337.158805 write 0x01cdce thru 0x021611 (0x4844 bytes) 000530 1775194337.161523 read 0x00a48d thru 0x010aa6 (0x661a bytes) 000531 1775194337.169662 trunc from 0x021612 to 0x01fffa (0x1617 bytes) 000532 1775194337.189567 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 4: 0xef36ca4e (stale) Mirror 8: 0xef36ca4e (stale) Mirror 9: 0xef36ca4e (stale) Mirror 10: 0xef36ca4e (stale) CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 1: 0x78a18f2d Mirror 11: 0x78a18f2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fffb: skipped 000533 1775194338.102258 trunc from 0x01bf91 to 0x01fffa (0x406a bytes) 000534 1775194338.171557 trunc from 0x01c98a to 0x01fffa (0x3671 bytes) 000535 1775194338.185509 write 0x00037d thru 0x00d3cb (0xd04f bytes) 000536 1775194338.234523 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 4: 0xef36ca4e (stale) Mirror 8: 0xef36ca4e (stale) Mirror 9: 0xef36ca4e (stale) Mirror 10: 0xef36ca4e (stale) Mirror 11: 0x78a18f2d (stale) CRC-32 checksum value for chunk [0, 0x1fffb): Mirror 1: 0x9cfac4d0 Mirror 12: 0x9cfac4d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fffb: skipped 000537 1775194339.390670 mapread 0x017f12 thru 0x01fffa (0x80e9 bytes) 000538 1775194339.428537 read 0x01eca5 thru 0x01fffa (0x1356 bytes) 000539 1775194339.435747 mapwrite 0x013be9 thru 0x021e84 (0xe29c bytes) 000540 1775194339.534999 write 0x00211b thru 0x00847c (0x6362 bytes) 000541 1775194339.544990 punch from 0x01cea1 to 0x01fcdd (0x2e3d bytes) 000543 1775194339.550612 read 0x01cbda thru 0x01f2e3 (0x270a bytes) 000545 1775194339.558434 mapwrite 0x010c67 thru 0x0174b8 (0x6852 bytes) 000546 1775194339.577245 mapread 0x007deb thru 0x01191c (0x9b32 bytes) 000547 1775194339.578459 write 0x03aae7 thru 0x03ffff (0x5519 bytes) 000548 1775194339.580593 trunc from 0x040000 to 0x030103 (0xfefc bytes) 000549 1775194339.597677 mapread 0x01ecd6 thru 0x0232ae (0x45d9 bytes) 000550 1775194339.613614 write 0x012dfc thru 0x02145b (0xe660 bytes) 000551 1775194339.615718 mapread 0x02a3e9 thru 0x02c914 (0x252c bytes) 000552 1775194339.616920 mapread 0x018008 thru 0x026bae (0xeba7 bytes) 000554 1775194339.619217 read 0x01dc9d thru 0x02181e (0x3b82 bytes) 000555 1775194339.620524 write 0x013a2e thru 0x0164e3 (0x2ab6 bytes) 000556 1775194339.622931 trunc from 0x030104 to 0x036cdc (0x6bd9 bytes) 000557 1775194339.637100 trunc from 0x02ad59 to 0x0313c1 (0x6669 bytes) 000558 1775194339.649128 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36cdd): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cdd): Mirror 4: 0xb42882d4 (stale) Mirror 8: 0xb42882d4 (stale) Mirror 9: 0xb42882d4 (stale) Mirror 10: 0x6302a2d5 (stale) Mirror 11: 0xfa142b25 (stale) Mirror 12: 0x75fb45d2 (stale) CRC-32 checksum value for chunk [0, 0x36cdd): Mirror 1: 0xedc33f83 Mirror 13: 0xedc33f83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cdd: skipped 000559 1775194340.765583 punch from 0x031a60 to 0x0347d5 (0x2d76 bytes) 000560 1775194340.845824 mapread 0x01b1f2 thru 0x01f7f4 (0x4603 bytes) 000561 1775194340.869748 mapwrite 0x033c41 thru 0x039b4b (0x5f0b bytes) 000562 1775194340.914304 mapwrite 0x028951 thru 0x02fed7 (0x7587 bytes) 000563 1775194340.937833 trunc from 0x039b4c to 0x008741 (0x3140a bytes) 000564 1775194340.951725 read 0x003da5 thru 0x005f6d (0x21c9 bytes) 000565 1775194340.957580 trunc from 0x008742 to 0x0049b8 (0x3d89 bytes) 000566 1775194340.980468 trunc from 0x0049b9 to 0x02f5b2 (0x2abfa bytes) 000567 1775194340.995481 write 0x01e4bd thru 0x028826 (0xa36a bytes) 000568 1775194341.015747 trunc from 0x02f5b3 to 0x029a58 (0x5b5a bytes) 000569 1775194341.035598 write 0x01d223 thru 0x028378 (0xb156 bytes) 000570 1775194341.037069 write 0x039fbb thru 0x03ffff (0x6045 bytes) 000571 1775194341.040002 punch from 0x009fb3 to 0x010311 (0x635f bytes) 000572 1775194341.043960 write 0x011186 thru 0x01bb12 (0xa98d bytes) 000573 1775194341.053015 mapread 0x019462 thru 0x0285ad (0xf14c bytes) 000574 1775194341.055181 trunc from 0x040000 to 0x0092f6 (0x36d09 bytes) 000575 1775194341.090333 mapwrite 0x0372c8 thru 0x03a531 (0x326a bytes) 000576 1775194341.142937 fallocate 0x02436e thru 0x0309cd (0xc660 bytes) 000578 1775194341.149195 mapread 0x00d7e0 thru 0x014052 (0x6873 bytes) 000579 1775194341.159704 write 0x0139a2 thru 0x015df9 (0x2458 bytes) 000580 1775194341.160962 punch from 0x02c601 to 0x03007a (0x3a7a bytes) 000581 1775194341.165116 read 0x02aa5c thru 0x038e62 (0xe407 bytes) 000582 1775194341.211287 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a532): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a532): Mirror 4: 0xda2acfa4 (stale) Mirror 8: 0xda2acfa4 (stale) Mirror 9: 0xda2acfa4 (stale) Mirror 10: 0xef40e140 (stale) Mirror 11: 0x71a929d6 (stale) Mirror 12: 0x2dfb70ce (stale) Mirror 13: 0xa901c88d (stale) CRC-32 checksum value for chunk [0, 0x3a532): Mirror 1: 0x7a70faa9 Mirror 14: 0x7a70faa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a532: skipped 000583 1775194342.249630 trunc from 0x03a532 to 0x01d6a4 (0x1ce8d bytes) 000584 1775194342.319689 trunc from 0x01d6a5 to 0x01c8be (0xde6 bytes) 000585 1775194342.344404 punch from 0x00e55d to 0x01583b (0x72df bytes) 000586 1775194342.349580 trunc from 0x01c8bf to 0x01ef57 (0x2699 bytes) 000587 1775194342.364175 mapwrite 0x01ef77 thru 0x020e46 (0x1ed0 bytes) 000588 1775194342.416467 trunc from 0x020e47 to 0x0105c0 (0x10886 bytes) 000589 1775194342.435134 write 0x0302d7 thru 0x03411b (0x3e45 bytes) 000590 1775194342.436362 mapread 0x00edaa thru 0x01c7ef (0xda46 bytes) 000591 1775194342.454568 trunc from 0x03411c to 0x007c4a (0x2c4d1 bytes) 000592 1775194342.491807 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7c4b): Mirror 4: 0xce3ba8b9 (stale) Mirror 8: 0xce3ba8b9 (stale) Mirror 9: 0xce3ba8b9 (stale) Mirror 10: 0xce3ba8b9 (stale) Mirror 11: 0xce3ba8b9 (stale) Mirror 12: 0x1e252d47 (stale) Mirror 14: 0xb3deb5f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7c4b): Mirror 4: 0xce3ba8b9 (stale) Mirror 8: 0xce3ba8b9 (stale) Mirror 9: 0xce3ba8b9 (stale) Mirror 10: 0xce3ba8b9 (stale) Mirror 11: 0xce3ba8b9 (stale) Mirror 12: 0x1e252d47 (stale) Mirror 14: 0xb3deb5f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7c4b: skipped 000593 1775194343.058529 write 0x002a6b thru 0x003fa0 (0x1536 bytes) 000594 1775194343.080531 trunc from 0x007c4b to 0x0327e9 (0x2ab9f bytes) 000595 1775194343.106661 read 0x024086 thru 0x030264 (0xc1df bytes) 000597 1775194343.128140 read 0x02ae2e thru 0x0327e9 (0x79bc bytes) 000598 1775194343.147366 trunc from 0x0327ea to 0x01ef9c (0x1384d bytes) 000599 1775194343.182728 trunc from 0x004e49 to 0x00a62e (0x57e6 bytes) 000601 1775194343.200932 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ef9d): Mirror 4: 0xa96cadb5 (stale) Mirror 8: 0xa96cadb5 (stale) Mirror 9: 0xa96cadb5 (stale) Mirror 10: 0xa96cadb5 (stale) Mirror 11: 0x7c20ed23 (stale) Mirror 12: 0x1a67c38c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef9d): Mirror 4: 0xa96cadb5 (stale) Mirror 8: 0xa96cadb5 (stale) Mirror 9: 0xa96cadb5 (stale) Mirror 10: 0xa96cadb5 (stale) Mirror 11: 0x7c20ed23 (stale) Mirror 12: 0x1a67c38c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef9d: skipped 000602 1775194343.613706 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1ef9d): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef9d): Mirror 4: 0xa96cadb5 (stale) Mirror 8: 0xa96cadb5 (stale) Mirror 9: 0xa96cadb5 (stale) Mirror 10: 0xa96cadb5 (stale) Mirror 11: 0x7c20ed23 (stale) Mirror 12: 0x1a67c38c (stale) CRC-32 checksum value for chunk [0, 0x1ef9d): Mirror 1: 0x77a60695 Mirror 13: 0x77a60695 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef9d: skipped 000603 1775194344.425388 MIRROR_RESYNC 8 mirrors 000604 1775194345.639109 read 0x001a89 thru 0x005638 (0x3bb0 bytes) 000605 1775194345.656277 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) 000606 1775194346.141673 mapwrite 0x001b6a thru 0x00a2ac (0x8743 bytes) 000607 1775194346.347668 fallocate 0x01e591 thru 0x025f76 (0x79e6 bytes) 000609 1775194346.359047 mapread 0x01d595 thru 0x01ef9c (0x1a08 bytes) 000610 1775194346.377715 mapread 0x007215 thru 0x00771a (0x506 bytes) 000612 1775194346.381444 mapwrite 0x03a045 thru 0x03ffff (0x5fbb bytes) 000613 1775194346.484701 punch from 0x03cdd0 to 0x03ffff (0x3230 bytes) 000614 1775194346.493514 mapread 0x018ffc thru 0x02180b (0x8810 bytes) 000615 1775194346.514994 trunc from 0x036efb to 0x03705a (0x160 bytes) 000616 1775194346.522058 read 0x02fbf0 thru 0x038f38 (0x9349 bytes) 000617 1775194346.537224 trunc from 0x040000 to 0x01c1f0 (0x23e0f bytes) 000618 1775194346.558556 trunc from 0x012d82 to 0x01b78d (0x8a0c bytes) 000619 1775194346.571619 mapwrite 0x012b50 thru 0x022760 (0xfc11 bytes) 000620 1775194346.652036 trunc from 0x022761 to 0x014898 (0xdec8 bytes) 000621 1775194346.678017 mapwrite 0x0142af thru 0x01b039 (0x6d8b bytes) 000622 1775194346.746455 fallocate 0x0373db thru 0x03ffff (0x8c25 bytes) 000623 1775194346.753820 mapwrite 0x00148e thru 0x010a9e (0xf611 bytes) 000624 1775194346.779636 write 0x0037da thru 0x006a5d (0x3284 bytes) 000625 1775194346.780773 mapwrite 0x01deed thru 0x021cfe (0x3e12 bytes) 000626 1775194346.850823 MIRROR_RESYNC 7 mirrors 000627 1775194349.100269 write 0x0197ea thru 0x02252a (0x8d41 bytes) 000628 1775194349.175483 trunc from 0x02252b to 0x02728b (0x4d61 bytes) 000629 1775194349.232571 mapread 0x011376 thru 0x018caf (0x793a bytes) 000630 1775194349.261286 write 0x0129ee thru 0x01e234 (0xb847 bytes) 000631 1775194349.264685 mapwrite 0x012302 thru 0x020d5e (0xea5d bytes) 000632 1775194349.303867 trunc from 0x02728c to 0x011a15 (0x15876 bytes) 000633 1775194349.354346 fallocate 0x00c2c4 thru 0x00d0e2 (0xe1f bytes) 000634 1775194349.366391 read 0x00e718 thru 0x00fcc9 (0x15b2 bytes) 000635 1775194349.368325 trunc from 0x011a16 to 0x00b332 (0x66e3 bytes) 000637 1775194349.393355 mapwrite 0x03c0f0 thru 0x03ffff (0x3f10 bytes) 000638 1775194349.527421 MIRROR_RESYNC 7 mirrors 000639 1775194351.262857 read 0x032171 thru 0x03e064 (0xbef4 bytes) 000640 1775194351.277024 fallocate 0x012b90 thru 0x01db39 (0xafaa bytes) 000641 1775194351.311910 trunc from 0x040000 to 0x0359e2 (0xa61d bytes) 000642 1775194351.335631 read 0x017b13 thru 0x01d239 (0x5727 bytes) 000643 1775194351.344669 read 0x0218d5 thru 0x02c71f (0xae4b bytes) 000644 1775194351.345906 trunc from 0x0359e3 to 0x01ae65 (0x1ab7d bytes) 000645 1775194351.371894 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1ae66): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae66): Mirror 4: 0x75a7eeba (stale) Mirror 8: 0x75a7eeba (stale) Mirror 9: 0x75a7eeba (stale) Mirror 10: 0x75a7eeba (stale) Mirror 11: 0x75a7eeba (stale) Mirror 13: 0x75a7eeba (stale) CRC-32 checksum value for chunk [0, 0x1ae66): Mirror 1: 0x75a7eeba Mirror 14: 0x75a7eeba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae66: skipped 000646 1775194352.263492 fallocate 0x005918 thru 0x011eac (0xc595 bytes) 000647 1775194352.309039 write 0x02b27f thru 0x03196d (0x66ef bytes) 000648 1775194352.315095 write 0x033ab0 thru 0x03ffff (0xc550 bytes) 000649 1775194352.316722 read 0x017313 thru 0x01df48 (0x6c36 bytes) 000650 1775194352.329600 trunc from 0x040000 to 0x03a188 (0x5e77 bytes) 000651 1775194352.374661 trunc from 0x012113 to 0x013514 (0x1402 bytes) 000652 1775194352.379474 mapwrite 0x03a0c6 thru 0x03ffff (0x5f3a bytes) 000653 1775194352.430793 trunc from 0x011bc4 to 0x01e184 (0xc5c1 bytes) 000654 1775194352.438779 read 0x01c20c thru 0x027f1a (0xbd0f bytes) 000655 1775194352.472266 trunc from 0x040000 to 0x02978b (0x16874 bytes) 000656 1775194352.504429 mapread 0x012057 thru 0x016d78 (0x4d22 bytes) 000657 1775194352.506640 mapwrite 0x02b0e8 thru 0x03609c (0xafb5 bytes) 000660 1775194352.701474 punch from 0x00691e to 0x015156 (0xe839 bytes) 000661 1775194352.708827 mapwrite 0x027369 thru 0x02f035 (0x7ccd bytes) 000662 1775194352.732638 mapread 0x012419 thru 0x01b305 (0x8eed bytes) 000663 1775194352.756271 mapread 0x0298d0 thru 0x030485 (0x6bb6 bytes) 000664 1775194352.757414 punch from 0x007b48 to 0x015241 (0xd6fa bytes) 000665 1775194352.768136 trunc from 0x03609d to 0x0203d3 (0x15cc9 bytes) 000666 1775194352.815733 write 0x02c344 thru 0x02fe72 (0x3b2f bytes) 000667 1775194352.817575 read 0x01a3dc thru 0x01b150 (0xd75 bytes) 000668 1775194352.828481 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fe73): Mirror 4: 0xd742289e (stale) Mirror 8: 0xd742289e (stale) Mirror 9: 0xd742289e (stale) Mirror 10: 0xd742289e (stale) Mirror 11: 0xd742289e (stale) Mirror 14: 0xd742289e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fe73): Mirror 4: 0xd742289e (stale) Mirror 8: 0xd742289e (stale) Mirror 9: 0xd742289e (stale) Mirror 10: 0xd742289e (stale) Mirror 11: 0xd742289e (stale) Mirror 14: 0xd742289e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fe73: skipped 000669 1775194353.639416 read 0x011d9d thru 0x01315f (0x13c3 bytes) 000670 1775194353.645434 read 0x00ea58 thru 0x01b105 (0xc6ae bytes) 000671 1775194353.655434 fallocate 0x005f81 thru 0x014c88 (0xed08 bytes) 000672 1775194353.661940 write 0x036964 thru 0x03b0a4 (0x4741 bytes) 000673 1775194353.667868 mapread 0x00476c thru 0x00be6e (0x7703 bytes) 000674 1775194353.671010 write 0x02e74a thru 0x03c9d7 (0xe28e bytes) 000676 1775194353.679321 trunc from 0x03c9d8 to 0x026ff9 (0x159de bytes) 000677 1775194353.703274 mapread 0x0240e0 thru 0x024743 (0x664 bytes) 000678 1775194353.707895 fallocate 0x02a5c5 thru 0x02c098 (0x1ad4 bytes) 000679 1775194353.713709 fallocate 0x03e8a8 thru 0x03ffff (0x1758 bytes) 000680 1775194353.719356 fallocate 0x0325c8 thru 0x03f839 (0xd272 bytes) 000681 1775194353.724926 trunc from 0x026ffa to 0x015770 (0x11889 bytes) 000682 1775194353.743701 read 0x007894 thru 0x00b844 (0x3fb1 bytes) 000683 1775194353.746433 trunc from 0x010971 to 0x015770 (0x4e00 bytes) 000684 1775194353.752598 write 0x02d84f thru 0x031225 (0x39d7 bytes) 000685 1775194353.754985 trunc from 0x00ca53 to 0x013308 (0x68b6 bytes) 000686 1775194353.763166 fallocate 0x010f43 thru 0x01825d (0x731b bytes) 000687 1775194353.769588 trunc from 0x031226 to 0x0287a5 (0x8a80 bytes) 000688 1775194353.793197 trunc from 0x0287a6 to 0x02e3ae (0x5c09 bytes) 000689 1775194353.810997 trunc from 0x02e3af to 0x039c1a (0xb86c bytes) 000690 1775194353.829954 read 0x02c05e thru 0x02d0de (0x1081 bytes) 000692 1775194353.841949 mapread 0x00d768 thru 0x011bea (0x4483 bytes) 000693 1775194353.843023 trunc from 0x024126 to 0x032a98 (0xe973 bytes) 000694 1775194353.848724 read 0x0226bd thru 0x02d25b (0xab9f bytes) 000695 1775194353.871122 trunc from 0x039c1b to 0x03866e (0x15ac bytes) 000698 1775194353.886846 write 0x03acb8 thru 0x03ffff (0x5348 bytes) 000699 1775194353.889021 read 0x011dae thru 0x01824d (0x64a0 bytes) 000701 1775194353.891600 mapwrite 0x031dd0 thru 0x03a736 (0x8967 bytes) 000702 1775194353.920512 punch from 0x001974 to 0x0032ca (0x1957 bytes) 000703 1775194353.925544 trunc from 0x040000 to 0x00c9a7 (0x33658 bytes) 000704 1775194353.947902 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: 0xb21e485a (stale) Mirror 8: 0xb21e485a (stale) Mirror 9: 0xb21e485a (stale) Mirror 10: 0xb21e485a (stale) Mirror 11: 0xb21e485a (stale) Mirror 14: 0xb21e485a (stale) CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 1: 0x3d28a770 Mirror 15: 0x3d28a770 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9a8: skipped 000705 1775194354.990979 read 0x002326 thru 0x005cec (0x39c7 bytes) 000706 1775194355.006024 trunc from 0x0025f6 to 0x00a3ee (0x7df9 bytes) 000707 1775194355.073237 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: 0xb21e485a (stale) Mirror 8: 0xb21e485a (stale) Mirror 9: 0xb21e485a (stale) Mirror 10: 0xb21e485a (stale) Mirror 11: 0xb21e485a (stale) Mirror 15: 0x3d28a770 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: 0xb21e485a (stale) Mirror 8: 0xb21e485a (stale) Mirror 9: 0xb21e485a (stale) Mirror 10: 0xb21e485a (stale) Mirror 11: 0xb21e485a (stale) Mirror 15: 0x3d28a770 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9a8: skipped 000708 1775194355.603613 fallocate 0x039681 thru 0x03ecc3 (0x5643 bytes) 000709 1775194355.626506 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 4: 0xb21e485a (stale) Mirror 8: 0xb21e485a (stale) Mirror 9: 0xb21e485a (stale) Mirror 10: 0xb21e485a (stale) Mirror 11: 0xb21e485a (stale) Mirror 15: 0x3d28a770 (stale) CRC-32 checksum value for chunk [0, 0xc9a8): Mirror 1: 0xc7d645c1 Mirror 16: 0xc7d645c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9a8: skipped 000710 1775194356.500860 trunc from 0x00c9a8 to 0x016f5f (0xa5b8 bytes) 000711 1775194356.580417 trunc from 0x00644e to 0x00a7ea (0x439d bytes) 000712 1775194356.586620 trunc from 0x016f60 to 0x01e664 (0x7705 bytes) 000714 1775194356.614517 trunc from 0x01e665 to 0x02ef3c (0x108d8 bytes) 000715 1775194356.635179 punch from 0x02cd86 to 0x02ef3c (0x21b7 bytes) 000716 1775194356.641368 mapread 0x019ca8 thru 0x029c2c (0xff85 bytes) 000718 1775194356.672526 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0xb77c6b50 (stale) Mirror 8: 0xb77c6b50 (stale) Mirror 9: 0xb77c6b50 (stale) Mirror 10: 0xb77c6b50 (stale) Mirror 15: 0x1c353bf7 (stale) Mirror 16: 0x2c3e417f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0xb77c6b50 (stale) Mirror 8: 0xb77c6b50 (stale) Mirror 9: 0xb77c6b50 (stale) Mirror 10: 0xb77c6b50 (stale) Mirror 15: 0x1c353bf7 (stale) Mirror 16: 0x2c3e417f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000719 1775194357.123541 fallocate 0x025030 thru 0x033fba (0xef8b bytes) 000721 1775194357.132507 read 0x022199 thru 0x02ef3c (0xcda4 bytes) 000722 1775194357.155916 mapwrite 0x00cda3 thru 0x00e147 (0x13a5 bytes) 000723 1775194357.202452 MIRROR_RESYNC 7 mirrors 000724 1775194358.623771 trunc from 0x01241b to 0x01a7ff (0x83e5 bytes) 000725 1775194358.658741 punch from 0x005672 to 0x005d3e (0x6cd bytes) 000726 1775194358.666618 MIRROR_RESYNC 7 mirrors 000727 1775194359.975870 mapwrite 0x00ae11 thru 0x017268 (0xc458 bytes) 000728 1775194360.073300 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 8: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 10: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 1: 0x672c7ace Mirror 17: 0x672c7ace lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000729 1775194361.076675 trunc from 0x000bc9 to 0x007d2c (0x7164 bytes) 000730 1775194361.117396 read 0x021cf4 thru 0x02d6cf (0xb9dc bytes) 000732 1775194361.132392 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 10: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 10: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000733 1775194361.651636 write 0x007119 thru 0x00c3da (0x52c2 bytes) 000734 1775194361.667678 trunc from 0x02ace5 to 0x02ef3c (0x4258 bytes) 000735 1775194361.678070 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000736 1775194362.057780 fallocate 0x008d4d thru 0x012af7 (0x9dab bytes) 000737 1775194362.070064 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 1: 0xc647de54 Mirror 18: 0xc647de54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000738 1775194362.849036 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 4: 0x116b79aa (stale) Mirror 9: 0x116b79aa (stale) Mirror 15: 0x116b79aa (stale) Mirror 16: 0x116b79aa (stale) Mirror 17: 0x672c7ace (stale) CRC-32 checksum value for chunk [0, 0x2ef3d): Mirror 1: 0xc647de54 Mirror 18: 0xc647de54 Mirror 19: 0xc647de54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef3d: skipped 000739 1775194363.785866 mapwrite 0x00921f thru 0x00be40 (0x2c22 bytes) 000740 1775194363.910704 mapwrite 0x0299d8 thru 0x0378ee (0xdf17 bytes) 000741 1775194363.990122 fallocate 0x01dbe5 thru 0x02a7e0 (0xcbfc bytes) 000742 1775194363.997519 write 0x031b81 thru 0x033157 (0x15d7 bytes) 000743 1775194363.998907 trunc from 0x01d28c to 0x0246d6 (0x744b bytes) 000745 1775194364.004788 mapwrite 0x01f457 thru 0x02b847 (0xc3f1 bytes) 000746 1775194364.035126 mapwrite 0x008a60 thru 0x017aa7 (0xf048 bytes) 000747 1775194364.069790 trunc from 0x0378ef to 0x03a4a8 (0x2bba bytes) 000748 1775194364.088308 trunc from 0x03a4a9 to 0x019266 (0x21242 bytes) 000749 1775194364.116416 fallocate 0x02e4ad thru 0x032d4f (0x48a3 bytes) 000750 1775194364.122535 punch from 0x01b633 to 0x02317b (0x7b49 bytes) 000751 1775194364.130431 read 0x00e90e thru 0x011d74 (0x3467 bytes) 000752 1775194364.133379 trunc from 0x00c256 to 0x00dba6 (0x1951 bytes) 000753 1775194364.138636 trunc from 0x027aa4 to 0x02aaa4 (0x3001 bytes) 000755 1775194364.144607 mapread 0x023620 thru 0x02a158 (0x6b39 bytes) 000756 1775194364.156535 punch from 0x024a71 to 0x032d4f (0xe2df bytes) 000757 1775194364.161643 trunc from 0x007b43 to 0x016167 (0xe625 bytes) 000758 1775194364.173361 punch from 0x008337 to 0x00c5a9 (0x4273 bytes) 000759 1775194364.178457 read 0x02bd59 thru 0x032d4f (0x6ff7 bytes) 000760 1775194364.187887 read 0x006a62 thru 0x0114a9 (0xaa48 bytes) 000761 1775194364.189019 trunc from 0x032d50 to 0x00a01b (0x28d34 bytes) 000762 1775194364.206138 trunc from 0x000c0b to 0x0052bb (0x46b1 bytes) 000763 1775194364.212117 mapwrite 0x033f41 thru 0x03ffff (0xc0bf bytes) 000764 1775194364.270948 read 0x029778 thru 0x039525 (0xfdae bytes) 000765 1775194364.272071 punch from 0x005a2b to 0x008251 (0x2827 bytes) 000766 1775194364.278121 mapwrite 0x00ed8e thru 0x01e728 (0xf99b bytes) 000767 1775194364.296845 trunc from 0x027ccf to 0x032251 (0xa583 bytes) 000768 1775194364.301538 punch from 0x019936 to 0x026fff (0xd6ca bytes) 000770 1775194364.311800 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xab7b51cf (stale) Mirror 9: 0xab7b51cf (stale) Mirror 15: 0xab7b51cf (stale) Mirror 16: 0xab7b51cf (stale) Mirror 17: 0x8c59c7b8 (stale) Mirror 19: 0xd89e4367 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xab7b51cf (stale) Mirror 9: 0xab7b51cf (stale) Mirror 15: 0xab7b51cf (stale) Mirror 16: 0xab7b51cf (stale) Mirror 17: 0x8c59c7b8 (stale) Mirror 19: 0xd89e4367 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1775194364.700801 fallocate 0x03e5cf thru 0x03ffff (0x1a31 bytes) 000772 1775194364.713618 punch from 0x02051d to 0x021b72 (0x1656 bytes) 000773 1775194364.719383 mapread 0x00cb64 thru 0x019de1 (0xd27e bytes) 000774 1775194364.780628 read 0x01d47d thru 0x02529c (0x7e20 bytes) 000775 1775194364.788134 mapwrite 0x0305e4 thru 0x03a7c4 (0xa1e1 bytes) 000776 1775194364.834203 mapwrite 0x033992 thru 0x03ffff (0xc66e bytes) 000778 1775194364.860905 mapread 0x00f21a thru 0x017fad (0x8d94 bytes) 000779 1775194364.863477 trunc from 0x036595 to 0x03ce5e (0x68ca bytes) 000780 1775194364.873308 mapwrite 0x001386 thru 0x00c074 (0xacef bytes) 000781 1775194364.920627 mapread 0x00e911 thru 0x01691e (0x800e bytes) 000782 1775194364.923258 mapread 0x009709 thru 0x016257 (0xcb4f bytes) 000783 1775194364.924706 mapread 0x02345f thru 0x02ad99 (0x793b bytes) 000784 1775194364.936103 MIRROR_RESYNC 7 mirrors 000785 1775194367.061735 mapread 0x0070fe thru 0x01240d (0xb310 bytes) 000786 1775194367.092685 mapwrite 0x020854 thru 0x02a981 (0xa12e bytes) 000787 1775194367.208778 mapread 0x02eea7 thru 0x0302cb (0x1425 bytes) 000789 1775194367.212770 mapwrite 0x02a73b thru 0x03676f (0xc035 bytes) 000790 1775194367.257387 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc87a67ab (stale) Mirror 9: 0xc87a67ab (stale) Mirror 15: 0xc87a67ab (stale) Mirror 17: 0xc87a67ab (stale) Mirror 19: 0xc87a67ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc87a67ab (stale) Mirror 9: 0xc87a67ab (stale) Mirror 15: 0xc87a67ab (stale) Mirror 17: 0xc87a67ab (stale) Mirror 19: 0xc87a67ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1775194367.746978 mapwrite 0x024919 thru 0x02690d (0x1ff5 bytes) 000792 1775194367.780953 mapread 0x00e5be thru 0x00e65d (0xa0 bytes) 000793 1775194367.792214 fallocate 0x002849 thru 0x005e21 (0x35d9 bytes) 000794 1775194367.796882 write 0x0118a2 thru 0x012e87 (0x15e6 bytes) 000795 1775194367.806801 fallocate 0x02adc0 thru 0x039028 (0xe269 bytes) 000796 1775194367.816117 read 0x01b6cd thru 0x01e3cd (0x2d01 bytes) 000797 1775194367.824416 punch from 0x0208ae to 0x028d69 (0x84bc bytes) 000798 1775194367.832641 write 0x022100 thru 0x028447 (0x6348 bytes) 000799 1775194367.862904 MIRROR_RESYNC 6 mirrors 000800 1775194369.719618 punch from 0x003d80 to 0x00e8a5 (0xab26 bytes) 000801 1775194369.746618 write 0x031a84 thru 0x03ffff (0xe57c bytes) 000802 1775194369.755041 fallocate 0x022f10 thru 0x02c8d8 (0x99c9 bytes) 000803 1775194369.762093 fallocate 0x03e5a6 thru 0x03ec4c (0x6a7 bytes) 000804 1775194369.768581 fallocate 0x02e8eb thru 0x03003d (0x1753 bytes) 000805 1775194369.774720 mapread 0x03a513 thru 0x03ffff (0x5aed bytes) 000806 1775194369.778383 mapwrite 0x00bddf thru 0x00c4fc (0x71e bytes) 000807 1775194369.818367 punch from 0x025312 to 0x03409a (0xed89 bytes) 000808 1775194369.829489 trunc from 0x01643d to 0x023162 (0xcd26 bytes) 000809 1775194369.841809 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9dfffc37 (stale) Mirror 9: 0x9dfffc37 (stale) Mirror 17: 0x9dfffc37 (stale) Mirror 19: 0x9dfffc37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9dfffc37 (stale) Mirror 9: 0x9dfffc37 (stale) Mirror 17: 0x9dfffc37 (stale) Mirror 19: 0x9dfffc37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000811 1775194370.288005 read 0x039f47 thru 0x03abae (0xc68 bytes) 000812 1775194370.300074 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9dfffc37 (stale) Mirror 17: 0x9dfffc37 (stale) Mirror 19: 0x9dfffc37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9dfffc37 (stale) Mirror 17: 0x9dfffc37 (stale) Mirror 19: 0x9dfffc37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1775194370.692812 MIRROR_RESYNC 4 mirrors 000815 1775194372.156552 mapread 0x01ce19 thru 0x020598 (0x3780 bytes) 000816 1775194372.186475 trunc from 0x0043be to 0x005e66 (0x1aa9 bytes) 000817 1775194372.221262 fallocate 0x02a8e5 thru 0x032cce (0x83ea bytes) 000818 1775194372.232613 trunc from 0x040000 to 0x02942a (0x16bd5 bytes) 000819 1775194372.256137 trunc from 0x02942b to 0x02a16a (0xd40 bytes) 000820 1775194372.269047 trunc from 0x02a16b to 0x0384eb (0xe381 bytes) 000821 1775194372.282611 mapwrite 0x008277 thru 0x012a70 (0xa7fa bytes) 000822 1775194372.332797 trunc from 0x03453e to 0x0384eb (0x3fae bytes) 000823 1775194372.337309 mapread 0x02bdae thru 0x02e371 (0x25c4 bytes) 000825 1775194372.345571 fallocate 0x01a246 thru 0x01f0f7 (0x4eb2 bytes) 000826 1775194372.354524 read 0x02529c thru 0x02abff (0x5964 bytes) 000827 1775194372.355421 read 0x01caa8 thru 0x028c9a (0xc1f3 bytes) 000829 1775194372.368430 write 0x006f8e thru 0x010b3c (0x9baf bytes) 000830 1775194372.371627 trunc from 0x0282a5 to 0x0319ac (0x9708 bytes) 000831 1775194372.378231 fallocate 0x03c405 thru 0x03ffff (0x3bfb bytes) 000832 1775194372.396692 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9e810285 (stale) Mirror 17: 0x9e810285 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9e810285 (stale) Mirror 17: 0x9e810285 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000833 1775194372.814789 mapwrite 0x039a8c thru 0x03ffff (0x6574 bytes) 000834 1775194372.853841 read 0x015340 thru 0x02037e (0xb03f bytes) 000835 1775194372.863088 mapread 0x036cea thru 0x03e241 (0x7558 bytes) 000836 1775194372.865007 mapread 0x0035f1 thru 0x00ea2a (0xb43a bytes) 000837 1775194372.875730 trunc from 0x040000 to 0x017315 (0x28cea bytes) 000838 1775194372.897460 read 0x00746a thru 0x00a7df (0x3376 bytes) 000839 1775194372.898109 trunc from 0x017316 to 0x03334a (0x1c035 bytes) 000840 1775194372.919534 trunc from 0x02d0ea to 0x031909 (0x4820 bytes) 000841 1775194372.927250 fallocate 0x00f01d thru 0x01d243 (0xe227 bytes) 000842 1775194372.934123 punch from 0x01590d to 0x024f3d (0xf631 bytes) 000843 1775194372.943042 trunc from 0x02d481 to 0x03334a (0x5eca bytes) 000844 1775194372.961430 mapread 0x010df1 thru 0x01994b (0x8b5b bytes) 000845 1775194372.993773 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3334b): Mirror 9: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3334b): Mirror 9: 0xffea43d (stale) Mirror 17: 0xffea43d (stale) CRC-32 checksum value for chunk [0, 0x3334b): Mirror 1: 0x19680d59 Mirror 18: 0x19680d59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3334b: skipped 000846 1775194374.052430 fallocate 0x01707d thru 0x01c7ae (0x5732 bytes) 000848 1775194374.093785 punch from 0x009c38 to 0x00bff3 (0x23bc bytes) 000849 1775194374.100370 punch from 0x00128b to 0x00ece3 (0xda59 bytes) 000850 1775194374.109357 trunc from 0x025d13 to 0x0307a0 (0xaa8e bytes) 000851 1775194374.114097 write 0x03b321 thru 0x03ffff (0x4cdf bytes) 000852 1775194374.119782 trunc from 0x040000 to 0x03a78a (0x5875 bytes) 000853 1775194374.149890 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a78b): Mirror 17: 0xe78bc4ff (stale) Mirror 18: 0x79be6d53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a78b): Mirror 17: 0xe78bc4ff (stale) Mirror 18: 0x79be6d53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a78b: skipped 000854 1775194374.454756 read 0x00ad68 thru 0x0182d6 (0xd56f bytes) 000855 1775194374.468780 mapwrite 0x007f48 thru 0x009a35 (0x1aee bytes) 000856 1775194374.498481 read 0x015df9 thru 0x0219ed (0xbbf5 bytes) 000857 1775194374.512357 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a78b): Mirror 17: 0xe78bc4ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a78b): Mirror 17: 0xe78bc4ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a78b: skipped 000858 1775194374.878236 read 0x01c918 thru 0x01d0ad (0x796 bytes) 000859 1775194374.887465 fallocate 0x0111b2 thru 0x0146d9 (0x3528 bytes) 000860 1775194374.892065 trunc from 0x03a78b to 0x031345 (0x9445 bytes) 000862 1775194374.926839 punch from 0x02f12d to 0x031345 (0x2219 bytes) 000863 1775194374.933962 trunc from 0x031346 to 0x01dee4 (0x13461 bytes) 000864 1775194374.950517 punch from 0x0042d4 to 0x013654 (0xf381 bytes) 000865 1775194374.955953 fallocate 0x01d8bc thru 0x020467 (0x2bac bytes) 000866 1775194374.960848 write 0x03f398 thru 0x03ffff (0xc68 bytes) 000867 1775194374.961900 mapread 0x01171d thru 0x01fbd0 (0xe4b4 bytes) 000868 1775194374.994975 mapread 0x01f581 thru 0x02d0a4 (0xdb24 bytes) 000869 1775194375.003829 write 0x02c852 thru 0x03a51f (0xdcce bytes) 000870 1775194375.008570 read 0x00cf91 thru 0x012e79 (0x5ee9 bytes) 000871 1775194375.014032 MIRROR_RESYNC 2 mirrors 000872 1775194375.717240 punch from 0x0106a3 to 0x01bc45 (0xb5a3 bytes) 000873 1775194375.768319 fallocate 0x00964f thru 0x00c273 (0x2c25 bytes) 000874 1775194375.773747 punch from 0x013953 to 0x018e11 (0x54bf bytes) 000875 1775194375.778423 read 0x0213c9 thru 0x029c36 (0x886e bytes) 000876 1775194375.790702 trunc from 0x040000 to 0x000dab (0x3f254 bytes) 000877 1775194375.854565 mapwrite 0x00a85c thru 0x01440a (0x9baf bytes) 000879 1775194375.954432 punch from 0x007f49 to 0x01440a (0xc4c2 bytes) 000880 1775194375.961331 mapwrite 0x031a79 thru 0x03f650 (0xdbd8 bytes) 000881 1775194376.040370 mapread 0x03defd thru 0x03f650 (0x1754 bytes) 000882 1775194376.044532 mapwrite 0x03bc89 thru 0x03dc55 (0x1fcd bytes) 000883 1775194376.061295 mapwrite 0x004483 thru 0x008c58 (0x47d6 bytes) 000884 1775194376.100659 mapread 0x02ebcd thru 0x03a9d1 (0xbe05 bytes) 000885 1775194376.101851 punch from 0x00678a to 0x013f9a (0xd811 bytes) 000886 1775194376.107618 trunc from 0x01a264 to 0x022551 (0x82ee bytes) 000887 1775194376.117420 MIRROR_RESYNC 2 mirrors 000888 1775194376.884682 mapread 0x037201 thru 0x03cae2 (0x58e2 bytes) 000889 1775194376.905523 read 0x035098 thru 0x03e95d (0x98c6 bytes) 000890 1775194376.906634 trunc from 0x03f651 to 0x0047fe (0x3ae52 bytes) 000892 1775194377.030478 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) 000893 1775194377.228694 mapwrite 0x01e1fe thru 0x02b31d (0xd120 bytes) 000894 1775194377.370918 trunc from 0x0167c4 to 0x016bde (0x41b bytes) 000895 1775194377.382738 trunc from 0x02b31e to 0x01706a (0x142b3 bytes) 000896 1775194377.415561 punch from 0x00c513 to 0x00dec3 (0x19b1 bytes) 000897 1775194377.422716 mapwrite 0x01e499 thru 0x027319 (0x8e81 bytes) 000898 1775194377.518036 MIRROR_EXTEND to 2 mirrors 000900 1775194378.248763 trunc from 0x019cac to 0x01f80b (0x5b60 bytes) 000901 1775194378.293285 read 0x001291 thru 0x00c66e (0xb3de bytes) 000902 1775194378.306289 mapwrite 0x028b88 thru 0x02ec45 (0x60be bytes) 000903 1775194378.408326 read 0x01260c thru 0x014291 (0x1c86 bytes) 000904 1775194378.424935 trunc from 0x02ec46 to 0x03166a (0x2a25 bytes) 000905 1775194378.459506 read 0x018e1f thru 0x02686e (0xda50 bytes) 000906 1775194378.460678 fallocate 0x019805 thru 0x01b8fe (0x20fa bytes) 000907 1775194378.470880 read 0x008ec4 thru 0x012b81 (0x9cbe bytes) 000908 1775194378.471556 punch from 0x02f163 to 0x03166a (0x2508 bytes) 000909 1775194378.479362 mapwrite 0x0269df thru 0x02b374 (0x4996 bytes) 000910 1775194378.512947 trunc from 0x03166b to 0x004964 (0x2cd06 bytes) 000911 1775194378.534009 mapwrite 0x000dce thru 0x007782 (0x69b5 bytes) 000912 1775194378.579378 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7783): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7783): Mirror 2: 0x739f8a2f (stale) CRC-32 checksum value for chunk [0, 0x7783): Mirror 1: 0xac98b2a4 Mirror 3: 0xac98b2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7783: skipped 000914 1775194379.126833 trunc from 0x007783 to 0x011425 (0x9ca3 bytes) 000915 1775194379.215460 fallocate 0x006677 thru 0x0134ee (0xce78 bytes) 000916 1775194379.222514 trunc from 0x0134ef to 0x010008 (0x34e6 bytes) 000917 1775194379.247397 write 0x031b15 thru 0x03abd4 (0x90c0 bytes) 000918 1775194379.249028 mapwrite 0x00de76 thru 0x010eb9 (0x3044 bytes) 000919 1775194379.289511 write 0x000cf4 thru 0x00169c (0x9a9 bytes) 000920 1775194379.301376 read 0x006138 thru 0x00bc4f (0x5b18 bytes) 000921 1775194379.302274 punch from 0x026dcd to 0x02891f (0x1b53 bytes) 000922 1775194379.307376 mapwrite 0x01152c thru 0x012478 (0xf4d bytes) 000923 1775194379.324814 mapwrite 0x036d90 thru 0x03787c (0xaed bytes) 000924 1775194379.342254 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3abd5): Mirror 3: 0xbf3a99e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abd5): Mirror 3: 0xbf3a99e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abd5: skipped 000925 1775194379.635977 fallocate 0x02c7f5 thru 0x0318dd (0x50e9 bytes) 000926 1775194379.643941 mapwrite 0x015735 thru 0x02130f (0xbbdb bytes) 000927 1775194379.738199 write 0x027b3a thru 0x035db6 (0xe27d bytes) 000928 1775194379.763610 punch from 0x01077d to 0x0146d4 (0x3f58 bytes) 000929 1775194379.778097 punch from 0x02af9e to 0x0381de (0xd241 bytes) 000930 1775194379.806497 fallocate 0x01a0b4 thru 0x01c33a (0x2287 bytes) 000931 1775194379.815398 write 0x039720 thru 0x03ffff (0x68e0 bytes) 000932 1775194379.830814 trunc from 0x040000 to 0x03d116 (0x2ee9 bytes) 000933 1775194379.850982 trunc from 0x03ad50 to 0x03d116 (0x23c7 bytes) 000934 1775194379.870079 mapread 0x006f8a thru 0x00b4cb (0x4542 bytes) 000935 1775194379.882029 trunc from 0x008374 to 0x00ad8e (0x2a1b bytes) 000936 1775194379.897165 write 0x00b339 thru 0x0131c0 (0x7e88 bytes) 000938 1775194379.911995 mapread 0x02067b thru 0x027282 (0x6c08 bytes) 000939 1775194379.918330 fallocate 0x02e477 thru 0x03d2f5 (0xee7f bytes) 000940 1775194379.931279 mapwrite 0x02f523 thru 0x03b99a (0xc478 bytes) 000941 1775194380.012186 fallocate 0x00d9d6 thru 0x01b1ed (0xd818 bytes) 000942 1775194380.028469 punch from 0x031d6b to 0x0399f4 (0x7c8a bytes) 000944 1775194380.040945 punch from 0x00303b to 0x00fc8b (0xcc51 bytes) 000945 1775194380.053821 mapread 0x0067d9 thru 0x0167a8 (0xffd0 bytes) 000946 1775194380.091188 write 0x031aa1 thru 0x03ffff (0xe55f bytes) 000947 1775194380.124234 write 0x038913 thru 0x03ffff (0x76ed bytes) 000948 1775194380.125430 fallocate 0x0120d9 thru 0x021e05 (0xfd2d bytes) 000949 1775194380.132401 mapwrite 0x00c160 thru 0x0149fd (0x889e bytes) 000950 1775194380.157353 mapwrite 0x010a17 thru 0x01ad2d (0xa317 bytes) 000951 1775194380.198267 punch from 0x013faf to 0x014e6a (0xebc bytes) 000952 1775194380.212185 trunc from 0x040000 to 0x01c132 (0x23ecd bytes) 000953 1775194380.233371 read 0x008cd1 thru 0x015a7a (0xcdaa bytes) 000955 1775194380.245026 mapwrite 0x02a5af thru 0x030c84 (0x66d6 bytes) 000956 1775194380.299533 mapwrite 0x03f471 thru 0x03ffff (0xb8f bytes) 000957 1775194380.355480 write 0x01fab6 thru 0x02f0fe (0xf649 bytes) 000958 1775194380.368044 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe3358a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a2954e7 Mirror 4: 0x5a2954e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1775194381.514873 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe3358a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a2954e7 Mirror 4: 0x5a2954e7 Mirror 5: 0x5a2954e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000960 1775194382.458425 mapwrite 0x005112 thru 0x0062f1 (0x11e0 bytes) 000961 1775194382.601417 punch from 0x003db2 to 0x00d779 (0x99c8 bytes) 000962 1775194382.608613 trunc from 0x01a27a to 0x02076d (0x64f4 bytes) 000963 1775194382.613196 fallocate 0x004b15 thru 0x00b0d2 (0x65be bytes) 000964 1775194382.622711 mapwrite 0x025aa2 thru 0x028e61 (0x33c0 bytes) 000965 1775194382.670487 mapread 0x013bce thru 0x0225be (0xe9f1 bytes) 000966 1775194382.722564 punch from 0x007209 to 0x0072bd (0xb5 bytes) 000967 1775194382.728960 punch from 0x01d8d3 to 0x02769c (0x9dca bytes) 000968 1775194382.747217 mapread 0x0345cf thru 0x03eec2 (0xa8f4 bytes) 000970 1775194382.786526 fallocate 0x03f173 thru 0x03f40f (0x29d bytes) 000971 1775194382.796419 mapread 0x023140 thru 0x023a31 (0x8f2 bytes) 000972 1775194382.811069 fallocate 0x03db50 thru 0x03ffff (0x24b0 bytes) 000973 1775194382.817100 write 0x0387bf thru 0x03ffff (0x7841 bytes) 000974 1775194382.822861 write 0x006ec8 thru 0x0135c1 (0xc6fa bytes) 000975 1775194382.840087 write 0x025d42 thru 0x02ae3b (0x50fa bytes) 000976 1775194382.841282 punch from 0x016285 to 0x01d718 (0x7494 bytes) 000977 1775194382.849316 mapwrite 0x01840a thru 0x01ac23 (0x281a bytes) 000978 1775194382.883927 trunc from 0x001fdb to 0x008eba (0x6ee0 bytes) 000979 1775194382.888449 trunc from 0x017286 to 0x0221dd (0xaf58 bytes) 000982 1775194382.898466 trunc from 0x040000 to 0x007e79 (0x38186 bytes) 000983 1775194382.925559 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7e7a): Mirror 3: 0x5a7a31a1 (stale) Mirror 5: 0x3428be07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7e7a): Mirror 3: 0x5a7a31a1 (stale) Mirror 5: 0x3428be07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7e7a: skipped 000984 1775194383.211593 punch from 0x006657 to 0x007e79 (0x1823 bytes) 000985 1775194383.226132 read 0x006cbd thru 0x007e79 (0x11bd bytes) 000986 1775194383.239051 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x7e7a): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7e7a): Mirror 3: 0x5a7a31a1 (stale) Mirror 5: 0x3428be07 (stale) CRC-32 checksum value for chunk [0, 0x7e7a): Mirror 1: 0xb03d8088 Mirror 6: 0xb03d8088 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7e7a: skipped 000988 1775194384.248041 fallocate 0x01e2fc thru 0x02aec9 (0xcbce bytes) 000989 1775194384.304454 punch from 0x011c50 to 0x01ddee (0xc19f bytes) 000990 1775194384.335091 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aeca): Mirror 5: 0xfae9d77c (stale) Mirror 6: 0x577f62cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aeca): Mirror 5: 0xfae9d77c (stale) Mirror 6: 0x577f62cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aeca: skipped 000991 1775194384.758041 mapwrite 0x0192f2 thru 0x01e414 (0x5123 bytes) 000992 1775194384.811960 mapread 0x013fa0 thru 0x01da98 (0x9af9 bytes) 000993 1775194384.830122 read 0x01c79f thru 0x022303 (0x5b65 bytes) 000994 1775194384.845433 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aeca): Mirror 5: 0xfae9d77c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aeca): Mirror 5: 0xfae9d77c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aeca: skipped 000995 1775194385.168593 mapread 0x01afe8 thru 0x029178 (0xe191 bytes) 000997 1775194385.197277 mapread 0x01b957 thru 0x01f4d6 (0x3b80 bytes) 000998 1775194385.198354 trunc from 0x020949 to 0x027a0d (0x70c5 bytes) 000999 1775194385.203236 mapread 0x010022 thru 0x01b3d2 (0xb3b1 bytes) 001000 1775194385.214660 mapread 0x027c8e thru 0x02aec9 (0x323c bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 01:33:10 (1775194390) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x57:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 01:33:21 (1775194401) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187022 s, 11.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941065 s, 11.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x97:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x97:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 01:33:29 (1775194409) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1767776 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2400576 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 4262528 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5869664 .. Extending file size to 8292352 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8363136 .. Extending file size to 8427552 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8621920 .. Extending file size to 8722880 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9359584 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59736 59738 59739 59741 59743 done 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:34:53 (1775194493) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4627904 .. Extending file size to 7271520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5070816 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8377056 .. Extending file size to 8184288 .. Extending file size to 8871008 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8287616 .. Extending file size to 8962240 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9139488 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60971 60972 60973 60977 60978 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:36:18 (1775194578) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25596 3581424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43396 7170644 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25596 3581424 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43396 7170644 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0937372 s, 22.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216877 s, 9.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x922f0db (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:36:39 (1775194599) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145931 s, 14.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 01:36:47 (1775194607) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x66:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:36:54 (1775194614) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11165 s, 18.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:37:02 (1775194622) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0968888 s, 21.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:37:10 (1775194630) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0835587 s, 25.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:37:43 (1775194663) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147814 s, 7.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x246:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x246:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x247:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:38:34 (1775194714) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157477 s, 13.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x248:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:39:23 (1775194763) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg329-server: Pool lustre.test_204f created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131659 s, 15.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg329-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:40:40 (1775194840) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:40:49 (1775194849) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:40:56 (1775194856) oleg329-server: Pool lustre.test_206 created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg329-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:41:23 (1775194883) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537722 s, 19.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625829 s, 16.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6e42358c Mirror 3: 0x6e42358c PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:41:34 (1775194894) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:41:38 (1775194898) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:41:41 (1775194901) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110943 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1920000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1860000 Read this (4 bytes): BBBB 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:42:19 (1775194939) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0179464 s, 456 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:42:26 (1775194946) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517143 s, 20.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 01:42:27.000000000 -0400 Modify: 2026-04-03 01:42:27.000000000 -0400 Change: 2026-04-03 01:42:29.000000000 -0400 Birth: 2026-04-03 01:42:28.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:42:34 (1775194954) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556109 s, 18.9 MB/s oleg329-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg329-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg329-server: O/280000401/d29/bd: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:43:00 (1775194980) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.236756 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0303634 s, 135 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 3 01:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 3 01:43 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:43:09 (1775194989) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:43:16 (1775194996) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1701 sec =========== 01:43:25 (1775195005) === sanity-flr: start cleanup 01:43:27 (1775195007) === === sanity-flr: finish cleanup 01:43:34 (1775195014) ===