-----============= acceptance-small: sanity-flr ============----- Thu May 28 09:28:52 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg631-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:29:09 (1779974949) === oleg631-client.virtnet: executing check_config_client /mnt/lustre oleg631-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg631-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a3b46285800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a3b46285800.idle_timeout=debug disable quota as required oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:29:42 (1779974982) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8219] == sanity-flr test 0a: lfs mirror create with -N option == 09:29:44 (1779974984) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:30:00 (1779975000) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:30:04 (1779975004) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:30:08 (1779975008) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:30:31 (1779975031) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:30:35 (1779975035) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 09:30:40 (1779975040) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a3b46285800.stats=clear osc.lustre-OST0001-osc-ffff9a3b46285800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:30:47 (1779975047) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:30:57 (1779975057) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:31:04 (1779975064) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:31:08 (1779975068) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 09:31:16 (1779975076) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:31:23 (1779975083) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:31:31 (1779975091) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:31:41 (1779975101) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:31:52 (1779975112) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:32:00 (1779975120) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.08987 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11611 s, 9.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:32:10 (1779975130) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.12305 s, 34.1 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:32:17 (1779975137) Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:33:25 (1779975205) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.252894 s, 20.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:34:06 (1779975246) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:35:24 (1779975324) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.88716 s, 18.2 MB/s -rw-r--r-- 1 root root 52428800 May 28 09:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.273s user 0m0.000s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.938s user 0m0.001s sys 0m0.157s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.775s user 0m0.002s sys 0m0.194s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg631-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a3b46285800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:36:54 (1779975414) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:36:57 (1779975417) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:37:01 (1779975421) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 09:37:05 (1779975425) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:37:13 (1779975433) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.575962 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:37:28 (1779975448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354534 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349174 s, 30.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:37:37 (1779975457) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:37:46 (1779975466) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114043 s, 898 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109968 s, 931 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000787558 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127211 s, 805 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 09:37:56 (1779975476) create test files with size 15742516 .. f36975dd0cc7f63cc83dc3c4cd6876e6 - 63f6b9adee347f831d53b456c3118869 - 2cae9b6952c77e68ca8970e06fa4bdc6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 09:38:21 (1779975501) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:38:36 (1779975516) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:38:44 (1779975524) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105632 s, 9.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348667 s, 30.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:38:53 (1779975533) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0905091 s, 23.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.420138 s, 10.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243884 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 09:39:13 (1779975553) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:39:17 (1779975557) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:39:21 (1779975561) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528621 s, 19.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 09:39:31 (1779975571) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.1981 s, 15.9 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:39:41 (1779975581) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:39:51 (1779975591) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350517 s, 29.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:40:04 (1779975604) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a3b462ec800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.561304 s, 18.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a3b5148f000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:40:23 (1779975623) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:41:09 (1779975669) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:41:50 (1779975710) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:41:53 (1779975713) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.207616 s, 15.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138808 s, 22.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa24f4929 (stale) Mirror 2: 0xa24f4929 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: b7f6c30b3f0f62d86b2c5ce5c122a664 - mirror 1 checksum: 838e362722977e9059e6891b61576f51 - mirror 2 checksum: b7f6c30b3f0f62d86b2c5ce5c122a664 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 28 09:41:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 28 09:41:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b7f6c30b3f0f62d86b2c5ce5c122a664 - mirror 1 checksum: 838e362722977e9059e6891b61576f51 - mirror 2 checksum: 838e362722977e9059e6891b61576f51 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 28 09:41:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 28 09:41:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:42:03 (1779975723) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219379 s, 47.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:42:10 (1779975730) Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 09:42:23 (1779975743) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 09:42:48 (1779975768) targets are mounted 09:42:48 (1779975768) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364075 s, 28.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x59:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:43:03 (1779975783) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0446847 s, 458 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0227852 s, 899 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:43:12 (1779975792) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0480436 s, 426 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00928949 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:43:22 (1779975802) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0304725 s, 538 kB/s PASS 50c (13s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:43:35 (1779975815) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0726324 s, 282 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (13s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:43:48 (1779975828) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0217189 s, 189 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0565875 s, 72.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:43:56 (1779975836) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:44:24 (1779975864) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:44:55 (1779975895) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:45:18 (1779975918) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:45:42 (1779975942) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 09:45:49 (1779975949) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55957 mirror split pid 55958 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55957 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55958 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:46:57 (1779976017) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6019 skipping zero size read 000003 1779976019.956178 write 0x01ee3c thru 0x026557 (0x771c bytes) 000004 1779976019.958699 punch from 0x018c8e to 0x021e85 (0x91f8 bytes) 000005 1779976019.984107 MIRROR_EXTEND to 2 mirrors zero_range to largest ever: 0x1b62b 000006 1779976021.228170 trunc from 0x00ec49 to 0x01b62a (0xc9e2 bytes) 000007 1779976021.299689 mapwrite 0x003c24 thru 0x00dc0a (0x9fe7 bytes) 000008 1779976021.360056 read 0x0056c4 thru 0x007634 (0x1f71 bytes) 000009 1779976021.360876 write 0x01f433 thru 0x02059e (0x116c bytes) 000010 1779976021.391685 read 0x01e90d thru 0x020b7b (0x226f bytes) 000011 1779976021.393463 punch from 0x02108f to 0x02548c (0x43fe bytes) 000012 1779976021.403674 trunc from 0x0065b9 to 0x00bda4 (0x57ec bytes) fallocating to largest ever: 0x40000 000013 1779976021.414748 fallocate 0x03d383 thru 0x03ffff (0x2c7d bytes) 000014 1779976021.421481 punch from 0x018fc3 to 0x026557 (0xd595 bytes) 000015 1779976021.433954 write 0x010340 thru 0x01c6c1 (0xc382 bytes) 000016 1779976021.449401 mapread 0x007fda thru 0x0178aa (0xf8d1 bytes) 000017 1779976021.450819 read 0x018dcc thru 0x01cd94 (0x3fc9 bytes) 000018 1779976021.455103 mapwrite 0x03c71c thru 0x03ffff (0x38e4 bytes) 000019 1779976021.540668 trunc from 0x040000 to 0x022626 (0x1d9d9 bytes) 000020 1779976021.567091 MIRROR_RESYNC 2 mirrors 000021 1779976022.603174 punch from 0x01a20f to 0x01e598 (0x438a bytes) 000022 1779976022.654611 mapread 0x01366d thru 0x022626 (0xefba bytes) 000023 1779976022.736887 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22627): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22627): Mirror 2: 0x8be14e07 (stale) CRC-32 checksum value for chunk [0, 0x22627): Mirror 1: 0xa3f40348 Mirror 3: 0xa3f40348 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22627: skipped 000024 1779976023.951527 punch from 0x020533 to 0x022626 (0x20f4 bytes) 000025 1779976023.984012 trunc from 0x022627 to 0x031e52 (0xf82c bytes) 000026 1779976024.010048 punch from 0x021d7b to 0x02f92a (0xdbb0 bytes) 000027 1779976024.015000 write 0x01849d thru 0x020131 (0x7c95 bytes) 000028 1779976024.028521 fallocate 0x0166e1 thru 0x01f07c (0x899c bytes) 000029 1779976024.037242 read 0x0314ee thru 0x031e52 (0x965 bytes) 000030 1779976024.045237 mapwrite 0x03ffa6 thru 0x03ffff (0x5a bytes) 000031 1779976024.085370 write 0x024e90 thru 0x02be67 (0x6fd8 bytes) 000032 1779976024.096180 read 0x014622 thru 0x02166f (0xd04e bytes) 000033 1779976024.106264 punch from 0x021227 to 0x025e27 (0x4c01 bytes) 000034 1779976024.131304 trunc from 0x00ea45 to 0x01c070 (0xd62c bytes) 000035 1779976024.140186 trunc from 0x040000 to 0x029f74 (0x1608b bytes) 000036 1779976024.155517 write 0x005a01 thru 0x00d557 (0x7b57 bytes) 000037 1779976024.156772 write 0x010b9b thru 0x01ea93 (0xdef9 bytes) 000038 1779976024.162783 punch from 0x0225a9 to 0x029f74 (0x79cc bytes) 000039 1779976024.169775 trunc from 0x029f75 to 0x01edf7 (0xb17d bytes) 000040 1779976024.194054 fallocate 0x01acd7 thru 0x01e519 (0x3843 bytes) 000041 1779976024.201590 read 0x00323f thru 0x012e0a (0xfbcc bytes) 000042 1779976024.213177 read 0x0179d3 thru 0x01dd5d (0x638b bytes) 000043 1779976024.214642 trunc from 0x01edf8 to 0x02720f (0x8418 bytes) 000044 1779976024.236832 fallocate 0x034122 thru 0x03ffff (0xbede bytes) 000045 1779976024.250197 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27210): Mirror 3: 0xb40491cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27210): Mirror 3: 0xb40491cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27210: skipped 000046 1779976024.786583 fallocate 0x02dd24 thru 0x035715 (0x79f2 bytes) 000047 1779976024.798480 trunc from 0x027210 to 0x03a0e7 (0x12ed8 bytes) 000048 1779976024.910833 mapread 0x034fe4 thru 0x03a0e7 (0x5104 bytes) 000049 1779976024.979227 trunc from 0x03a0e8 to 0x03087c (0x986b bytes) 000050 1779976025.000913 trunc from 0x03087d to 0x03b847 (0xafcb bytes) 000052 1779976025.023257 read 0x00434a thru 0x013663 (0xf31a bytes) 000053 1779976025.064569 mapread 0x0082df thru 0x012620 (0xa342 bytes) 000055 1779976025.072216 mapwrite 0x004625 thru 0x012674 (0xe050 bytes) 000056 1779976025.136927 fallocate 0x01f251 thru 0x02104c (0x1dfc bytes) 000057 1779976025.145860 mapwrite 0x00850d thru 0x00a9d5 (0x24c9 bytes) 000058 1779976025.175886 trunc from 0x024f36 to 0x034e80 (0xff4b bytes) 000059 1779976025.185393 punch from 0x02fef6 to 0x030567 (0x672 bytes) 000060 1779976025.196590 mapread 0x001030 thru 0x005325 (0x42f6 bytes) 000061 1779976025.215354 write 0x01da76 thru 0x02ca3c (0xefc7 bytes) 000062 1779976025.235224 mapread 0x02d30c thru 0x033e23 (0x6b18 bytes) 000063 1779976025.254805 trunc from 0x03b848 to 0x00d89e (0x2dfa9 bytes) 000064 1779976025.285162 mapwrite 0x02efc5 thru 0x034258 (0x5294 bytes) 000065 1779976025.340568 MIRROR_RESYNC 2 mirrors 000067 1779976026.262875 fallocate 0x011dfc thru 0x016109 (0x430e bytes) 000069 1779976026.305447 punch from 0x016b4d to 0x02534c (0xe800 bytes) 000070 1779976026.312067 trunc from 0x01379e to 0x018093 (0x48f6 bytes) 000071 1779976026.323834 MIRROR_RESYNC 2 mirrors 000072 1779976027.010562 mapwrite 0x006971 thru 0x010fa8 (0xa638 bytes) 000073 1779976027.206464 trunc from 0x034259 to 0x01414a (0x2010e bytes) 000074 1779976027.270285 write 0x007dba thru 0x00eaba (0x6d01 bytes) 000075 1779976027.299106 MIRROR_RESYNC 2 mirrors 000076 1779976028.238352 read 0x012cac thru 0x01414a (0x149f bytes) 000077 1779976028.249301 fallocate 0x03cb6e thru 0x03ffff (0x3492 bytes) 000078 1779976028.334884 trunc from 0x0197ec to 0x01d7ad (0x3fc2 bytes) 000079 1779976028.345910 trunc from 0x03d577 to 0x03ffff (0x2a89 bytes) 000080 1779976028.358503 punch from 0x019855 to 0x021613 (0x7dbf bytes) 000081 1779976028.367435 trunc from 0x040000 to 0x00a43d (0x35bc2 bytes) 000082 1779976028.417055 fallocate 0x03fa0b thru 0x03ffff (0x5f5 bytes) 000083 1779976028.424920 punch from 0x00279e to 0x0088e6 (0x6149 bytes) 000084 1779976028.435601 trunc from 0x00a43e to 0x0058e5 (0x4b58 bytes) 000085 1779976028.459146 read 0x005745 thru 0x0058e5 (0x1a1 bytes) 000086 1779976028.469552 read 0x00183f thru 0x0058e5 (0x40a7 bytes) 000088 1779976028.471524 mapread 0x000f17 thru 0x0058e5 (0x49cf bytes) 000089 1779976028.479595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) 000090 1779976028.736333 mapwrite 0x033a26 thru 0x0348b9 (0xe94 bytes) 000091 1779976028.803697 mapwrite 0x016ae9 thru 0x01f303 (0x881b bytes) 000094 1779976028.963833 fallocate 0x03fc19 thru 0x03ffff (0x3e7 bytes) 000095 1779976028.977664 trunc from 0x040000 to 0x01fc1d (0x203e2 bytes) 000096 1779976029.021387 read 0x01bc7a thru 0x01fc1d (0x3fa4 bytes) 000099 1779976029.031493 fallocate 0x0361c7 thru 0x03ffff (0x9e39 bytes) 000101 1779976029.046088 read 0x01f6a9 thru 0x02a62c (0xaf84 bytes) 000102 1779976029.068338 read 0x03949f thru 0x03ffff (0x6b61 bytes) 000104 1779976029.069650 fallocate 0x00170f thru 0x00e159 (0xca4b bytes) 000105 1779976029.073686 write 0x0329e4 thru 0x033907 (0xf24 bytes) 000107 1779976029.082012 trunc from 0x040000 to 0x02b5a0 (0x14a5f bytes) 000108 1779976029.111261 mapwrite 0x010096 thru 0x0113a9 (0x1314 bytes) 000109 1779976029.143242 mapwrite 0x020de1 thru 0x0237b3 (0x29d3 bytes) 000110 1779976029.171435 MIRROR_EXTEND to 2 mirrors 000111 1779976029.938730 trunc from 0x017c05 to 0x024c9e (0xd09a bytes) 000112 1779976030.019843 write 0x011254 thru 0x012332 (0x10df bytes) 000114 1779976030.046508 trunc from 0x02b5a1 to 0x01aea3 (0x106fd bytes) 000115 1779976030.160253 trunc from 0x01aea4 to 0x016676 (0x482d bytes) 000116 1779976030.176983 fallocate 0x00b59a thru 0x01129f (0x5d06 bytes) 000117 1779976030.186547 trunc from 0x016677 to 0x008138 (0xe53e bytes) 000118 1779976030.217736 mapread 0x006abf thru 0x008138 (0x167a bytes) 000119 1779976030.230039 punch from 0x006b51 to 0x008138 (0x15e8 bytes) 000120 1779976030.243613 mapwrite 0x02eb51 thru 0x031da2 (0x3252 bytes) 000121 1779976030.338441 mapread 0x00df2c thru 0x01bc37 (0xdd0c bytes) 000122 1779976030.362516 write 0x0017e9 thru 0x003034 (0x184c bytes) 000123 1779976030.375173 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31da3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31da3): Mirror 2: 0x9c760651 (stale) CRC-32 checksum value for chunk [0, 0x31da3): Mirror 1: 0x7962d397 Mirror 3: 0x7962d397 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31da3: skipped 000124 1779976031.398129 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31da3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31da3): Mirror 2: 0x9c760651 (stale) CRC-32 checksum value for chunk [0, 0x31da3): Mirror 1: 0x7962d397 Mirror 3: 0x7962d397 Mirror 4: 0x7962d397 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31da3: skipped 000125 1779976032.371094 MIRROR_RESYNC 4 mirrors 000126 1779976033.355355 punch from 0x01b6b0 to 0x01eaf5 (0x3446 bytes) 000128 1779976033.410784 mapwrite 0x0119e7 thru 0x01b4f0 (0x9b0a bytes) 000129 1779976033.533648 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31da3): Mirror 3: 0x7962d397 (stale) Mirror 4: 0x7962d397 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31da3): Mirror 3: 0x7962d397 (stale) Mirror 4: 0x7962d397 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31da3: skipped 000131 1779976034.146134 fallocate 0x01c74e thru 0x025697 (0x8f4a bytes) 000132 1779976034.152063 fallocate 0x01a43b thru 0x01a7f6 (0x3bc bytes) 000135 1779976034.166789 trunc from 0x031da3 to 0x02b4e1 (0x68c1 bytes) 000136 1779976034.273203 fallocate 0x03096b thru 0x032ade (0x2174 bytes) 000137 1779976034.288046 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b4e2): Mirror 4: 0x37c238b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4e2): Mirror 4: 0x37c238b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4e2: skipped 000138 1779976034.670726 mapwrite 0x00e203 thru 0x013852 (0x5650 bytes) 000139 1779976034.767465 mapwrite 0x027b7c thru 0x029d3a (0x21bf bytes) 000141 1779976034.836708 trunc from 0x02b4e2 to 0x021a53 (0x9a8e bytes) 000142 1779976034.868027 mapread 0x003d43 thru 0x01313d (0xf3fb bytes) 000143 1779976034.892931 fallocate 0x03243d thru 0x037b7d (0x5741 bytes) 000144 1779976034.899581 mapwrite 0x00dcac thru 0x01a6e1 (0xca36 bytes) 000145 1779976034.947086 punch from 0x00ce98 to 0x014419 (0x7582 bytes) 000146 1779976034.955656 trunc from 0x00899e to 0x011169 (0x87cc bytes) 000147 1779976034.964564 punch from 0x01feda to 0x021a53 (0x1b7a bytes) 000148 1779976034.973602 trunc from 0x021a54 to 0x01c901 (0x5152 bytes) 000149 1779976035.001850 punch from 0x013638 to 0x01a06a (0x6a33 bytes) 000150 1779976035.009683 punch from 0x01218f to 0x01af1d (0x8d8f bytes) 000151 1779976035.014515 read 0x009e90 thru 0x00e49b (0x460c bytes) 000152 1779976035.024495 write 0x02ad12 thru 0x02afd9 (0x2c8 bytes) 000153 1779976035.026623 write 0x034cc9 thru 0x03d36d (0x86a5 bytes) 000154 1779976035.030266 trunc from 0x03af5e to 0x03d36d (0x2410 bytes) 000155 1779976035.065484 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d36e): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d36e): Mirror 4: 0x771e09d0 (stale) CRC-32 checksum value for chunk [0, 0x3d36e): Mirror 1: 0x7a7df9ac Mirror 5: 0x7a7df9ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d36e: skipped 000156 1779976036.198156 mapwrite 0x02901a thru 0x030e61 (0x7e48 bytes) 000157 1779976036.352069 trunc from 0x03d36e to 0x001fb8 (0x3b3b5 bytes) 000158 1779976036.454714 fallocate 0x012e9b thru 0x01e178 (0xb2de bytes) 000159 1779976036.462129 mapread 0x00ff9d thru 0x017860 (0x78c4 bytes) 000160 1779976036.499935 write 0x029d6d thru 0x03909c (0xf330 bytes) 000161 1779976036.504069 trunc from 0x03909d to 0x02d7df (0xb8bd bytes) 000162 1779976036.539591 write 0x00adc4 thru 0x01891d (0xdb5a bytes) 000163 1779976036.557114 mapread 0x01cce7 thru 0x0292da (0xc5f4 bytes) 000164 1779976036.588856 trunc from 0x02d7e0 to 0x0217be (0xc021 bytes) 000165 1779976036.614513 trunc from 0x0217bf to 0x0006cb (0x210f3 bytes) 000166 1779976036.636434 trunc from 0x0006cc to 0x026ec1 (0x267f6 bytes) 000167 1779976036.657007 punch from 0x010397 to 0x01dd42 (0xd9ac bytes) 000168 1779976036.661618 punch from 0x002237 to 0x00b6ad (0x9477 bytes) 000169 1779976036.668372 fallocate 0x038e91 thru 0x03ffff (0x716f bytes) 000170 1779976036.678390 punch from 0x00cf06 to 0x00e2e6 (0x13e1 bytes) 000171 1779976036.692491 punch from 0x012558 to 0x020eee (0xe997 bytes) 000172 1779976036.719226 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 4: 0xa0cb2006 (stale) Mirror 5: 0x2eeba6c0 (stale) CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 1: 0x153a2093 Mirror 6: 0x153a2093 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ec2: skipped 000173 1779976037.749923 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 4: 0xa0cb2006 (stale) Mirror 5: 0x2eeba6c0 (stale) CRC-32 checksum value for chunk [0, 0x26ec2): Mirror 1: 0x153a2093 Mirror 6: 0x153a2093 Mirror 7: 0x153a2093 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ec2: skipped 000174 1779976038.711731 trunc from 0x00a641 to 0x00f0cc (0x4a8c bytes) 000175 1779976038.750100 write 0x0145d7 thru 0x01869d (0x40c7 bytes) 000177 1779976038.765689 read 0x009aba thru 0x00df19 (0x4460 bytes) 000178 1779976038.773172 mapwrite 0x0303d7 thru 0x032be5 (0x280f bytes) 000179 1779976038.818965 punch from 0x005a14 to 0x00aa44 (0x5031 bytes) 000180 1779976038.823969 trunc from 0x032be6 to 0x006acb (0x2c11a bytes) 000181 1779976038.844964 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x6acc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6acc): Mirror 4: 0x221fb8e0 (stale) Mirror 5: 0x221fb8e0 (stale) Mirror 6: 0x51e47446 (stale) Mirror 7: 0x51e47446 (stale) CRC-32 checksum value for chunk [0, 0x6acc): Mirror 1: 0x51e47446 Mirror 8: 0x51e47446 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6acc: skipped 000182 1779976039.586461 trunc from 0x006acc to 0x0322c5 (0x2b7fa bytes) 000183 1779976039.658128 read 0x004653 thru 0x00c922 (0x82d0 bytes) 000184 1779976039.681757 mapread 0x010a78 thru 0x01230f (0x1898 bytes) 000185 1779976039.717234 trunc from 0x02da07 to 0x02dc57 (0x251 bytes) 000186 1779976039.722521 mapread 0x01d92b thru 0x0203ae (0x2a84 bytes) 000187 1779976039.740076 punch from 0x01d4a3 to 0x02868c (0xb1ea bytes) 000188 1779976039.754911 mapwrite 0x0130a2 thru 0x01bfa5 (0x8f04 bytes) 000189 1779976039.807395 trunc from 0x006a6b to 0x009a87 (0x301d bytes) 000190 1779976039.814846 trunc from 0x007075 to 0x00ad87 (0x3d13 bytes) 000191 1779976039.826305 write 0x03a434 thru 0x03ffff (0x5bcc bytes) 000192 1779976039.834662 fallocate 0x038f2b thru 0x03fa0b (0x6ae1 bytes) 000193 1779976039.840966 punch from 0x030fa8 to 0x03d299 (0xc2f2 bytes) 000194 1779976039.863896 write 0x018fbc thru 0x02158c (0x85d1 bytes) 000195 1779976039.876694 write 0x03e03d thru 0x03ffff (0x1fc3 bytes) 000196 1779976039.884123 punch from 0x03f14b to 0x03ffff (0xeb5 bytes) 000197 1779976039.931178 write 0x031724 thru 0x03d25f (0xbb3c bytes) 000198 1779976039.976368 mapread 0x03b89d thru 0x03ffff (0x4763 bytes) 000199 1779976040.007962 MIRROR_RESYNC 6 mirrors 000200 1779976041.397880 trunc from 0x040000 to 0x03b7ad (0x4852 bytes) 000201 1779976041.443594 trunc from 0x03b7ae to 0x03b836 (0x89 bytes) 000202 1779976041.472104 mapread 0x017ebe thru 0x02243c (0xa57f bytes) 000203 1779976041.501469 read 0x01265f thru 0x014e93 (0x2835 bytes) 000204 1779976041.502191 write 0x010637 thru 0x017fbb (0x7985 bytes) 000205 1779976041.517333 trunc from 0x004a51 to 0x00ea3b (0x9feb bytes) 000206 1779976041.525524 punch from 0x002cd5 to 0x0090b3 (0x63df bytes) 000208 1779976041.532759 punch from 0x000c3f to 0x008ea8 (0x826a bytes) 000209 1779976041.537516 punch from 0x007b96 to 0x00f33c (0x77a7 bytes) 000210 1779976041.554142 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b837): Mirror 4: 0x18478996 (stale) Mirror 5: 0x18478996 (stale) Mirror 6: 0x18478996 (stale) Mirror 7: 0x18478996 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b837): Mirror 4: 0x18478996 (stale) Mirror 5: 0x18478996 (stale) Mirror 6: 0x18478996 (stale) Mirror 7: 0x18478996 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b837: skipped 000212 1779976041.967248 mapwrite 0x025aac thru 0x02626a (0x7bf bytes) 000213 1779976041.987476 write 0x02f8e7 thru 0x03219b (0x28b5 bytes) 000214 1779976042.005085 punch from 0x021044 to 0x029687 (0x8644 bytes) 000215 1779976042.010228 trunc from 0x00bf7a to 0x012489 (0x6510 bytes) 000216 1779976042.015011 trunc from 0x03b837 to 0x032b13 (0x8d23 bytes) 000217 1779976042.029118 punch from 0x029aa7 to 0x032b13 (0x906d bytes) 000218 1779976042.040490 mapwrite 0x001fcf thru 0x00e5f5 (0xc627 bytes) 000220 1779976042.076394 mapwrite 0x03a195 thru 0x03ffff (0x5e6b bytes) 000222 1779976042.154027 fallocate 0x002557 thru 0x010492 (0xdf3c bytes) 000223 1779976042.159804 trunc from 0x012d39 to 0x021a39 (0xed01 bytes) 000224 1779976042.170353 trunc from 0x040000 to 0x028276 (0x17d89 bytes) 000225 1779976042.192614 mapwrite 0x0271c3 thru 0x029a2f (0x286d bytes) 000226 1779976042.242845 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29a30): Mirror 4: 0xd0f0d4c7 (stale) Mirror 5: 0xd0f0d4c7 (stale) Mirror 6: 0xd0f0d4c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29a30): Mirror 4: 0xd0f0d4c7 (stale) Mirror 5: 0xd0f0d4c7 (stale) Mirror 6: 0xd0f0d4c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29a30: skipped 000227 1779976042.678479 fallocate 0x0164cf thru 0x0165e6 (0x118 bytes) 000228 1779976042.694301 write 0x02e081 thru 0x0365b4 (0x8534 bytes) 000229 1779976042.705478 write 0x03fe19 thru 0x03ffff (0x1e7 bytes) 000230 1779976042.706621 write 0x0070a0 thru 0x010775 (0x96d6 bytes) 000231 1779976042.739000 mapread 0x017378 thru 0x0269df (0xf668 bytes) 000232 1779976042.802564 MIRROR_RESYNC 4 mirrors 000233 1779976044.653420 mapwrite 0x03e753 thru 0x03ffff (0x18ad bytes) 000234 1779976044.772961 mapwrite 0x03fe0f thru 0x03ffff (0x1f1 bytes) 000235 1779976044.801250 trunc from 0x040000 to 0x025b2c (0x1a4d3 bytes) 000236 1779976044.854984 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: 0x873e9c95 (stale) Mirror 5: 0x873e9c95 (stale) Mirror 6: 0x873e9c95 (stale) CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 1: 0x873e9c95 Mirror 7: 0x873e9c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b2d: skipped 000237 1779976045.727447 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: 0x873e9c95 (stale) Mirror 5: 0x873e9c95 (stale) Mirror 6: 0x873e9c95 (stale) CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 1: 0x873e9c95 Mirror 7: 0x873e9c95 Mirror 8: 0x873e9c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b2d: skipped 000238 1779976046.929253 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 4: 0x873e9c95 (stale) Mirror 5: 0x873e9c95 (stale) Mirror 6: 0x873e9c95 (stale) CRC-32 checksum value for chunk [0, 0x25b2d): Mirror 1: 0x873e9c95 Mirror 7: 0x873e9c95 Mirror 8: 0x873e9c95 Mirror 9: 0x873e9c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b2d: skipped 000239 1779976047.905567 fallocate 0x01b274 thru 0x0267e9 (0xb576 bytes) 000240 1779976047.945608 fallocate 0x031c6d thru 0x032856 (0xbea bytes) 000241 1779976047.950910 mapread 0x01c4b8 thru 0x020fed (0x4b36 bytes) 000242 1779976047.967451 read 0x0236d0 thru 0x025b2c (0x245d bytes) 000243 1779976047.968282 punch from 0x00f807 to 0x016150 (0x694a bytes) 000244 1779976047.973421 trunc from 0x025b2d to 0x0144ee (0x1163e bytes) 000245 1779976048.005475 mapwrite 0x03af7e thru 0x03ffff (0x5082 bytes) 000246 1779976048.085296 fallocate 0x0045c1 thru 0x005fa3 (0x19e3 bytes) 000247 1779976048.099040 read 0x00d12f thru 0x017082 (0x9f54 bytes) 000248 1779976048.127876 punch from 0x039442 to 0x03a58e (0x114d bytes) 000249 1779976048.137620 write 0x03dd53 thru 0x03ffff (0x22ad bytes) 000250 1779976048.138762 trunc from 0x040000 to 0x00414d (0x3beb2 bytes) 000251 1779976048.174678 fallocate 0x01d0b4 thru 0x01d31d (0x26a bytes) 000252 1779976048.190093 fallocate 0x03721a thru 0x03ffff (0x8de6 bytes) 000253 1779976048.206192 punch from 0x02861d to 0x0331c9 (0xabad bytes) 000255 1779976048.216146 read 0x005605 thru 0x0076e9 (0x20e5 bytes) 000256 1779976048.229950 trunc from 0x040000 to 0x00810d (0x37ef2 bytes) 000258 1779976048.243532 fallocate 0x00a60d thru 0x013091 (0x8a85 bytes) 000259 1779976048.249256 trunc from 0x007657 to 0x00810d (0xab7 bytes) 000260 1779976048.257708 read 0x001728 thru 0x003376 (0x1c4f bytes) 000261 1779976048.258708 punch from 0x0076d8 to 0x00810d (0xa36 bytes) 000262 1779976048.273167 fallocate 0x01001f thru 0x01d7ff (0xd7e1 bytes) 000263 1779976048.279058 punch from 0x000161 to 0x003e17 (0x3cb7 bytes) 000264 1779976048.284263 trunc from 0x01d800 to 0x0195b1 (0x424e bytes) 000265 1779976048.299555 mapwrite 0x0287f3 thru 0x038106 (0xf914 bytes) 000266 1779976048.373535 write 0x007ac9 thru 0x013615 (0xbb4d bytes) 000267 1779976048.377085 write 0x00d876 thru 0x015853 (0x7fde bytes) 000268 1779976048.379603 trunc from 0x033a45 to 0x038106 (0x46c2 bytes) 000269 1779976048.387585 write 0x02cf6b thru 0x033851 (0x68e7 bytes) 000270 1779976048.395460 write 0x003283 thru 0x011c90 (0xea0e bytes) 000271 1779976048.398275 punch from 0x004616 to 0x0124fd (0xdee8 bytes) 000272 1779976048.425088 trunc from 0x038107 to 0x02b4a1 (0xcc65 bytes) 000273 1779976048.445156 punch from 0x00f7c5 to 0x016547 (0x6d83 bytes) 000274 1779976048.453214 trunc from 0x0030cb to 0x0045ae (0x14e4 bytes) 000275 1779976048.459817 trunc from 0x02b4a2 to 0x02e2f5 (0x2e54 bytes) 000276 1779976048.477242 punch from 0x012ec5 to 0x0163a1 (0x34dd bytes) 000277 1779976048.483187 trunc from 0x029dda to 0x02e2f5 (0x451c bytes) 000278 1779976048.489143 read 0x029267 thru 0x02e2f5 (0x508f bytes) 000279 1779976048.512267 read 0x0077a8 thru 0x00c996 (0x51ef bytes) 000280 1779976048.513472 mapread 0x0146c6 thru 0x01ad4b (0x6686 bytes) 000281 1779976048.515568 trunc from 0x004e13 to 0x0109a0 (0xbb8e bytes) 000282 1779976048.523264 read 0x007f8d thru 0x010120 (0x8194 bytes) 000283 1779976048.544475 fallocate 0x001ce3 thru 0x00523f (0x355d bytes) 000284 1779976048.550555 read 0x0128c4 thru 0x017b31 (0x526e bytes) 000285 1779976048.556397 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e2f6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2f6): Mirror 4: 0x494f6437 (stale) Mirror 5: 0x494f6437 (stale) Mirror 6: 0x494f6437 (stale) Mirror 7: 0xaabbaf46 (stale) Mirror 8: 0xaabbaf46 (stale) Mirror 9: 0xaabbaf46 (stale) CRC-32 checksum value for chunk [0, 0x2e2f6): Mirror 1: 0xac14ccc1 Mirror 10: 0xac14ccc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2f6: skipped 000286 1779976049.695438 write 0x0025de thru 0x004992 (0x23b5 bytes) 000287 1779976049.749047 write 0x02c71c thru 0x035206 (0x8aeb bytes) 000288 1779976049.761469 write 0x031cb0 thru 0x032ef5 (0x1246 bytes) 000289 1779976049.762743 write 0x03b249 thru 0x03e189 (0x2f41 bytes) 000291 1779976049.768740 write 0x005829 thru 0x01192a (0xc102 bytes) 000292 1779976049.791152 read 0x03d3ce thru 0x03e189 (0xdbc bytes) 000293 1779976049.792161 punch from 0x01f467 to 0x02aee8 (0xba82 bytes) 000294 1779976049.802178 punch from 0x0090a1 to 0x017b26 (0xea86 bytes) 000295 1779976049.824097 punch from 0x00963c to 0x018d0e (0xf6d3 bytes) 000296 1779976049.840847 trunc from 0x02aef8 to 0x02e081 (0x318a bytes) 000297 1779976049.851517 mapwrite 0x00f490 thru 0x01eec7 (0xfa38 bytes) 000298 1779976049.913872 mapread 0x0107a2 thru 0x01c180 (0xb9df bytes) 000299 1779976049.916080 mapread 0x02e646 thru 0x03c33d (0xdcf8 bytes) 000300 1779976049.943220 trunc from 0x03e18a to 0x038dc5 (0x53c4 bytes) 000301 1779976049.973055 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38dc6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38dc6): Mirror 4: 0xc1c7d13d (stale) Mirror 5: 0xc1c7d13d (stale) Mirror 6: 0xc1c7d13d (stale) Mirror 7: 0x35060350 (stale) Mirror 8: 0x35060350 (stale) Mirror 9: 0x35060350 (stale) Mirror 10: 0xa75afd35 (stale) CRC-32 checksum value for chunk [0, 0x38dc6): Mirror 1: 0x11797802 Mirror 11: 0x11797802 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38dc6: skipped 000303 1779976051.348790 mapwrite 0x00022e thru 0x009380 (0x9153 bytes) 000304 1779976051.528563 punch from 0x02c33a to 0x037443 (0xb10a bytes) 000305 1779976051.553554 MIRROR_RESYNC 9 mirrors 000306 1779976053.845837 mapread 0x02660d thru 0x03123b (0xac2f bytes) 000307 1779976053.891469 trunc from 0x02d2b9 to 0x036dd2 (0x9b1a bytes) 000308 1779976053.934208 punch from 0x027549 to 0x0286f2 (0x11aa bytes) 000309 1779976053.940223 read 0x012606 thru 0x01da80 (0xb47b bytes) 000310 1779976053.951063 read 0x025250 thru 0x026ff4 (0x1da5 bytes) 000311 1779976053.974485 trunc from 0x03313e to 0x037967 (0x482a bytes) 000313 1779976053.981450 mapread 0x0218c7 thru 0x030661 (0xed9b bytes) 000314 1779976053.984281 write 0x038353 thru 0x03ffff (0x7cad bytes) 000315 1779976053.986876 read 0x01a554 thru 0x01b64b (0x10f8 bytes) 000316 1779976053.988100 write 0x0058fe thru 0x0080eb (0x27ee bytes) 000317 1779976053.994170 MIRROR_RESYNC 9 mirrors 000318 1779976056.001721 punch from 0x039bd1 to 0x03b241 (0x1671 bytes) 000319 1779976056.033014 mapread 0x03d21a thru 0x03ffff (0x2de6 bytes) 000320 1779976056.070175 mapwrite 0x03d52e thru 0x03ffff (0x2ad2 bytes) 000321 1779976056.094874 write 0x0305ee thru 0x03aada (0xa4ed bytes) 000322 1779976056.107552 read 0x01d723 thru 0x022dfa (0x56d8 bytes) 000324 1779976056.117221 fallocate 0x026dcf thru 0x02c016 (0x5248 bytes) 000325 1779976056.126008 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf7c304c4 (stale) Mirror 5: 0xf7c304c4 (stale) Mirror 6: 0xf7c304c4 (stale) Mirror 7: 0xf7c304c4 (stale) Mirror 8: 0xf7c304c4 (stale) Mirror 9: 0xf7c304c4 (stale) Mirror 10: 0xf7c304c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf7c304c4 (stale) Mirror 5: 0xf7c304c4 (stale) Mirror 6: 0xf7c304c4 (stale) Mirror 7: 0xf7c304c4 (stale) Mirror 8: 0xf7c304c4 (stale) Mirror 9: 0xf7c304c4 (stale) Mirror 10: 0xf7c304c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1779976056.670588 mapread 0x022897 thru 0x02e205 (0xb96f bytes) 000327 1779976056.698910 trunc from 0x014882 to 0x02164b (0xcdca bytes) 000328 1779976056.703356 trunc from 0x040000 to 0x0031ca (0x3ce35 bytes) 000329 1779976056.726225 trunc from 0x000103 to 0x0031ca (0x30c8 bytes) 000331 1779976056.732824 mapwrite 0x03bbaf thru 0x03ffff (0x4451 bytes) 000332 1779976056.772061 mapwrite 0x009886 thru 0x00f8f6 (0x6071 bytes) 000333 1779976056.794436 punch from 0x020597 to 0x02bc78 (0xb6e2 bytes) 000334 1779976056.798788 trunc from 0x040000 to 0x00b2c0 (0x34d3f bytes) 000335 1779976056.811429 read 0x0032c9 thru 0x00b2c0 (0x7ff8 bytes) 000336 1779976056.812168 trunc from 0x0010c7 to 0x003451 (0x238b bytes) 000337 1779976056.818020 punch from 0x009a70 to 0x00b2c0 (0x1851 bytes) 000338 1779976056.822929 mapwrite 0x032afe thru 0x03b133 (0x8636 bytes) 000339 1779976056.864012 read 0x02a991 thru 0x02b51e (0xb8e bytes) 000340 1779976056.874919 write 0x0068a5 thru 0x007b0a (0x1266 bytes) 000341 1779976056.876366 trunc from 0x03b134 to 0x032b0e (0x8625 bytes) 000342 1779976056.892141 read 0x0086f8 thru 0x011a73 (0x937c bytes) 000343 1779976056.893174 trunc from 0x032b0f to 0x01242d (0x206e1 bytes) 000344 1779976056.911056 trunc from 0x011dee to 0x01242d (0x640 bytes) 000345 1779976056.915661 trunc from 0x01242e to 0x024538 (0x1210b bytes) 000346 1779976056.927790 trunc from 0x019f65 to 0x024538 (0xa5d4 bytes) 000347 1779976056.938581 punch from 0x009d68 to 0x00fef0 (0x6189 bytes) 000348 1779976056.943326 trunc from 0x005e30 to 0x00a9de (0x4baf bytes) 000349 1779976056.960338 mapread 0x00ffe2 thru 0x01d273 (0xd292 bytes) 000350 1779976056.973840 mapwrite 0x03a304 thru 0x03ab42 (0x83f bytes) 000351 1779976057.006464 trunc from 0x03ab43 to 0x020647 (0x1a4fb bytes) 000352 1779976057.023407 write 0x02be45 thru 0x02cac1 (0xc7d bytes) 000353 1779976057.033855 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cac2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cac2): Mirror 4: 0x9ff26fff (stale) Mirror 5: 0x9ff26fff (stale) Mirror 6: 0x9ff26fff (stale) Mirror 7: 0x9ff26fff (stale) Mirror 8: 0x9ff26fff (stale) Mirror 9: 0x9ff26fff (stale) Mirror 10: 0x9ff26fff (stale) CRC-32 checksum value for chunk [0, 0x2cac2): Mirror 1: 0xf0a63269 Mirror 11: 0xf0a63269 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cac2: skipped 000354 1779976057.942225 punch from 0x021371 to 0x02b12b (0x9dbb bytes) 000355 1779976057.978666 trunc from 0x015bcd to 0x022dc2 (0xd1f6 bytes) 000357 1779976057.984601 fallocate 0x02897c thru 0x0304cf (0x7b54 bytes) 000358 1779976057.990096 mapread 0x003f3c thru 0x00cd46 (0x8e0b bytes) 000359 1779976058.014675 MIRROR_RESYNC 9 mirrors 000360 1779976060.426078 trunc from 0x0304d0 to 0x000b50 (0x2f97f bytes) 000361 1779976060.519108 trunc from 0x000b51 to 0x006c2e (0x60de bytes) 000362 1779976060.562297 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6c2f): Mirror 4: 0x28795135 (stale) Mirror 5: 0x28795135 (stale) Mirror 7: 0x28795135 (stale) Mirror 8: 0x28795135 (stale) Mirror 9: 0x28795135 (stale) Mirror 10: 0x28795135 (stale) Mirror 11: 0x28795135 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6c2f): Mirror 4: 0x28795135 (stale) Mirror 5: 0x28795135 (stale) Mirror 7: 0x28795135 (stale) Mirror 8: 0x28795135 (stale) Mirror 9: 0x28795135 (stale) Mirror 10: 0x28795135 (stale) Mirror 11: 0x28795135 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6c2f: skipped 000363 1779976061.125485 punch from 0x0027f6 to 0x004b9c (0x23a7 bytes) 000364 1779976061.131220 punch from 0x005f5b to 0x006c2e (0xcd4 bytes) 000365 1779976061.136237 mapread 0x0021b9 thru 0x006c2e (0x4a76 bytes) 000366 1779976061.155119 fallocate 0x03e7b9 thru 0x03ffff (0x1847 bytes) 000367 1779976061.160230 punch from 0x0064b9 to 0x006c2e (0x776 bytes) 000368 1779976061.169883 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6c2f): Mirror 5: 0x28795135 (stale) Mirror 7: 0x28795135 (stale) Mirror 8: 0x28795135 (stale) Mirror 9: 0x28795135 (stale) Mirror 10: 0x28795135 (stale) Mirror 11: 0x28795135 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6c2f): Mirror 5: 0x28795135 (stale) Mirror 7: 0x28795135 (stale) Mirror 8: 0x28795135 (stale) Mirror 9: 0x28795135 (stale) Mirror 10: 0x28795135 (stale) Mirror 11: 0x28795135 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6c2f: skipped 000369 1779976061.589304 punch from 0x005cd7 to 0x0062f2 (0x61c bytes) 000370 1779976061.598934 read 0x000b65 thru 0x006c2e (0x60ca bytes) 000371 1779976061.611382 punch from 0x000b95 to 0x006c2e (0x609a bytes) 000372 1779976061.618167 write 0x00aad0 thru 0x01730a (0xc83b bytes) 000373 1779976061.628239 fallocate 0x024461 thru 0x02b328 (0x6ec8 bytes) 000374 1779976061.635262 trunc from 0x011869 to 0x015eea (0x4682 bytes) 000375 1779976061.652491 read 0x007f51 thru 0x016f1f (0xefcf bytes) 000376 1779976061.662143 mapwrite 0x01e07e thru 0x0221a0 (0x4123 bytes) 000377 1779976061.711432 trunc from 0x0221a1 to 0x0126b7 (0xfae9 bytes) 000378 1779976061.729485 fallocate 0x02ceaf thru 0x033a04 (0x6b56 bytes) 000380 1779976061.738426 write 0x014fa5 thru 0x01dd33 (0x8d8f bytes) 000381 1779976061.753973 punch from 0x0096c4 to 0x017fcf (0xe90c bytes) 000382 1779976061.771048 mapwrite 0x038656 thru 0x03ffff (0x79aa bytes) 000383 1779976061.817443 punch from 0x014bfb to 0x016523 (0x1929 bytes) 000384 1779976061.823365 mapwrite 0x0063a3 thru 0x00df3c (0x7b9a bytes) 000385 1779976061.874270 write 0x02efba thru 0x03e1bb (0xf202 bytes) 000386 1779976061.883358 MIRROR_RESYNC 7 mirrors 000387 1779976063.921149 mapread 0x033baf thru 0x03ffff (0xc451 bytes) 000389 1779976063.946607 fallocate 0x021451 thru 0x024a56 (0x3606 bytes) 000390 1779976063.978439 MIRROR_RESYNC 7 mirrors 000391 1779976065.592986 trunc from 0x03cd5f to 0x03ffff (0x32a1 bytes) 000392 1779976065.619171 punch from 0x006ad9 to 0x00d4de (0x6a06 bytes) 000393 1779976065.624380 fallocate 0x039b22 thru 0x03ffff (0x64de bytes) 000394 1779976065.629226 write 0x0073fe thru 0x010f4a (0x9b4d bytes) 000396 1779976065.640965 write 0x00e5dd thru 0x01ce14 (0xe838 bytes) 000397 1779976065.647034 trunc from 0x01e634 to 0x02687f (0x824c bytes) 000398 1779976065.653369 punch from 0x030d16 to 0x038917 (0x7c02 bytes) 000399 1779976065.660608 write 0x0332e6 thru 0x03ca9b (0x97b6 bytes) 000401 1779976065.676725 MIRROR_RESYNC 7 mirrors 000402 1779976067.904078 MIRROR_EXTEND to 8 mirrors 000403 1779976068.755840 read 0x025fd7 thru 0x028c0e (0x2c38 bytes) 000404 1779976068.763239 trunc from 0x040000 to 0x02d1e9 (0x12e16 bytes) 000405 1779976068.824070 MIRROR_RESYNC 8 mirrors 000406 1779976070.139059 fallocate 0x000118 thru 0x01007f (0xff68 bytes) 000407 1779976070.206385 mapwrite 0x0054fb thru 0x00c53a (0x7040 bytes) 000409 1779976070.288873 mapread 0x02cbdb thru 0x02d1e9 (0x60f bytes) 000410 1779976070.311689 write 0x00edde thru 0x01190a (0x2b2d bytes) 000411 1779976070.329988 mapwrite 0x0283a2 thru 0x02a70e (0x236d bytes) 000412 1779976070.369266 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2d1ea): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1ea): Mirror 5: 0x7c02390 (stale) Mirror 7: 0x7c02390 (stale) Mirror 8: 0x7c02390 (stale) Mirror 9: 0x7c02390 (stale) Mirror 10: 0x7c02390 (stale) Mirror 11: 0x7c02390 (stale) Mirror 12: 0x7c02390 (stale) CRC-32 checksum value for chunk [0, 0x2d1ea): Mirror 1: 0xe5133f46 Mirror 13: 0xe5133f46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1ea: skipped 000414 1779976071.774432 trunc from 0x00cc89 to 0x01a522 (0xd89a bytes) 000415 1779976071.820560 trunc from 0x02d1ea to 0x02e55a (0x1371 bytes) 000416 1779976071.917249 trunc from 0x00e870 to 0x01267a (0x3e0b bytes) 000417 1779976071.929896 mapwrite 0x0330f9 thru 0x03c0ad (0x8fb5 bytes) 000418 1779976072.052366 read 0x03a7f8 thru 0x03c0ad (0x18b6 bytes) 000419 1779976072.053582 punch from 0x0004c3 to 0x00133b (0xe79 bytes) 000420 1779976072.067294 trunc from 0x0154e0 to 0x015cd6 (0x7f7 bytes) 000421 1779976072.078898 mapread 0x03440a thru 0x03c0ad (0x7ca4 bytes) 000422 1779976072.084981 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c0ae): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0ae): Mirror 5: 0x3a83b695 (stale) Mirror 7: 0x3a83b695 (stale) Mirror 8: 0x3a83b695 (stale) Mirror 9: 0x3a83b695 (stale) Mirror 10: 0x3a83b695 (stale) Mirror 11: 0x3a83b695 (stale) Mirror 12: 0x3a83b695 (stale) Mirror 13: 0x91ab285c (stale) CRC-32 checksum value for chunk [0, 0x3c0ae): Mirror 1: 0x7785d877 Mirror 14: 0x7785d877 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0ae: skipped 000423 1779976073.490598 fallocate 0x00f256 thru 0x01f214 (0xffbf bytes) 000424 1779976073.528353 write 0x030905 thru 0x03c68b (0xbd87 bytes) 000425 1779976073.545786 mapread 0x0302fe thru 0x031c48 (0x194b bytes) 000426 1779976073.547805 fallocate 0x004dbd thru 0x008de1 (0x4025 bytes) 000427 1779976073.557836 punch from 0x00a269 to 0x015425 (0xb1bd bytes) 000428 1779976073.576161 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c68c): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c68c): Mirror 5: 0x2a943027 (stale) Mirror 7: 0x2a943027 (stale) Mirror 8: 0x2a943027 (stale) Mirror 9: 0x2a943027 (stale) Mirror 10: 0x2a943027 (stale) Mirror 11: 0x2a943027 (stale) Mirror 12: 0x2a943027 (stale) Mirror 13: 0x473206e8 (stale) Mirror 14: 0xd81e8aa8 (stale) CRC-32 checksum value for chunk [0, 0x3c68c): Mirror 1: 0x815433eb Mirror 15: 0x815433eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c68c: skipped 000429 1779976075.021845 punch from 0x01e006 to 0x021c09 (0x3c04 bytes) 000430 1779976075.069584 write 0x0375d0 thru 0x03ffff (0x8a30 bytes) 000431 1779976075.079768 trunc from 0x040000 to 0x036537 (0x9ac8 bytes) 000432 1779976075.102751 fallocate 0x016bcf thru 0x01e6d5 (0x7b07 bytes) 000433 1779976075.107462 mapread 0x0146b3 thru 0x018c6e (0x45bc bytes) 000434 1779976075.119353 mapwrite 0x01b3e4 thru 0x0286de (0xd2fb bytes) 000435 1779976075.174496 read 0x01e9b0 thru 0x022297 (0x38e8 bytes) 000436 1779976075.175227 trunc from 0x036538 to 0x01c026 (0x1a511 bytes) 000437 1779976075.196410 mapread 0x0188e7 thru 0x01c026 (0x3740 bytes) 000439 1779976075.198997 mapwrite 0x036009 thru 0x03ffff (0x9ff7 bytes) 000440 1779976075.282514 MIRROR_RESYNC 11 mirrors 000441 1779976078.054239 trunc from 0x029ddc to 0x02e8ae (0x4ad3 bytes) 000442 1779976078.107874 read 0x03ff48 thru 0x03ffff (0xb8 bytes) 000443 1779976078.117391 fallocate 0x03ccae thru 0x03d715 (0xa68 bytes) 000444 1779976078.124476 write 0x005efa thru 0x00997d (0x3a84 bytes) 000445 1779976078.134368 fallocate 0x025e9c thru 0x02e332 (0x8497 bytes) 000446 1779976078.143202 punch from 0x03b6d7 to 0x03ffff (0x4929 bytes) 000447 1779976078.149324 mapwrite 0x0343ba thru 0x03c4f5 (0x813c bytes) 000448 1779976078.202955 trunc from 0x0063db to 0x00eabd (0x86e3 bytes) 000449 1779976078.209649 trunc from 0x00da25 to 0x012a9f (0x507b bytes) 000450 1779976078.214487 mapwrite 0x01f477 thru 0x024ce1 (0x586b bytes) 000451 1779976078.264994 punch from 0x01bcb2 to 0x02055f (0x48ae bytes) 000452 1779976078.276522 trunc from 0x040000 to 0x02e95a (0x116a5 bytes) 000453 1779976078.309467 fallocate 0x0003cf thru 0x00e793 (0xe3c5 bytes) 000455 1779976078.316579 write 0x035c71 thru 0x03ffff (0xa38f bytes) 000456 1779976078.318632 fallocate 0x0321d9 thru 0x03ffff (0xde27 bytes) 000457 1779976078.326900 read 0x03f9db thru 0x03ffff (0x625 bytes) 000458 1779976078.327560 mapwrite 0x01e2f3 thru 0x02d61a (0xf328 bytes) 000459 1779976078.388105 fallocate 0x036978 thru 0x03ffff (0x9688 bytes) 000460 1779976078.393838 mapwrite 0x021b6d thru 0x02e649 (0xcadd bytes) 000462 1779976078.424280 mapread 0x02f462 thru 0x039419 (0x9fb8 bytes) 000463 1779976078.425590 punch from 0x02bb80 to 0x039cb1 (0xe132 bytes) 000464 1779976078.432947 write 0x020a78 thru 0x021cf6 (0x127f bytes) 000465 1779976078.434006 trunc from 0x00ce0c to 0x010f4e (0x4143 bytes) 000466 1779976078.448629 mapwrite 0x00590a thru 0x006552 (0xc49 bytes) 000467 1779976078.465709 fallocate 0x039649 thru 0x03ffff (0x69b7 bytes) 000468 1779976078.477202 mapwrite 0x010e46 thru 0x012608 (0x17c3 bytes) 000469 1779976078.503106 trunc from 0x010e14 to 0x011bc9 (0xdb6 bytes) 000470 1779976078.515069 mapread 0x02cbdf thru 0x030fd4 (0x43f6 bytes) 000471 1779976078.544979 trunc from 0x01cc0d to 0x02154c (0x4940 bytes) 000472 1779976078.551153 mapread 0x02dba7 thru 0x0345b6 (0x6a10 bytes) 000473 1779976078.586532 trunc from 0x040000 to 0x001515 (0x3eaea bytes) 000474 1779976078.617710 punch from 0x000577 to 0x001515 (0xf9f bytes) 000475 1779976078.622781 mapread 0x000745 thru 0x001515 (0xdd1 bytes) 000476 1779976078.639477 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1516): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1516): Mirror 5: 0x92e5c751 (stale) Mirror 7: 0x92e5c751 (stale) Mirror 8: 0x92e5c751 (stale) Mirror 9: 0x92e5c751 (stale) Mirror 10: 0x92e5c751 (stale) Mirror 11: 0x92e5c751 (stale) Mirror 12: 0x92e5c751 (stale) Mirror 13: 0x92e5c751 (stale) Mirror 14: 0x92e5c751 (stale) Mirror 15: 0x92e5c751 (stale) CRC-32 checksum value for chunk [0, 0x1516): Mirror 1: 0x92e5c751 Mirror 16: 0x92e5c751 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1516: skipped 000477 1779976079.453757 fallocate 0x010915 thru 0x01e6de (0xddca bytes) 000478 1779976079.516209 read 0x00bc90 thru 0x016779 (0xaaea bytes) 000479 1779976079.540786 trunc from 0x01e6df to 0x025d42 (0x7664 bytes) 000480 1779976079.579107 write 0x00484f thru 0x004970 (0x122 bytes) 000481 1779976079.587009 trunc from 0x00b867 to 0x016bd0 (0xb36a bytes) 000483 1779976079.594792 mapwrite 0x027271 thru 0x034013 (0xcda3 bytes) 000484 1779976079.686019 mapread 0x00c149 thru 0x00e2df (0x2197 bytes) 000485 1779976079.706680 punch from 0x0061c3 to 0x00dac3 (0x7901 bytes) 000486 1779976079.715893 trunc from 0x02eb74 to 0x033b3a (0x4fc7 bytes) 000487 1779976079.725213 write 0x01523a thru 0x0240ba (0xee81 bytes) 000488 1779976079.746555 read 0x00aafe thru 0x0114f5 (0x69f8 bytes) 000489 1779976079.758144 mapread 0x0276e8 thru 0x034013 (0xc92c bytes) 000490 1779976079.759462 write 0x01dbc1 thru 0x02060e (0x2a4e bytes) 000491 1779976079.763749 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x34014): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34014): Mirror 5: 0x1d0c305 (stale) Mirror 7: 0x1d0c305 (stale) Mirror 8: 0x1d0c305 (stale) Mirror 9: 0x1d0c305 (stale) Mirror 10: 0x1d0c305 (stale) Mirror 11: 0x1d0c305 (stale) Mirror 12: 0x1d0c305 (stale) Mirror 13: 0x1d0c305 (stale) Mirror 14: 0x1d0c305 (stale) Mirror 15: 0x1d0c305 (stale) Mirror 16: 0x7037ca51 (stale) CRC-32 checksum value for chunk [0, 0x34014): Mirror 1: 0x2c8f587c Mirror 17: 0x2c8f587c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34014: skipped 000492 1779976080.792601 fallocate 0x000a2e thru 0x01058f (0xfb62 bytes) 000493 1779976080.866357 mapread 0x01d855 thru 0x0282fe (0xaaaa bytes) 000494 1779976080.931211 read 0x02ca9f thru 0x032a62 (0x5fc4 bytes) 000495 1779976080.938121 fallocate 0x01f4ac thru 0x02bc9e (0xc7f3 bytes) 000497 1779976080.947246 read 0x02b149 thru 0x034013 (0x8ecb bytes) 000498 1779976080.952448 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34014): Mirror 5: 0x1d0c305 (stale) Mirror 7: 0x1d0c305 (stale) Mirror 8: 0x1d0c305 (stale) Mirror 9: 0x1d0c305 (stale) Mirror 10: 0x1d0c305 (stale) Mirror 12: 0x1d0c305 (stale) Mirror 13: 0x1d0c305 (stale) Mirror 14: 0x1d0c305 (stale) Mirror 15: 0x1d0c305 (stale) Mirror 16: 0x7037ca51 (stale) Mirror 17: 0x2c8f587c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34014): Mirror 5: 0x1d0c305 (stale) Mirror 7: 0x1d0c305 (stale) Mirror 8: 0x1d0c305 (stale) Mirror 9: 0x1d0c305 (stale) Mirror 10: 0x1d0c305 (stale) Mirror 12: 0x1d0c305 (stale) Mirror 13: 0x1d0c305 (stale) Mirror 14: 0x1d0c305 (stale) Mirror 15: 0x1d0c305 (stale) Mirror 16: 0x7037ca51 (stale) Mirror 17: 0x2c8f587c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34014: skipped 000499 1779976081.550747 punch from 0x027ff1 to 0x03360e (0xb61e bytes) 000500 1779976081.562159 MIRROR_RESYNC 12 mirrors 000501 1779976085.237241 fallocate 0x01ec54 thru 0x020fa1 (0x234e bytes) 000502 1779976085.276600 read 0x01e9d2 thru 0x02d427 (0xea56 bytes) 000503 1779976085.288920 MIRROR_RESYNC 12 mirrors 000505 1779976088.183378 mapwrite 0x02ed05 thru 0x02f993 (0xc8f bytes) 000507 1779976088.274257 mapread 0x026876 thru 0x026984 (0x10f bytes) 000508 1779976088.283716 punch from 0x02b7a3 to 0x031dc5 (0x6623 bytes) 000510 1779976088.291207 write 0x01977b thru 0x01b466 (0x1cec bytes) 000511 1779976088.307193 write 0x0396d2 thru 0x03ffff (0x692e bytes) 000512 1779976088.310037 read 0x02b7ad thru 0x035130 (0x9984 bytes) 000513 1779976088.323962 mapread 0x00a575 thru 0x019a21 (0xf4ad bytes) 000514 1779976088.336206 read 0x02bc62 thru 0x02ea93 (0x2e32 bytes) 000515 1779976088.337115 trunc from 0x00d569 to 0x0158c1 (0x8359 bytes) 000516 1779976088.343284 write 0x01e96e thru 0x024b91 (0x6224 bytes) 000517 1779976088.344550 mapread 0x013d67 thru 0x018adb (0x4d75 bytes) 000518 1779976088.355602 trunc from 0x011bd3 to 0x01b2b2 (0x96e0 bytes) 000519 1779976088.370179 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf5a9393c (stale) Mirror 7: 0xf5a9393c (stale) Mirror 8: 0xf5a9393c (stale) Mirror 9: 0xf5a9393c (stale) Mirror 10: 0xf5a9393c (stale) Mirror 12: 0xf5a9393c (stale) Mirror 13: 0xf5a9393c (stale) Mirror 14: 0xf5a9393c (stale) Mirror 15: 0xf5a9393c (stale) Mirror 16: 0xf5a9393c (stale) Mirror 17: 0xf5a9393c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x755f7f17 Mirror 18: 0x755f7f17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000520 1779976089.549838 mapwrite 0x02ebf4 thru 0x0350fc (0x6509 bytes) 000521 1779976089.664589 read 0x026c2d thru 0x02a17e (0x3552 bytes) 000522 1779976089.680542 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf5a9393c (stale) Mirror 7: 0xf5a9393c (stale) Mirror 8: 0xf5a9393c (stale) Mirror 9: 0xf5a9393c (stale) Mirror 10: 0xf5a9393c (stale) Mirror 12: 0xf5a9393c (stale) Mirror 13: 0xf5a9393c (stale) Mirror 14: 0xf5a9393c (stale) Mirror 15: 0xf5a9393c (stale) Mirror 16: 0xf5a9393c (stale) Mirror 17: 0xf5a9393c (stale) Mirror 18: 0x755f7f17 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x87e8b99b Mirror 19: 0x87e8b99b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1779976090.830233 trunc from 0x040000 to 0x0080e2 (0x37f1d bytes) 000524 1779976090.877645 read 0x0053b0 thru 0x0080e2 (0x2d33 bytes) 000525 1779976090.887997 read 0x004de7 thru 0x005f3c (0x1156 bytes) 000526 1779976090.900293 read 0x003dc4 thru 0x00602c (0x2269 bytes) 000527 1779976090.905710 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x80e3): Mirror 7: 0xefedc7e3 (stale) Mirror 8: 0xefedc7e3 (stale) Mirror 9: 0xefedc7e3 (stale) Mirror 10: 0xefedc7e3 (stale) Mirror 12: 0xefedc7e3 (stale) Mirror 13: 0xefedc7e3 (stale) Mirror 14: 0xefedc7e3 (stale) Mirror 15: 0xefedc7e3 (stale) Mirror 16: 0xefedc7e3 (stale) Mirror 17: 0xefedc7e3 (stale) Mirror 18: 0xefedc7e3 (stale) Mirror 19: 0xefedc7e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x80e3): Mirror 7: 0xefedc7e3 (stale) Mirror 8: 0xefedc7e3 (stale) Mirror 9: 0xefedc7e3 (stale) Mirror 10: 0xefedc7e3 (stale) Mirror 12: 0xefedc7e3 (stale) Mirror 13: 0xefedc7e3 (stale) Mirror 14: 0xefedc7e3 (stale) Mirror 15: 0xefedc7e3 (stale) Mirror 16: 0xefedc7e3 (stale) Mirror 17: 0xefedc7e3 (stale) Mirror 18: 0xefedc7e3 (stale) Mirror 19: 0xefedc7e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x80e3: skipped 000529 1779976091.436054 fallocate 0x03da3c thru 0x03e46d (0xa32 bytes) 000530 1779976091.443286 mapwrite 0x014747 thru 0x01c9df (0x8299 bytes) 000531 1779976091.493658 read 0x01ad9e thru 0x023106 (0x8369 bytes) 000532 1779976091.503110 read 0x017644 thru 0x019ffc (0x29b9 bytes) 000533 1779976091.504934 punch from 0x0076af to 0x01708e (0xf9e0 bytes) 000534 1779976091.511152 fallocate 0x008f6a thru 0x01738f (0xe426 bytes) 000535 1779976091.520096 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e46e): Mirror 7: 0x43fdfd7 (stale) Mirror 8: 0x43fdfd7 (stale) Mirror 9: 0x43fdfd7 (stale) Mirror 12: 0x43fdfd7 (stale) Mirror 13: 0x43fdfd7 (stale) Mirror 14: 0x43fdfd7 (stale) Mirror 15: 0x43fdfd7 (stale) Mirror 16: 0x43fdfd7 (stale) Mirror 17: 0x43fdfd7 (stale) Mirror 18: 0x1d5793f (stale) Mirror 19: 0x153abb2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e46e): Mirror 7: 0x43fdfd7 (stale) Mirror 8: 0x43fdfd7 (stale) Mirror 9: 0x43fdfd7 (stale) Mirror 12: 0x43fdfd7 (stale) Mirror 13: 0x43fdfd7 (stale) Mirror 14: 0x43fdfd7 (stale) Mirror 15: 0x43fdfd7 (stale) Mirror 16: 0x43fdfd7 (stale) Mirror 17: 0x43fdfd7 (stale) Mirror 18: 0x1d5793f (stale) Mirror 19: 0x153abb2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e46e: skipped 000536 1779976092.165064 punch from 0x00be00 to 0x01b5fc (0xf7fd bytes) 000538 1779976092.174409 fallocate 0x01f4e1 thru 0x026775 (0x7295 bytes) 000539 1779976092.184997 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e46e): Mirror 7: 0x43fdfd7 (stale) Mirror 8: 0x43fdfd7 (stale) Mirror 9: 0x43fdfd7 (stale) Mirror 12: 0x43fdfd7 (stale) Mirror 13: 0x43fdfd7 (stale) Mirror 15: 0x43fdfd7 (stale) Mirror 16: 0x43fdfd7 (stale) Mirror 17: 0x43fdfd7 (stale) Mirror 18: 0x1d5793f (stale) Mirror 19: 0x153abb2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e46e): Mirror 7: 0x43fdfd7 (stale) Mirror 8: 0x43fdfd7 (stale) Mirror 9: 0x43fdfd7 (stale) Mirror 12: 0x43fdfd7 (stale) Mirror 13: 0x43fdfd7 (stale) Mirror 15: 0x43fdfd7 (stale) Mirror 16: 0x43fdfd7 (stale) Mirror 17: 0x43fdfd7 (stale) Mirror 18: 0x1d5793f (stale) Mirror 19: 0x153abb2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e46e: skipped 000540 1779976092.977198 fallocate 0x037173 thru 0x03fbd9 (0x8a67 bytes) 000541 1779976092.984491 fallocate 0x00633a thru 0x00f595 (0x925c bytes) 000542 1779976092.990689 mapread 0x018b86 thru 0x02287d (0x9cf8 bytes) 000543 1779976093.051985 trunc from 0x03fbda to 0x01a01d (0x25bbc bytes) 000544 1779976093.121032 read 0x00402c thru 0x010bda (0xcbaf bytes) 000545 1779976093.159028 punch from 0x0021f6 to 0x00ebe5 (0xc9f0 bytes) 000546 1779976093.169110 fallocate 0x01055c thru 0x012e13 (0x28b8 bytes) 000547 1779976093.178581 write 0x0386db thru 0x03f617 (0x6f3d bytes) 000548 1779976093.184604 read 0x0245c2 thru 0x02e771 (0xa1b0 bytes) 000549 1779976093.201660 fallocate 0x02b255 thru 0x033e00 (0x8bac bytes) 000550 1779976093.214926 write 0x02e893 thru 0x03c60f (0xdd7d bytes) 000551 1779976093.223194 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f618): Mirror 8: 0xc7bc5f20 (stale) Mirror 9: 0xc7bc5f20 (stale) Mirror 12: 0xc7bc5f20 (stale) Mirror 13: 0xc7bc5f20 (stale) Mirror 15: 0xc7bc5f20 (stale) Mirror 16: 0xc7bc5f20 (stale) Mirror 17: 0xc7bc5f20 (stale) Mirror 18: 0x1d946f9f (stale) Mirror 19: 0x3f18e9cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f618): Mirror 8: 0xc7bc5f20 (stale) Mirror 9: 0xc7bc5f20 (stale) Mirror 12: 0xc7bc5f20 (stale) Mirror 13: 0xc7bc5f20 (stale) Mirror 15: 0xc7bc5f20 (stale) Mirror 16: 0xc7bc5f20 (stale) Mirror 17: 0xc7bc5f20 (stale) Mirror 18: 0x1d946f9f (stale) Mirror 19: 0x3f18e9cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f618: skipped 000552 1779976094.104194 read 0x03ca07 thru 0x03d744 (0xd3e bytes) 000553 1779976094.116293 write 0x02fafa thru 0x030346 (0x84d bytes) 000554 1779976094.141061 mapwrite 0x007afa thru 0x012157 (0xa65e bytes) 000555 1779976094.192609 punch from 0x015925 to 0x01dbe0 (0x82bc bytes) 000556 1779976094.199708 fallocate 0x019fed thru 0x01ca60 (0x2a74 bytes) 000557 1779976094.208189 fallocate 0x0207fb thru 0x02b56e (0xad74 bytes) 000558 1779976094.223478 mapread 0x014741 thru 0x023741 (0xf001 bytes) 000559 1779976094.254861 write 0x01a202 thru 0x0274a9 (0xd2a8 bytes) 000560 1779976094.267459 punch from 0x00d849 to 0x00f2a3 (0x1a5b bytes) 000561 1779976094.274213 fallocate 0x006efc thru 0x00d780 (0x6885 bytes) 000562 1779976094.288798 punch from 0x00865b to 0x0115e9 (0x8f8f bytes) 000563 1779976094.299190 trunc from 0x0315aa to 0x03ee65 (0xd8bc bytes) 000564 1779976094.319696 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f618): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f618): Mirror 8: 0xc7bc5f20 (stale) Mirror 9: 0xc7bc5f20 (stale) Mirror 12: 0xc7bc5f20 (stale) Mirror 13: 0xc7bc5f20 (stale) Mirror 15: 0xc7bc5f20 (stale) Mirror 16: 0xc7bc5f20 (stale) Mirror 17: 0xc7bc5f20 (stale) Mirror 18: 0x1d946f9f (stale) Mirror 19: 0x3f18e9cb (stale) CRC-32 checksum value for chunk [0, 0x3f618): Mirror 1: 0x31182933 Mirror 20: 0x31182933 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f618: skipped 000565 1779976096.026151 read 0x01a919 thru 0x02819e (0xd886 bytes) 000566 1779976096.047247 punch from 0x003ac0 to 0x008b10 (0x5051 bytes) 000567 1779976096.144103 trunc from 0x03f618 to 0x0256c2 (0x19f55 bytes) 000568 1779976096.186419 write 0x00968c thru 0x00d0c8 (0x3a3d bytes) 000569 1779976096.207513 write 0x02da32 thru 0x031dd8 (0x43a7 bytes) 000570 1779976096.209189 mapwrite 0x03eae2 thru 0x03ffff (0x151e bytes) 000571 1779976096.286021 trunc from 0x00750c to 0x011851 (0xa346 bytes) 000572 1779976096.297186 fallocate 0x03e88e thru 0x03ffff (0x1772 bytes) 000573 1779976096.307071 read 0x029d6a thru 0x032adc (0x8d73 bytes) 000574 1779976096.327846 write 0x0088dd thru 0x014c25 (0xc349 bytes) 000575 1779976096.332723 trunc from 0x02fd02 to 0x039782 (0x9a81 bytes) 000576 1779976096.341605 punch from 0x00229b to 0x00e5e4 (0xc34a bytes) 000577 1779976096.359057 read 0x03de53 thru 0x03ffff (0x21ad bytes) 000578 1779976096.361193 trunc from 0x040000 to 0x00a5d7 (0x35a28 bytes) 000579 1779976096.383615 mapwrite 0x03920b thru 0x03ffff (0x6df5 bytes) 000581 1779976096.440702 trunc from 0x040000 to 0x00f9ef (0x30610 bytes) 000582 1779976096.461472 fallocate 0x03f546 thru 0x03ffff (0xaba bytes) 000583 1779976096.469233 fallocate 0x024077 thru 0x031fb3 (0xdf3d bytes) 000584 1779976096.475192 read 0x00ce69 thru 0x012e7d (0x6015 bytes) 000585 1779976096.487800 read 0x010b2a thru 0x01afbe (0xa495 bytes) 000586 1779976096.489247 write 0x000f02 thru 0x005b61 (0x4c60 bytes) 000587 1779976096.491835 mapread 0x01deee thru 0x01e6b6 (0x7c9 bytes) 000588 1779976096.501462 MIRROR_RESYNC 11 mirrors 000589 1779976098.776821 trunc from 0x01f1e4 to 0x02180d (0x262a bytes) 000590 1779976098.821064 read 0x00cf9c thru 0x011519 (0x457e bytes) 000591 1779976098.833626 trunc from 0x040000 to 0x00a0aa (0x35f55 bytes) 000592 1779976098.854235 mapread 0x002caf thru 0x00a0aa (0x73fc bytes) 000593 1779976098.885276 fallocate 0x019d7c thru 0x0201b9 (0x643e bytes) 000594 1779976098.896465 trunc from 0x001214 to 0x005046 (0x3e33 bytes) 000595 1779976098.909721 trunc from 0x000c68 to 0x00901e (0x83b7 bytes) 000596 1779976098.935208 mapread 0x003354 thru 0x00a0aa (0x6d57 bytes) 000597 1779976098.966755 trunc from 0x00a0ab to 0x025ed9 (0x1be2f bytes) 000598 1779976099.022048 trunc from 0x022f1e to 0x025ed9 (0x2fbc bytes) 000599 1779976099.030977 mapwrite 0x0153f5 thru 0x017773 (0x237f bytes) 000601 1779976099.086529 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x25eda): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25eda): Mirror 8: 0x790df3c (stale) Mirror 9: 0x790df3c (stale) Mirror 12: 0x790df3c (stale) Mirror 13: 0x790df3c (stale) Mirror 15: 0x790df3c (stale) Mirror 16: 0x790df3c (stale) Mirror 17: 0x790df3c (stale) Mirror 18: 0x790df3c (stale) Mirror 19: 0x790df3c (stale) Mirror 20: 0x790df3c (stale) CRC-32 checksum value for chunk [0, 0x25eda): Mirror 1: 0xfa73210c Mirror 21: 0xfa73210c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25eda: skipped 000602 1779976100.667594 MIRROR_RESYNC 12 mirrors 000603 1779976102.527673 trunc from 0x025eda to 0x00a868 (0x1b671 bytes) 000604 1779976102.714282 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa869): Mirror 8: 0x47056bf4 (stale) Mirror 9: 0x47056bf4 (stale) Mirror 12: 0x47056bf4 (stale) Mirror 13: 0x47056bf4 (stale) Mirror 15: 0x47056bf4 (stale) Mirror 16: 0x47056bf4 (stale) Mirror 17: 0x47056bf4 (stale) Mirror 18: 0x47056bf4 (stale) Mirror 19: 0x47056bf4 (stale) Mirror 20: 0x47056bf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa869): Mirror 8: 0x47056bf4 (stale) Mirror 9: 0x47056bf4 (stale) Mirror 12: 0x47056bf4 (stale) Mirror 13: 0x47056bf4 (stale) Mirror 15: 0x47056bf4 (stale) Mirror 16: 0x47056bf4 (stale) Mirror 17: 0x47056bf4 (stale) Mirror 18: 0x47056bf4 (stale) Mirror 19: 0x47056bf4 (stale) Mirror 20: 0x47056bf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa869: skipped 000606 1779976103.506694 write 0x031e53 thru 0x0396e0 (0x788e bytes) 000607 1779976103.511622 write 0x02d3ad thru 0x0348f6 (0x754a bytes) 000608 1779976103.519732 punch from 0x036d9c to 0x0396e0 (0x2945 bytes) 000609 1779976103.531490 write 0x0078d4 thru 0x012f31 (0xb65e bytes) 000610 1779976103.556699 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396e1): Mirror 8: 0x7c36d564 (stale) Mirror 9: 0x7c36d564 (stale) Mirror 12: 0x7c36d564 (stale) Mirror 13: 0x7c36d564 (stale) Mirror 15: 0x7c36d564 (stale) Mirror 16: 0x7c36d564 (stale) Mirror 17: 0x7c36d564 (stale) Mirror 18: 0x7c36d564 (stale) Mirror 19: 0x7c36d564 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e1): Mirror 8: 0x7c36d564 (stale) Mirror 9: 0x7c36d564 (stale) Mirror 12: 0x7c36d564 (stale) Mirror 13: 0x7c36d564 (stale) Mirror 15: 0x7c36d564 (stale) Mirror 16: 0x7c36d564 (stale) Mirror 17: 0x7c36d564 (stale) Mirror 18: 0x7c36d564 (stale) Mirror 19: 0x7c36d564 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e1: skipped 000611 1779976104.151596 mapwrite 0x0219fd thru 0x026ceb (0x52ef bytes) 000612 1779976104.222984 punch from 0x01da68 to 0x021764 (0x3cfd bytes) 000613 1779976104.228891 fallocate 0x0266bc thru 0x0294cd (0x2e12 bytes) 000614 1779976104.234851 mapwrite 0x014988 thru 0x01764b (0x2cc4 bytes) 000615 1779976104.280498 trunc from 0x013244 to 0x020cbc (0xda79 bytes) 000616 1779976104.293190 mapwrite 0x02cac3 thru 0x037521 (0xaa5f bytes) 000617 1779976104.403159 mapwrite 0x01d684 thru 0x01fbad (0x252a bytes) 000618 1779976104.437075 trunc from 0x038eba to 0x0396e0 (0x827 bytes) 000619 1779976104.453796 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x396e1): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e1): Mirror 8: 0x7c36d564 (stale) Mirror 9: 0x7c36d564 (stale) Mirror 12: 0x7c36d564 (stale) Mirror 13: 0x7c36d564 (stale) Mirror 15: 0x7c36d564 (stale) Mirror 16: 0x7c36d564 (stale) Mirror 17: 0x7c36d564 (stale) Mirror 18: 0x7c36d564 (stale) Mirror 19: 0x7c36d564 (stale) CRC-32 checksum value for chunk [0, 0x396e1): Mirror 1: 0xe782a1e4 Mirror 20: 0xe782a1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e1: skipped 000620 1779976106.011656 trunc from 0x0396e1 to 0x024e0f (0x148d1 bytes) 000621 1779976106.065613 mapwrite 0x01c8a0 thru 0x01f16b (0x28cc bytes) 000622 1779976106.104036 write 0x01b42e thru 0x0209d2 (0x55a5 bytes) 000623 1779976106.106472 write 0x0227fc thru 0x028002 (0x5807 bytes) 000624 1779976106.108227 mapwrite 0x03d878 thru 0x03ffff (0x2788 bytes) 000625 1779976106.167917 write 0x01a87e thru 0x01e20b (0x398e bytes) 000626 1779976106.175755 MIRROR_RESYNC 11 mirrors 000627 1779976109.427898 read 0x0232f9 thru 0x0299df (0x66e7 bytes) 000628 1779976109.436832 mapread 0x0047eb thru 0x00a24c (0x5a62 bytes) 000629 1779976109.469866 punch from 0x027e7a to 0x02c4bb (0x4642 bytes) 000630 1779976109.510491 trunc from 0x01cc8e to 0x0201c0 (0x3533 bytes) 000631 1779976109.529512 MIRROR_RESYNC 11 mirrors 000632 1779976113.217299 MIRROR_RESYNC 11 mirrors 000634 1779976114.359842 trunc from 0x040000 to 0x02e106 (0x11ef9 bytes) 000635 1779976114.415525 mapwrite 0x038c20 thru 0x03951b (0x8fc bytes) 000636 1779976114.461063 read 0x033bed thru 0x03951b (0x592f bytes) 000638 1779976114.493052 fallocate 0x02ba9c thru 0x0362ae (0xa813 bytes) 000639 1779976114.503657 MIRROR_RESYNC 11 mirrors 000640 1779976118.118995 write 0x029b57 thru 0x02a173 (0x61d bytes) 000641 1779976118.162656 read 0x036743 thru 0x037615 (0xed3 bytes) 000642 1779976118.173102 trunc from 0x02a09c to 0x03791c (0xd881 bytes) 000643 1779976118.204041 fallocate 0x03e4a6 thru 0x03ffff (0x1b5a bytes) 000644 1779976118.228003 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96444868 (stale) Mirror 9: 0x96444868 (stale) Mirror 12: 0x96444868 (stale) Mirror 13: 0x96444868 (stale) Mirror 16: 0x96444868 (stale) Mirror 17: 0x96444868 (stale) Mirror 18: 0x96444868 (stale) Mirror 19: 0x96444868 (stale) Mirror 20: 0x96444868 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x96444868 (stale) Mirror 9: 0x96444868 (stale) Mirror 12: 0x96444868 (stale) Mirror 13: 0x96444868 (stale) Mirror 16: 0x96444868 (stale) Mirror 17: 0x96444868 (stale) Mirror 18: 0x96444868 (stale) Mirror 19: 0x96444868 (stale) Mirror 20: 0x96444868 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1779976118.908950 write 0x032ba9 thru 0x03c5d2 (0x9a2a bytes) 000646 1779976118.938819 MIRROR_RESYNC 10 mirrors 000647 1779976122.662657 trunc from 0x01bec6 to 0x021a3b (0x5b76 bytes) 000648 1779976122.698021 punch from 0x033612 to 0x03fbef (0xc5de bytes) 000649 1779976122.702862 punch from 0x02b4d0 to 0x037332 (0xbe63 bytes) 000650 1779976122.706751 read 0x00f1e4 thru 0x01899c (0x97b9 bytes) 000653 1779976122.717869 fallocate 0x0211fd thru 0x026410 (0x5214 bytes) 000654 1779976122.722850 mapread 0x00e4e2 thru 0x0100f7 (0x1c16 bytes) 000655 1779976122.731742 mapwrite 0x01e13a thru 0x025ae9 (0x79b0 bytes) 000656 1779976122.796771 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1689b465 (stale) Mirror 12: 0x1689b465 (stale) Mirror 13: 0x1689b465 (stale) Mirror 16: 0x1689b465 (stale) Mirror 17: 0x1689b465 (stale) Mirror 18: 0x1689b465 (stale) Mirror 19: 0x1689b465 (stale) Mirror 20: 0x1689b465 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1689b465 (stale) Mirror 12: 0x1689b465 (stale) Mirror 13: 0x1689b465 (stale) Mirror 16: 0x1689b465 (stale) Mirror 17: 0x1689b465 (stale) Mirror 18: 0x1689b465 (stale) Mirror 19: 0x1689b465 (stale) Mirror 20: 0x1689b465 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1779976123.629805 fallocate 0x001ffd thru 0x005304 (0x3308 bytes) 000658 1779976123.639935 mapwrite 0x00c55c thru 0x017514 (0xafb9 bytes) 000659 1779976123.715684 read 0x00bcfe thru 0x01ae19 (0xf11c bytes) 000660 1779976123.716791 mapwrite 0x031eab thru 0x039a97 (0x7bed bytes) 000661 1779976123.781035 trunc from 0x01813f to 0x024134 (0xbff6 bytes) 000662 1779976123.798052 trunc from 0x007714 to 0x010836 (0x9123 bytes) 000663 1779976123.825289 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1689b465 (stale) Mirror 12: 0x1689b465 (stale) Mirror 13: 0x1689b465 (stale) Mirror 16: 0x1689b465 (stale) Mirror 17: 0x1689b465 (stale) Mirror 18: 0x1689b465 (stale) Mirror 19: 0x1689b465 (stale) Mirror 20: 0x1689b465 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad189977 Mirror 21: 0xad189977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1779976125.133677 mapread 0x03ecf7 thru 0x03ffff (0x1309 bytes) 000665 1779976125.197931 trunc from 0x03030c to 0x0373ad (0x70a2 bytes) 000666 1779976125.250666 mapread 0x01d20e thru 0x02b5b9 (0xe3ac bytes) 000667 1779976125.291598 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1689b465 (stale) Mirror 12: 0x1689b465 (stale) Mirror 16: 0x1689b465 (stale) Mirror 17: 0x1689b465 (stale) Mirror 18: 0x1689b465 (stale) Mirror 19: 0x1689b465 (stale) Mirror 20: 0x1689b465 (stale) Mirror 21: 0xad189977 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1689b465 (stale) Mirror 12: 0x1689b465 (stale) Mirror 16: 0x1689b465 (stale) Mirror 17: 0x1689b465 (stale) Mirror 18: 0x1689b465 (stale) Mirror 19: 0x1689b465 (stale) Mirror 20: 0x1689b465 (stale) Mirror 21: 0xad189977 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000668 1779976126.082588 fallocate 0x01f481 thru 0x027a61 (0x85e1 bytes) 000669 1779976126.088632 punch from 0x014c08 to 0x017989 (0x2d82 bytes) 000670 1779976126.096345 write 0x008cf7 thru 0x01121c (0x8526 bytes) 000671 1779976126.115888 fallocate 0x033dd4 thru 0x03f951 (0xbb7e bytes) 000672 1779976126.123220 trunc from 0x0005fd to 0x00d2da (0xccde bytes) 000673 1779976126.137475 fallocate 0x01c361 thru 0x023d9a (0x7a3a bytes) 000674 1779976126.148774 trunc from 0x022786 to 0x02315b (0x9d6 bytes) 000675 1779976126.162269 mapread 0x00bc76 thru 0x00beda (0x265 bytes) 000676 1779976126.188878 punch from 0x0307ce to 0x035e73 (0x56a6 bytes) 000677 1779976126.205084 mapread 0x001300 thru 0x00f460 (0xe161 bytes) 000678 1779976126.222010 trunc from 0x02a973 to 0x02cc0c (0x229a bytes) 000679 1779976126.230963 read 0x0292ef thru 0x038db8 (0xfaca bytes) 000680 1779976126.251125 punch from 0x02c293 to 0x03396d (0x76db bytes) 000681 1779976126.270454 mapread 0x01548e thru 0x02023a (0xadad bytes) 000684 1779976126.369541 mapread 0x01709a thru 0x018fae (0x1f15 bytes) 000685 1779976126.372842 fallocate 0x008576 thru 0x00b752 (0x31dd bytes) 000686 1779976126.391854 trunc from 0x0047ea to 0x009743 (0x4f5a bytes) 000687 1779976126.401800 mapwrite 0x03917f thru 0x03ffff (0x6e81 bytes) 000688 1779976126.474493 MIRROR_RESYNC 9 mirrors 000689 1779976130.259974 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) 000690 1779976130.614420 MIRROR_EXTEND to 9 mirrors 000691 1779976131.628734 trunc from 0x021d57 to 0x02a279 (0x8523 bytes) 000692 1779976131.695080 fallocate 0x034654 thru 0x039922 (0x52cf bytes) 000693 1779976131.712425 MIRROR_RESYNC 9 mirrors 000694 1779976134.088465 trunc from 0x040000 to 0x023bd8 (0x1c427 bytes) 000696 1779976134.126372 fallocate 0x00879a thru 0x012a54 (0xa2bb bytes) 000697 1779976134.133478 punch from 0x008c5e to 0x012c5b (0x9ffe bytes) 000698 1779976134.141539 MIRROR_RESYNC 9 mirrors 000699 1779976136.438438 punch from 0x001d72 to 0x00fb52 (0xdde1 bytes) 000700 1779976136.467665 trunc from 0x023bd9 to 0x036f34 (0x1335c bytes) 000701 1779976136.486793 MIRROR_RESYNC 9 mirrors 000702 1779976138.657040 read 0x03477f thru 0x036f34 (0x27b6 bytes) 000703 1779976138.662548 mapwrite 0x006d0f thru 0x00ced2 (0x61c4 bytes) 000704 1779976138.783863 write 0x01cbdf thru 0x027510 (0xa932 bytes) 000705 1779976138.810987 trunc from 0x036f35 to 0x008ba2 (0x2e392 bytes) 000706 1779976138.846200 mapread 0x00262a thru 0x008ba2 (0x6579 bytes) 000707 1779976138.855092 mapread 0x007654 thru 0x008ba2 (0x154f bytes) 000709 1779976138.865012 mapwrite 0x00fc84 thru 0x0198e2 (0x9c5f bytes) 000710 1779976138.972953 punch from 0x00eb63 to 0x019119 (0xa5b7 bytes) 000712 1779976138.994935 read 0x0115ca thru 0x0198e2 (0x8319 bytes) 000713 1779976139.023774 mapwrite 0x029e96 thru 0x03521c (0xb387 bytes) 000714 1779976139.151820 mapwrite 0x0070cd thru 0x01408d (0xcfc1 bytes) 000715 1779976139.230691 trunc from 0x03521d to 0x01b1cf (0x1a04d bytes) 000716 1779976139.252836 punch from 0x008ea0 to 0x0104d7 (0x7638 bytes) 000717 1779976139.261457 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1b1d0): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1d0): Mirror 9: 0x746342b0 (stale) Mirror 12: 0x746342b0 (stale) Mirror 16: 0x746342b0 (stale) Mirror 17: 0x746342b0 (stale) Mirror 18: 0x746342b0 (stale) Mirror 19: 0x746342b0 (stale) Mirror 20: 0x746342b0 (stale) Mirror 21: 0x746342b0 (stale) CRC-32 checksum value for chunk [0, 0x1b1d0): Mirror 1: 0x44294759 Mirror 22: 0x44294759 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1d0: skipped 000718 1779976140.617879 punch from 0x016113 to 0x0174aa (0x1398 bytes) 000719 1779976140.694959 trunc from 0x015d5f to 0x01b1cf (0x5471 bytes) 000720 1779976140.710040 read 0x01163d thru 0x011e87 (0x84b bytes) 000721 1779976140.718574 trunc from 0x017a9b to 0x01b1cf (0x3735 bytes) 000722 1779976140.739976 write 0x0079e0 thru 0x011102 (0x9723 bytes) 000723 1779976140.763434 punch from 0x01a8c9 to 0x01b1cf (0x907 bytes) 000724 1779976140.774526 write 0x00365b thru 0x0048c7 (0x126d bytes) 000725 1779976140.798740 mapwrite 0x00ae78 thru 0x013001 (0x818a bytes) 000726 1779976140.878208 read 0x0032b7 thru 0x00f052 (0xbd9c bytes) 000727 1779976140.882119 fallocate 0x008bbe thru 0x009e12 (0x1255 bytes) 000729 1779976140.905722 trunc from 0x01b1d0 to 0x039b3e (0x1e96f bytes) 000730 1779976140.994269 fallocate 0x0102d5 thru 0x016089 (0x5db5 bytes) 000731 1779976141.006785 mapwrite 0x02e374 thru 0x03a936 (0xc5c3 bytes) 000732 1779976141.069069 write 0x029796 thru 0x03203b (0x88a6 bytes) 000733 1779976141.070438 write 0x00f6d8 thru 0x01b774 (0xc09d bytes) 000734 1779976141.072060 write 0x00486f thru 0x00e8a8 (0xa03a bytes) 000735 1779976141.073358 mapread 0x026b12 thru 0x029d93 (0x3282 bytes) 000736 1779976141.074642 trunc from 0x03a937 to 0x00e5d0 (0x2c366 bytes) 000737 1779976141.088609 punch from 0x00707e to 0x00a994 (0x3917 bytes) 000739 1779976141.107335 trunc from 0x00e00a to 0x00e5d0 (0x5c7 bytes) 000740 1779976141.112469 write 0x0156f1 thru 0x01a923 (0x5233 bytes) 000741 1779976141.113828 fallocate 0x00fc83 thru 0x011d88 (0x2106 bytes) 000742 1779976141.121218 write 0x002106 thru 0x00d031 (0xaf2c bytes) 000743 1779976141.122565 read 0x0093cf thru 0x009587 (0x1b9 bytes) 000744 1779976141.123769 punch from 0x00dfbf to 0x017076 (0x90b8 bytes) 000745 1779976141.137920 mapread 0x00ab0b thru 0x015e59 (0xb34f bytes) 000746 1779976141.151220 trunc from 0x00cb3a to 0x013254 (0x671b bytes) 000747 1779976141.156201 trunc from 0x010b09 to 0x01a923 (0x9e1b bytes) 000748 1779976141.165794 MIRROR_RESYNC 10 mirrors 000749 1779976143.146340 mapread 0x0178e7 thru 0x01a923 (0x303d bytes) 000750 1779976143.168592 MIRROR_RESYNC 10 mirrors 000751 1779976143.620982 read 0x009263 thru 0x00bd97 (0x2b35 bytes) 000752 1779976143.628486 mapwrite 0x0145ae thru 0x01b0c1 (0x6b14 bytes) 000753 1779976143.706345 mapread 0x007e4f thru 0x013772 (0xb924 bytes) 000755 1779976143.713911 punch from 0x003e3e to 0x010788 (0xc94b bytes) 000756 1779976143.718511 fallocate 0x012da8 thru 0x0138f2 (0xb4b bytes) 000757 1779976143.722992 mapread 0x00e76c thru 0x0102e4 (0x1b79 bytes) 000759 1779976143.730524 read 0x014c79 thru 0x01b0c1 (0x6449 bytes) 000760 1779976143.731192 write 0x033a98 thru 0x038d97 (0x5300 bytes) 000761 1779976143.732802 fallocate 0x01304c thru 0x013ad0 (0xa85 bytes) 000762 1779976143.740079 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38d98): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d98): Mirror 9: 0xb67163c9 (stale) Mirror 12: 0xb67163c9 (stale) Mirror 16: 0xb67163c9 (stale) Mirror 17: 0xb67163c9 (stale) Mirror 18: 0xb67163c9 (stale) Mirror 19: 0xb67163c9 (stale) Mirror 20: 0xb67163c9 (stale) Mirror 21: 0xb67163c9 (stale) Mirror 22: 0xb67163c9 (stale) CRC-32 checksum value for chunk [0, 0x38d98): Mirror 1: 0x9ab6e26a Mirror 23: 0x9ab6e26a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d98: skipped 000763 1779976144.762114 mapwrite 0x03ef8d thru 0x03ffff (0x1073 bytes) 000764 1779976144.856297 write 0x00a721 thru 0x019872 (0xf152 bytes) 000765 1779976144.879179 MIRROR_RESYNC 11 mirrors 000766 1779976147.814726 trunc from 0x0330c5 to 0x03419b (0x10d7 bytes) 000767 1779976147.846663 punch from 0x0366fd to 0x03e594 (0x7e98 bytes) 000768 1779976147.855557 mapread 0x006e43 thru 0x00d53f (0x66fd bytes) 000769 1779976147.874373 trunc from 0x01a1c8 to 0x02062a (0x6463 bytes) 000770 1779976147.879366 trunc from 0x040000 to 0x024ffc (0x1b003 bytes) 000771 1779976147.909614 fallocate 0x033eae thru 0x03f837 (0xb98a bytes) 000772 1779976147.914568 mapwrite 0x009177 thru 0x00a07a (0xf04 bytes) 000773 1779976147.941792 trunc from 0x0283f1 to 0x02e1dd (0x5ded bytes) 000774 1779976147.955619 read 0x037a75 thru 0x03f837 (0x7dc3 bytes) 000775 1779976147.981161 trunc from 0x0377b6 to 0x03f837 (0x8082 bytes) 000777 1779976147.995851 punch from 0x02f65c to 0x03dc9d (0xe642 bytes) 000778 1779976148.020608 fallocate 0x02c0f0 thru 0x032f30 (0x6e41 bytes) 000779 1779976148.047047 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f838): Mirror 12: 0x193516ed (stale) Mirror 16: 0x193516ed (stale) Mirror 17: 0x193516ed (stale) Mirror 18: 0x193516ed (stale) Mirror 19: 0x193516ed (stale) Mirror 20: 0x193516ed (stale) Mirror 21: 0x193516ed (stale) Mirror 22: 0x193516ed (stale) Mirror 23: 0x193516ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f838): Mirror 12: 0x193516ed (stale) Mirror 16: 0x193516ed (stale) Mirror 17: 0x193516ed (stale) Mirror 18: 0x193516ed (stale) Mirror 19: 0x193516ed (stale) Mirror 20: 0x193516ed (stale) Mirror 21: 0x193516ed (stale) Mirror 22: 0x193516ed (stale) Mirror 23: 0x193516ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f838: skipped 000780 1779976148.733519 trunc from 0x03f838 to 0x00e139 (0x316fe bytes) 000781 1779976148.749798 trunc from 0x00e13a to 0x02e496 (0x2035d bytes) 000782 1779976148.760627 punch from 0x022b85 to 0x02e496 (0xb912 bytes) 000783 1779976148.765532 trunc from 0x02e497 to 0x008060 (0x26436 bytes) 000784 1779976148.778864 write 0x0200e7 thru 0x020279 (0x193 bytes) 000785 1779976148.781561 trunc from 0x02027a to 0x006d32 (0x19547 bytes) 000788 1779976148.800872 trunc from 0x003544 to 0x006d32 (0x37ef bytes) 000789 1779976148.808125 mapwrite 0x023cdd thru 0x0318ec (0xdc10 bytes) 000790 1779976148.892498 trunc from 0x0318ed to 0x009c52 (0x27c9a bytes) 000791 1779976148.906521 trunc from 0x007cb7 to 0x008da9 (0x10f3 bytes) 000792 1779976148.910888 read 0x001fcb thru 0x006102 (0x4138 bytes) 000793 1779976148.921547 read 0x000fb3 thru 0x009429 (0x8477 bytes) 000794 1779976148.929705 punch from 0x003792 to 0x007a9f (0x430e bytes) 000797 1779976148.936426 read 0x00111a thru 0x008bff (0x7ae6 bytes) 000798 1779976148.943820 write 0x01c942 thru 0x02b2bc (0xe97b bytes) 000799 1779976148.945792 mapwrite 0x02ed70 thru 0x034e23 (0x60b4 bytes) 000800 1779976148.991379 MIRROR_RESYNC 10 mirrors 000801 1779976151.108887 trunc from 0x034e24 to 0x018818 (0x1c60b bytes) 000802 1779976151.154602 fallocate 0x00aceb thru 0x01ac12 (0xff28 bytes) 000804 1779976151.163569 punch from 0x00c0c9 to 0x00fa45 (0x397d bytes) 000805 1779976151.167472 mapread 0x00ac3c thru 0x015b90 (0xaf55 bytes) 000806 1779976151.222835 trunc from 0x018819 to 0x03cc57 (0x2443f bytes) 000807 1779976151.246721 trunc from 0x0111d5 to 0x015690 (0x44bc bytes) 000808 1779976151.252614 mapwrite 0x023a0e thru 0x023a4f (0x42 bytes) 000809 1779976151.310335 write 0x013415 thru 0x01902f (0x5c1b bytes) 000810 1779976151.343357 MIRROR_RESYNC 10 mirrors 000812 1779976154.497647 read 0x00abe1 thru 0x00ca95 (0x1eb5 bytes) 000814 1779976154.503217 fallocate 0x03e5d7 thru 0x03ffff (0x1a29 bytes) 000815 1779976154.545747 mapwrite 0x016f22 thru 0x01caab (0x5b8a bytes) 000816 1779976154.585248 trunc from 0x03cc58 to 0x025952 (0x17305 bytes) 000817 1779976154.604002 read 0x00c66a thru 0x018b4c (0xc4e3 bytes) 000819 1779976154.617857 read 0x0037d0 thru 0x006888 (0x30b9 bytes) 000820 1779976154.618868 write 0x02da88 thru 0x0333b0 (0x5929 bytes) 000821 1779976154.624671 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333b1): Mirror 16: 0xd6ffa25 (stale) Mirror 17: 0xd6ffa25 (stale) Mirror 18: 0xd6ffa25 (stale) Mirror 19: 0xd6ffa25 (stale) Mirror 20: 0xd6ffa25 (stale) Mirror 21: 0xd6ffa25 (stale) Mirror 22: 0xd6ffa25 (stale) Mirror 23: 0xd6ffa25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333b1): Mirror 16: 0xd6ffa25 (stale) Mirror 17: 0xd6ffa25 (stale) Mirror 18: 0xd6ffa25 (stale) Mirror 19: 0xd6ffa25 (stale) Mirror 20: 0xd6ffa25 (stale) Mirror 21: 0xd6ffa25 (stale) Mirror 22: 0xd6ffa25 (stale) Mirror 23: 0xd6ffa25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333b1: skipped 000822 1779976155.312448 fallocate 0x03248f thru 0x0353ab (0x2f1d bytes) 000823 1779976155.321233 write 0x0319bf thru 0x0380a2 (0x66e4 bytes) 000824 1779976155.336441 MIRROR_RESYNC 9 mirrors 000827 1779976157.803599 trunc from 0x0380a3 to 0x024ef9 (0x131a9 bytes) 000828 1779976157.838648 punch from 0x015726 to 0x024974 (0xf24f bytes) 000829 1779976157.843766 fallocate 0x006eab thru 0x007749 (0x89f bytes) 000830 1779976157.848210 write 0x03035f thru 0x03fb7a (0xf81c bytes) 000831 1779976157.851417 trunc from 0x03fb7b to 0x02a81a (0x15360 bytes) 000832 1779976157.867848 read 0x022439 thru 0x029328 (0x6ef0 bytes) 000834 1779976157.875603 write 0x039f4f thru 0x03c53a (0x25ec bytes) 000835 1779976157.876758 write 0x024f4c thru 0x028951 (0x3a06 bytes) 000836 1779976157.882502 MIRROR_RESYNC 9 mirrors 000837 1779976160.102527 trunc from 0x03c53b to 0x026b40 (0x159fa bytes) 000839 1779976160.170187 mapread 0x008a76 thru 0x012c36 (0xa1c1 bytes) 000840 1779976160.210812 MIRROR_RESYNC 9 mirrors 000841 1779976162.466692 trunc from 0x00e2e4 to 0x016b79 (0x8896 bytes) 000842 1779976162.571467 trunc from 0x026b41 to 0x00144e (0x256f2 bytes) 000843 1779976162.619526 fallocate 0x03b888 thru 0x03f128 (0x38a1 bytes) 000844 1779976162.629482 read 0x0278ca thru 0x03618c (0xe8c3 bytes) 000845 1779976162.642046 punch from 0x036a7b to 0x03a18b (0x3711 bytes) 000846 1779976162.648981 mapwrite 0x010b58 thru 0x014c8f (0x4138 bytes) 000847 1779976162.714835 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 18: 0xd02243e9 (stale) Mirror 19: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 18: 0xd02243e9 (stale) Mirror 19: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f129: skipped 000848 1779976163.343372 fallocate 0x0087d2 thru 0x01555f (0xcd8e bytes) 000849 1779976163.351929 mapread 0x013639 thru 0x0225ce (0xef96 bytes) 000850 1779976163.415087 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 18: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 18: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f129: skipped 000851 1779976163.970790 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 18: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) CRC-32 checksum value for chunk [0, 0x3f129): Mirror 1: 0xd2afcffa Mirror 24: 0xd2afcffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f129: skipped 000852 1779976164.891867 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f129): Mirror 16: 0xd02243e9 (stale) Mirror 17: 0xd02243e9 (stale) Mirror 20: 0xd02243e9 (stale) Mirror 22: 0xd02243e9 (stale) Mirror 23: 0xd02243e9 (stale) CRC-32 checksum value for chunk [0, 0x3f129): Mirror 1: 0xd2afcffa Mirror 24: 0xd2afcffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f129: skipped 000853 1779976165.666519 punch from 0x009554 to 0x009731 (0x1de bytes) 000854 1779976165.732256 punch from 0x0282f4 to 0x0369b5 (0xe6c2 bytes) 000856 1779976165.739075 trunc from 0x03f129 to 0x0254bf (0x19c69 bytes) 000857 1779976165.760242 punch from 0x001635 to 0x009d82 (0x874e bytes) 000858 1779976165.766135 trunc from 0x0254c0 to 0x02bffd (0x6b3e bytes) 000859 1779976165.782763 mapread 0x023c6e thru 0x024ca2 (0x1035 bytes) 000861 1779976165.794165 write 0x0348df thru 0x03da48 (0x916a bytes) 000863 1779976165.797222 mapwrite 0x029ac1 thru 0x02ecfb (0x523b bytes) 000865 1779976165.830377 trunc from 0x010fec to 0x0160de (0x50f3 bytes) 000866 1779976165.848436 write 0x03d3bb thru 0x03ffff (0x2c45 bytes) 000867 1779976165.849498 punch from 0x033288 to 0x0375e5 (0x435e bytes) 000868 1779976165.864385 mapread 0x0024e1 thru 0x0053b0 (0x2ed0 bytes) 000869 1779976165.903361 trunc from 0x040000 to 0x009353 (0x36cac bytes) 000870 1779976165.920999 read 0x003263 thru 0x004b0c (0x18aa bytes) 000871 1779976165.926230 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9354): Mirror 16: 0x3aff8ea0 (stale) Mirror 17: 0x3aff8ea0 (stale) Mirror 20: 0x3aff8ea0 (stale) Mirror 22: 0x3aff8ea0 (stale) Mirror 23: 0x3aff8ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9354): Mirror 16: 0x3aff8ea0 (stale) Mirror 17: 0x3aff8ea0 (stale) Mirror 20: 0x3aff8ea0 (stale) Mirror 22: 0x3aff8ea0 (stale) Mirror 23: 0x3aff8ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9354: skipped 000872 1779976166.282004 fallocate 0x023cc9 thru 0x025db5 (0x20ed bytes) 000874 1779976166.292571 punch from 0x004772 to 0x009353 (0x4be2 bytes) 000876 1779976166.298026 trunc from 0x009354 to 0x033d8f (0x2aa3c bytes) 000877 1779976166.315300 trunc from 0x033d90 to 0x0276cf (0xc6c0 bytes) 000878 1779976166.334263 trunc from 0x0276d0 to 0x01a785 (0xcf4a bytes) 000879 1779976166.359560 read 0x00e40f thru 0x01a785 (0xc377 bytes) 000880 1779976166.389669 read 0x009692 thru 0x00f846 (0x61b5 bytes) 000881 1779976166.406934 mapread 0x00f249 thru 0x017161 (0x7f19 bytes) 000882 1779976166.410755 trunc from 0x012593 to 0x01a785 (0x81f3 bytes) 000883 1779976166.421070 mapwrite 0x022d26 thru 0x0259a0 (0x2c7b bytes) 000884 1779976166.488204 mapread 0x00718a thru 0x00c379 (0x51f0 bytes) 000885 1779976166.490557 mapwrite 0x023b39 thru 0x024c84 (0x114c bytes) 000886 1779976166.502749 fallocate 0x03015f thru 0x03ef49 (0xedeb bytes) 000887 1779976166.508478 read 0x016887 thru 0x019f13 (0x368d bytes) 000888 1779976166.524054 mapwrite 0x01897b thru 0x0222da (0x9960 bytes) 000889 1779976166.550224 write 0x01fbfa thru 0x023d05 (0x410c bytes) 000890 1779976166.551452 mapread 0x01e03e thru 0x020a58 (0x2a1b bytes) 000891 1779976166.552440 read 0x00f5f8 thru 0x0102c1 (0xcca bytes) 000892 1779976166.554228 mapread 0x0019e3 thru 0x00b80e (0x9e2c bytes) 000893 1779976166.556837 mapwrite 0x033d24 thru 0x038f0b (0x51e8 bytes) 000894 1779976166.600107 mapread 0x00b1b4 thru 0x018033 (0xce80 bytes) 000895 1779976166.603445 fallocate 0x02a9a5 thru 0x036053 (0xb6af bytes) 000896 1779976166.609774 punch from 0x0267a2 to 0x03665c (0xfebb bytes) 000897 1779976166.617148 punch from 0x0339a4 to 0x038382 (0x49df bytes) 000898 1779976166.625660 trunc from 0x038f0c to 0x037661 (0x18aa bytes) 000899 1779976166.675027 punch from 0x0103c9 to 0x018fb8 (0x8bf0 bytes) 000900 1779976166.686060 trunc from 0x037662 to 0x0346bd (0x2fa4 bytes) 000901 1779976166.726431 trunc from 0x0151c8 to 0x01b528 (0x6361 bytes) 000902 1779976166.734806 fallocate 0x022688 thru 0x0231c9 (0xb42 bytes) 000903 1779976166.747392 read 0x016ec1 thru 0x01a9d0 (0x3b10 bytes) 000904 1779976166.764412 write 0x00dfe8 thru 0x00e516 (0x52f bytes) 000905 1779976166.766648 mapread 0x00befe thru 0x010e5c (0x4f5f bytes) 000907 1779976166.769855 read 0x0107e3 thru 0x016cf4 (0x6512 bytes) 000908 1779976166.772803 mapwrite 0x0224fc thru 0x03211f (0xfc24 bytes) 000909 1779976166.822948 mapwrite 0x023a09 thru 0x0240f0 (0x6e8 bytes) 000910 1779976166.852134 punch from 0x0154ae to 0x023e5d (0xe9b0 bytes) 000911 1779976166.856455 trunc from 0x0346be to 0x00d57f (0x2713e bytes) 000912 1779976166.890457 MIRROR_RESYNC 6 mirrors 000913 1779976168.113675 punch from 0x000ee4 to 0x002c21 (0x1d3e bytes) 000914 1779976168.166831 write 0x0293ca thru 0x035c60 (0xc897 bytes) 000915 1779976168.173603 mapwrite 0x00d8ab thru 0x01cb8f (0xf2e5 bytes) 000916 1779976168.255691 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35c61): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c61): Mirror 16: 0xa24bdf3a (stale) Mirror 17: 0xa24bdf3a (stale) Mirror 20: 0xa24bdf3a (stale) Mirror 22: 0xa24bdf3a (stale) Mirror 23: 0xa24bdf3a (stale) CRC-32 checksum value for chunk [0, 0x35c61): Mirror 1: 0x9d950dfd Mirror 24: 0x9d950dfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c61: skipped 000917 1779976169.252964 read 0x025ee0 thru 0x030303 (0xa424 bytes) 000918 1779976169.272256 write 0x001cd8 thru 0x00e05f (0xc388 bytes) 000919 1779976169.336419 mapwrite 0x008584 thru 0x00db1c (0x5599 bytes) 000920 1779976169.352986 read 0x010010 thru 0x0179cf (0x79c0 bytes) 000921 1779976169.364291 trunc from 0x035c61 to 0x003c99 (0x31fc7 bytes) 000922 1779976169.406754 mapread 0x003753 thru 0x003c99 (0x547 bytes) 000923 1779976169.413106 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9a): Mirror 16: 0x68ce7930 (stale) Mirror 17: 0x68ce7930 (stale) Mirror 22: 0x68ce7930 (stale) Mirror 23: 0x68ce7930 (stale) Mirror 24: 0x68ce7930 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9a): Mirror 16: 0x68ce7930 (stale) Mirror 17: 0x68ce7930 (stale) Mirror 22: 0x68ce7930 (stale) Mirror 23: 0x68ce7930 (stale) Mirror 24: 0x68ce7930 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9a: skipped 000924 1779976169.833860 mapwrite 0x03eded thru 0x03ffff (0x1213 bytes) 000925 1779976169.884940 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 24: 0xe22862b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21cf6e33 Mirror 25: 0x21cf6e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000926 1779976171.075299 trunc from 0x040000 to 0x03ce25 (0x31da bytes) 000927 1779976171.139166 mapwrite 0x0071ac thru 0x0102aa (0x90ff bytes) 000928 1779976171.210579 punch from 0x02627e to 0x032938 (0xc6bb bytes) 000929 1779976171.218439 write 0x032b19 thru 0x03baa0 (0x8f88 bytes) 000930 1779976171.234100 MIRROR_RESYNC 7 mirrors 000931 1779976173.236591 MIRROR_EXTEND to 8 mirrors 000932 1779976174.038531 fallocate 0x03a36e thru 0x03a683 (0x316 bytes) 000933 1779976174.086676 mapwrite 0x018026 thru 0x02293f (0xa91a bytes) 000934 1779976174.120758 mapwrite 0x0024f9 thru 0x005986 (0x348e bytes) 000935 1779976174.160497 mapread 0x0221c0 thru 0x02b6cf (0x9510 bytes) 000936 1779976174.181698 write 0x007dc6 thru 0x00a17f (0x23ba bytes) 000937 1779976174.188102 read 0x039411 thru 0x03ce25 (0x3a15 bytes) 000938 1779976174.198650 trunc from 0x02a785 to 0x02b032 (0x8ae bytes) 000939 1779976174.203758 trunc from 0x03ce26 to 0x017bd4 (0x25251 bytes) 000940 1779976174.228413 mapread 0x0127ba thru 0x015371 (0x2bb8 bytes) 000941 1779976174.235735 write 0x024dcc thru 0x025b18 (0xd4d bytes) 000943 1779976174.237759 read 0x01e677 thru 0x02344e (0x4dd8 bytes) 000944 1779976174.245641 mapwrite 0x024374 thru 0x03054b (0xc1d8 bytes) 000945 1779976174.287550 write 0x022a96 thru 0x02e3de (0xb949 bytes) 000946 1779976174.289958 trunc from 0x03054c to 0x02ae65 (0x56e6 bytes) 000947 1779976174.304716 read 0x018123 thru 0x01933b (0x1219 bytes) 000948 1779976174.305584 trunc from 0x020716 to 0x028001 (0x78ec bytes) 000949 1779976174.320293 punch from 0x021107 to 0x027817 (0x6711 bytes) 000950 1779976174.326852 mapread 0x026743 thru 0x02ae65 (0x4723 bytes) 000951 1779976174.344863 punch from 0x0003a5 to 0x0102bb (0xff17 bytes) 000952 1779976174.353606 write 0x00e230 thru 0x00f211 (0xfe2 bytes) 000954 1779976174.360372 trunc from 0x01f77b to 0x02257d (0x2e03 bytes) 000955 1779976174.364464 punch from 0x00c28b to 0x00ff12 (0x3c88 bytes) 000956 1779976174.375151 trunc from 0x02ae66 to 0x009659 (0x2180c bytes) 000957 1779976174.387149 mapwrite 0x011867 thru 0x01ee3f (0xd5d9 bytes) 000958 1779976174.568248 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ee40): Mirror 16: 0x672dc71f (stale) Mirror 17: 0x672dc71f (stale) Mirror 22: 0x672dc71f (stale) Mirror 23: 0x672dc71f (stale) Mirror 24: 0x672dc71f (stale) Mirror 26: 0x672dc71f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee40): Mirror 16: 0x672dc71f (stale) Mirror 17: 0x672dc71f (stale) Mirror 22: 0x672dc71f (stale) Mirror 23: 0x672dc71f (stale) Mirror 24: 0x672dc71f (stale) Mirror 26: 0x672dc71f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee40: skipped 000959 1779976175.087395 punch from 0x011911 to 0x011af2 (0x1e2 bytes) 000960 1779976175.093643 fallocate 0x01b43d thru 0x027986 (0xc54a bytes) 000963 1779976175.100541 trunc from 0x027987 to 0x02ea94 (0x710e bytes) 000964 1779976175.136208 mapwrite 0x02603c thru 0x02edca (0x8d8f bytes) 000965 1779976175.212736 trunc from 0x02edcb to 0x016627 (0x187a3 bytes) 000967 1779976175.253997 trunc from 0x016628 to 0x011df4 (0x4833 bytes) 000969 1779976175.268725 mapread 0x006d45 thru 0x011df4 (0xb0b0 bytes) 000970 1779976175.289087 read 0x00a968 thru 0x00bdc3 (0x145c bytes) 000971 1779976175.290249 mapwrite 0x02ec37 thru 0x0345bb (0x5985 bytes) 000972 1779976175.348251 fallocate 0x032464 thru 0x03e3c3 (0xbf60 bytes) 000973 1779976175.360807 mapread 0x010048 thru 0x015766 (0x571f bytes) 000974 1779976175.388290 trunc from 0x0345bc to 0x0233ec (0x111cf bytes) 000975 1779976175.426228 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x233ed): Mirror 16: 0x931c95f8 (stale) Mirror 17: 0x931c95f8 (stale) Mirror 23: 0x931c95f8 (stale) Mirror 24: 0x931c95f8 (stale) Mirror 26: 0x931c95f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233ed): Mirror 16: 0x931c95f8 (stale) Mirror 17: 0x931c95f8 (stale) Mirror 23: 0x931c95f8 (stale) Mirror 24: 0x931c95f8 (stale) Mirror 26: 0x931c95f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233ed: skipped 000976 1779976175.939227 trunc from 0x0199e9 to 0x019e65 (0x47d bytes) 000978 1779976175.945734 trunc from 0x0233ed to 0x0004e3 (0x22f09 bytes) 000979 1779976175.969200 read 0x0000bf thru 0x0004e3 (0x425 bytes) 000980 1779976175.980717 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4e4): Mirror 16: 0x42ae2b30 (stale) Mirror 17: 0x42ae2b30 (stale) Mirror 23: 0x42ae2b30 (stale) Mirror 26: 0x42ae2b30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4e4): Mirror 16: 0x42ae2b30 (stale) Mirror 17: 0x42ae2b30 (stale) Mirror 23: 0x42ae2b30 (stale) Mirror 26: 0x42ae2b30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4e4: skipped 000981 1779976176.370657 mapwrite 0x017cc3 thru 0x027803 (0xfb41 bytes) 000982 1779976176.472353 read 0x017fa0 thru 0x01ab22 (0x2b83 bytes) 000983 1779976176.473802 write 0x02588e thru 0x02c964 (0x70d7 bytes) 000984 1779976176.476671 mapread 0x00a69f thru 0x011ab0 (0x7412 bytes) 000985 1779976176.488564 trunc from 0x005398 to 0x01052d (0xb196 bytes) 000986 1779976176.493973 mapwrite 0x0272dc thru 0x035347 (0xe06c bytes) 000987 1779976176.546140 mapread 0x001183 thru 0x00bafc (0xa97a bytes) 000988 1779976176.561274 fallocate 0x003222 thru 0x007ad3 (0x48b2 bytes) 000989 1779976176.567188 fallocate 0x002531 thru 0x00dcf2 (0xb7c2 bytes) 000990 1779976176.572996 write 0x0007d0 thru 0x00a4be (0x9cef bytes) 000991 1779976176.579999 trunc from 0x0238c9 to 0x024733 (0xe6b bytes) 000992 1779976176.586203 fallocate 0x000c48 thru 0x00b00c (0xa3c5 bytes) 000993 1779976176.591551 fallocate 0x006e5c thru 0x009fb3 (0x3158 bytes) 000994 1779976176.601656 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35348): Mirror 16: 0x82af9631 (stale) Mirror 17: 0x82af9631 (stale) Mirror 23: 0x82af9631 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35348): Mirror 16: 0x82af9631 (stale) Mirror 17: 0x82af9631 (stale) Mirror 23: 0x82af9631 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35348: skipped 000995 1779976177.073186 mapwrite 0x03f217 thru 0x03ffff (0xde9 bytes) 000996 1779976177.114694 mapread 0x0387a8 thru 0x03d4a5 (0x4cfe bytes) 000997 1779976177.130613 write 0x01eb85 thru 0x02e027 (0xf4a3 bytes) 000998 1779976177.144246 trunc from 0x040000 to 0x03d35e (0x2ca1 bytes) 000999 1779976177.164630 trunc from 0x02fc84 to 0x03d35e (0xd6db bytes) 001000 1779976177.170174 write 0x01c5d6 thru 0x0264fb (0x9f26 bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 09:49:43 (1779976183) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1c8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 09:49:52 (1779976192) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219391 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154765 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:50:00 (1779976200) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2418400 .. Extending file size to 2384192 .. Extending file size to 6116928 .. Extending file size to 4268096 .. Extending file size to 7963936 .. Extending file size to 8714272 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8901184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9143456 .. Extending file size to 8838304 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9110048 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 59790 59791 59792 59793 59797 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) 192.168.206.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:51:20 (1779976280) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7827840 .. Extending file size to 2262752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6607712 .. Extending file size to 8698528 .. Extending file size to 8553056 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9280992 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9028672 .. Extending file size to 9422656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61636 61637 61638 61642 61643 done 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) 192.168.206.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:52:42 (1779976362) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2296 1269008 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25268 3581752 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19092 3587928 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44360 7169680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2296 1269008 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25268 3581752 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19092 3587928 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44360 7169680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183612 s, 11.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277796 s, 7.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbd1b7ba6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:53:02 (1779976382) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0728376 s, 28.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 09:53:10 (1779976390) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:53:20 (1779976400) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113705 s, 18.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:53:29 (1779976409) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0867582 s, 24.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:53:38 (1779976418) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109137 s, 19.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:10 (1779976450) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105596 s, 9.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:56 (1779976496) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250286 s, 8.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:55:47 (1779976547) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg631-server: Pool lustre.test_204f created Adding targets to pool oleg631-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg631-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104223 s, 20.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg631-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg631-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg631-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:03 (1779976623) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:11 (1779976631) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:19 (1779976639) oleg631-server: Pool lustre.test_206 created Adding targets to pool oleg631-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg631-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg631-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg631-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg631-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:57:47 (1779976667) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711992 s, 14.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042933 s, 24.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xadb62408 Mirror 3: 0xadb62408 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:57:59 (1779976679) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:58:03 (1779976683) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:58:06 (1779976686) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0438313 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x740000 Read this (4 bytes): BBBB 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:58:41 (1779976721) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0152506 s, 537 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:58:50 (1779976730) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037147 s, 28.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x57 lmm_fid: [0x200000405:0x57:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x57 lmm_fid: [0x200000405:0x57:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-28 09:58:52.000000000 -0400 Modify: 2026-05-28 09:58:52.000000000 -0400 Change: 2026-05-28 09:58:54.000000000 -0400 Birth: 2026-05-28 09:58:52.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:58:58 (1779976738) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695781 s, 15.1 MB/s oleg631-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg631-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg631-server: O/280000401/d17/b1: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:59:19 (1779976759) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.142121 s, 288 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0178072 s, 230 kB/s size after second write -rw-r--r-- 1 root root 4096 May 28 09:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 28 09:59 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:59:26 (1779976766) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:59:34 (1779976774) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1848 sec =========== 09:59:42 (1779976782) === sanity-flr: start cleanup 09:59:44 (1779976784) === === sanity-flr: finish cleanup 09:59:52 (1779976792) ===