-----============= acceptance-small: sanity-flr ============----- Tue Nov 25 16:26:45 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg146-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:26:54 (1764106014) === oleg146-client.virtnet: executing check_config_client /mnt/lustre oleg146-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg146-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0db86d81000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0db86d81000.idle_timeout=debug disable quota as required oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:27:09 (1764106029) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8186] == sanity-flr test 0a: lfs mirror create with -N option == 16:27:10 (1764106030) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:27:17 (1764106037) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:27:18 (1764106038) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:27:20 (1764106040) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:27:32 (1764106052) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:27:34 (1764106054) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 16:27:35 (1764106055) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa0db86d81000.stats=clear osc.lustre-OST0001-osc-ffffa0db86d81000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:27:39 (1764106059) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:27:45 (1764106065) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:27:49 (1764106069) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:27:51 (1764106071) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 16:27:55 (1764106075) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:27:58 (1764106078) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:28:02 (1764106082) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:28:06 (1764106086) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:28:11 (1764106091) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:28:14 (1764106094) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.364097 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438754 s, 23.9 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:28:18 (1764106098) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.18646 s, 90.0 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:28:22 (1764106102) Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (37s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:28:59 (1764106139) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0973588 s, 43.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:29:14 (1764106154) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (33s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:29:47 (1764106187) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30913 s, 40.0 MB/s -rw-r--r-- 1 root root 52428800 Nov 25 16:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.525s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.452s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg146-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg146-server oleg146-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.497s user 0m0.002s sys 0m0.186s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg146-server: oleg146-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg146-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg146-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0db86d81000.ost_server_uuid in FULL state after 0 sec PASS 33b (40s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:30:27 (1764106227) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:30:28 (1764106228) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:30:30 (1764106230) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 16:30:31 (1764106231) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 16:30:35 (1764106235) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.22649 s, 55.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:30:40 (1764106240) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154129 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224078 s, 46.8 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:30:44 (1764106244) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:30:48 (1764106248) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00347179 s, 295 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108179 s, 947 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000742047 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000815458 s, 1.3 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 16:30:52 (1764106252) create test files with size 15733765 .. cec72ad0692cf65678bbeeaedf86c123 - f6f6bb7c1a2e556ac651dfb294870a4e - cc0d8d81fe1e5abcb37c04d6ec121e1b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 16:31:01 (1764106261) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:31:08 (1764106268) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:31:11 (1764106271) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290295 s, 36.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155763 s, 67.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 16:31:15 (1764106275) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0499073 s, 42.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.15661 s, 26.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0903648 s, 23.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 16:31:21 (1764106281) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 16:31:22 (1764106282) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:31:24 (1764106284) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164582 s, 63.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 16:31:27 (1764106287) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.10418 s, 30.2 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:31:30 (1764106290) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:31:34 (1764106294) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.124728 s, 84.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:31:50 (1764106310) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa0db92f99800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.13845 s, 75.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg146-client.virtnet /mnt/lustre (opts:) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa0db82f79000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:31:56 (1764106316) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:32:12 (1764106332) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:32:26 (1764106346) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 16:32:28 (1764106348) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.11025 s, 28.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0763462 s, 41.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd0456cdd (stale) Mirror 2: 0xd0456cdd (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 4b0fb33a893dba4f6cb4aa2b6692edc3 - mirror 1 checksum: c0fed03e3c6da1c5ada74e6f8b176134 - mirror 2 checksum: 4b0fb33a893dba4f6cb4aa2b6692edc3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 25 16:32:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 25 16:32:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4b0fb33a893dba4f6cb4aa2b6692edc3 - mirror 1 checksum: c0fed03e3c6da1c5ada74e6f8b176134 - mirror 2 checksum: c0fed03e3c6da1c5ada74e6f8b176134 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 25 16:32:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 25 16:32:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:32:33 (1764106353) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018681 s, 56.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:32:36 (1764106356) Failing mds1 on oleg146-server Stopping /mnt/lustre-mds1 (opts:) on oleg146-server 16:32:45 (1764106365) shut down facet: mds1 facet_host: oleg146-server facet_failover_host: oleg146-server Failover mds1 to oleg146-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 16:33:00 (1764106380) targets are mounted 16:33:00 (1764106380) facet_failover done oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191463 s, 54.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x58:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:33:09 (1764106389) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0198329 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0084039 s, 2.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:33:13 (1764106393) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0240278 s, 852 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00701871 s, 2.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:33:18 (1764106398) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00346313 s, 4.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:33:21 (1764106401) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0264031 s, 776 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:33:25 (1764106405) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00414279 s, 989 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187481 s, 218 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:33:28 (1764106408) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:33:37 (1764106417) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:33:58 (1764106438) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:34:16 (1764106456) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:34:35 (1764106475) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 16:34:38 (1764106478) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55853 mirror split pid 55854 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55853 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55854 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 16:35:41 (1764106541) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6542 skipping zero length zero range skipping zero length zero range skipping zero length punch hole truncating to largest ever: 0x146a3 000006 1764106542.653509 trunc from 00000000 to 0x0146a2 (0x146a3 bytes) fallocating to largest ever: 0x164e3 000008 1764106542.663925 fallocate 0x012e7d thru 0x0164e2 (0x3666 bytes) 000009 1764106542.667427 mapwrite 0x018905 thru 0x023388 (0xaa84 bytes) fallocating to largest ever: 0x30c3d 000011 1764106542.706801 fallocate 0x0230f7 thru 0x030c3c (0xdb46 bytes) 000012 1764106542.710276 read 0x00fe07 thru 0x012932 (0x2b2c bytes) 000014 1764106542.715252 trunc from 0x030c3d to 0x019a8c (0x171b0 bytes) 000015 1764106542.723405 punch from 0x006294 to 0x00d0b6 (0x6e23 bytes) 000016 1764106542.726745 mapread 0x00a39d thru 0x017030 (0xcc94 bytes) 000017 1764106542.734668 mapread 0x00b144 thru 0x00ed88 (0x3c45 bytes) 000018 1764106542.735718 trunc from 0x0037cc to 0x00ef9a (0xb7cf bytes) 000019 1764106542.739151 punch from 0x01737d to 0x019a8c (0x2710 bytes) 000020 1764106542.742254 fallocate 0x008655 thru 0x014d7c (0xc728 bytes) 000021 1764106542.744967 read 0x008b7b thru 0x00e349 (0x57cf bytes) 000022 1764106542.750003 write 0x006bd7 thru 0x00e7a8 (0x7bd2 bytes) 000023 1764106542.750949 trunc from 0x019a8d to 0x02a51f (0x10a93 bytes) truncating to largest ever: 0x3e139 000024 1764106542.758346 trunc from 0x02a520 to 0x03e138 (0x13c19 bytes) 000025 1764106542.764862 fallocate 0x0366b9 thru 0x038dc4 (0x270c bytes) 000026 1764106542.767797 write 0x03f298 thru 0x03ffff (0xd68 bytes) 000028 1764106542.768936 trunc from 0x00942f to 0x01543e (0xc010 bytes) 000030 1764106542.778138 write 0x000853 thru 0x00ef80 (0xe72e bytes) 000031 1764106542.784739 trunc from 0x040000 to 0x003a1e (0x3c5e1 bytes) 000032 1764106542.792821 trunc from 0x003a1f to 0x0074c3 (0x3aa5 bytes) 000034 1764106542.801875 trunc from 0x0074c4 to 0x037418 (0x2ff55 bytes) 000035 1764106542.811417 read 0x02f8c1 thru 0x037418 (0x7b58 bytes) 000036 1764106542.817630 write 0x015e87 thru 0x018aa3 (0x2c1d bytes) 000037 1764106542.818805 punch from 0x0023db to 0x0084e3 (0x6109 bytes) 000038 1764106542.826689 mapread 0x0092ce thru 0x00d2d5 (0x4008 bytes) 000040 1764106542.827981 trunc from 0x037419 to 0x00b044 (0x2c3d4 bytes) 000041 1764106542.838248 fallocate 0x01b21b thru 0x01eff1 (0x3dd7 bytes) 000042 1764106542.841943 read 0x016d35 thru 0x01c59d (0x5869 bytes) 000043 1764106542.848526 write 0x03a5f0 thru 0x03ffff (0x5a10 bytes) 000044 1764106542.849881 read 0x033195 thru 0x036642 (0x34ae bytes) 000045 1764106542.856465 mapwrite 0x01b86d thru 0x0260b7 (0xa84b bytes) 000046 1764106542.875111 write 0x03b0ef thru 0x03f0a9 (0x3fbb bytes) 000047 1764106542.875999 trunc from 0x02628b to 0x035386 (0xf0fc bytes) 000048 1764106542.881921 MIRROR_EXTEND to 2 mirrors 000049 1764106543.202967 read 0x019d6b thru 0x0205c0 (0x6856 bytes) 000050 1764106543.213236 trunc from 0x038243 to 0x03ffff (0x7dbd bytes) 000051 1764106543.235249 trunc from 0x021d33 to 0x0236d7 (0x19a5 bytes) 000052 1764106543.239646 mapwrite 0x0130fc thru 0x01b1f8 (0x80fd bytes) 000054 1764106543.260652 mapwrite 0x014143 thru 0x01f204 (0xb0c2 bytes) 000055 1764106543.281191 write 0x029aca thru 0x0379af (0xdee6 bytes) 000056 1764106543.295238 write 0x009af3 thru 0x00a05d (0x56b bytes) 000057 1764106543.300118 read 0x01bfc9 thru 0x01c59b (0x5d3 bytes) 000058 1764106543.300733 trunc from 0x040000 to 0x037a25 (0x85da bytes) 000059 1764106543.312651 mapwrite 0x017400 thru 0x022b1b (0xb71c bytes) 000060 1764106543.328286 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: 0x7838f7d5 (stale) CRC-32 checksum value for chunk [0, 0x37a26): Mirror 1: 0xd05dc9fc Mirror 3: 0xd05dc9fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a26: skipped 000061 1764106543.700981 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: 0x7838f7d5 (stale) CRC-32 checksum value for chunk [0, 0x37a26): Mirror 1: 0xd05dc9fc Mirror 3: 0xd05dc9fc Mirror 4: 0xd05dc9fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a26: skipped 000062 1764106544.070618 mapwrite 0x018c37 thru 0x021012 (0x83dc bytes) 000063 1764106544.120095 write 0x03b6eb thru 0x03ec44 (0x355a bytes) 000064 1764106544.123257 write 0x032932 thru 0x0379e1 (0x50b0 bytes) 000065 1764106544.128132 mapwrite 0x03c505 thru 0x03ffff (0x3afb bytes) 000066 1764106544.162500 fallocate 0x008c9f thru 0x012cf8 (0xa05a bytes) 000067 1764106544.166354 trunc from 0x035f5b to 0x038c83 (0x2d29 bytes) 000068 1764106544.169525 punch from 0x02f4c7 to 0x036005 (0x6b3f bytes) 000069 1764106544.172275 read 0x03afdd thru 0x03ffff (0x5023 bytes) 000070 1764106544.172925 mapread 0x036ec3 thru 0x03ffff (0x913d bytes) truncating to largest ever: 0x3e30a 000071 1764106544.178075 trunc from 0x040000 to 0x03e309 (0x1cf6 bytes) 000072 1764106544.188362 punch from 0x028821 to 0x02d8dd (0x50bd bytes) 000073 1764106544.194569 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e30a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e30a): Mirror 2: 0xb67a2b03 (stale) Mirror 3: 0x1196ec79 (stale) Mirror 4: 0x1196ec79 (stale) CRC-32 checksum value for chunk [0, 0x3e30a): Mirror 1: 0x1574e071 Mirror 5: 0x1574e071 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e30a: skipped 000074 1764106544.639044 fallocate 0x00a6ce thru 0x012965 (0x8298 bytes) 000075 1764106544.658235 mapread 0x029678 thru 0x035595 (0xbf1e bytes) 000076 1764106544.671572 write 0x0148d7 thru 0x01c388 (0x7ab2 bytes) 000079 1764106544.679020 read 0x025974 thru 0x034e15 (0xf4a2 bytes) 000080 1764106544.679553 mapread 0x00c4a6 thru 0x00ea65 (0x25c0 bytes) fallocating to largest ever: 0x40000 000081 1764106544.683748 fallocate 0x03d5b8 thru 0x03ffff (0x2a48 bytes) 000082 1764106544.686474 read 0x038b0f thru 0x0396f0 (0xbe2 bytes) 000083 1764106544.687077 write 0x0189c4 thru 0x0239aa (0xafe7 bytes) 000084 1764106544.691899 trunc from 0x01f370 to 0x026c98 (0x7929 bytes) 000085 1764106544.699144 write 0x00cf24 thru 0x01ce5b (0xff38 bytes) 000086 1764106544.700356 read 0x0229ed thru 0x02b6cd (0x8ce1 bytes) 000088 1764106544.705661 trunc from 0x03d368 to 0x03e309 (0xfa2 bytes) 000090 1764106544.709457 trunc from 0x03e30a to 0x029c0d (0x146fc bytes) 000091 1764106544.720052 punch from 0x008642 to 0x00f1d6 (0x6b95 bytes) 000092 1764106544.728184 mapwrite 0x03a9c5 thru 0x03d801 (0x2e3d bytes) 000093 1764106544.756262 punch from 0x002d42 to 0x010a2b (0xdcea bytes) 000094 1764106544.759498 punch from 0x035e18 to 0x0390f8 (0x32e1 bytes) 000096 1764106544.763675 mapwrite 0x033362 thru 0x03df48 (0xabe7 bytes) 000097 1764106544.822632 MIRROR_RESYNC 5 mirrors 000098 1764106545.554398 MIRROR_EXTEND to 6 mirrors 000099 1764106545.918965 write 0x028f29 thru 0x0384af (0xf587 bytes) 000100 1764106545.947703 read 0x0184b5 thru 0x01cc43 (0x478f bytes) 000101 1764106545.953805 trunc from 0x01ad22 to 0x02783d (0xcb1c bytes) 000103 1764106545.957890 trunc from 0x00b986 to 0x01106d (0x56e8 bytes) 000104 1764106545.961428 read 0x038bdf thru 0x03df48 (0x536a bytes) 000105 1764106545.970000 read 0x018522 thru 0x0254f5 (0xcfd4 bytes) 000106 1764106545.970647 write 0x03521c thru 0x03dab6 (0x889b bytes) 000107 1764106545.971827 mapread 0x0319f2 thru 0x0354b6 (0x3ac5 bytes) 000108 1764106545.972680 read 0x03a348 thru 0x03d9f9 (0x36b2 bytes) 000109 1764106545.973204 read 0x010a76 thru 0x01a918 (0x9ea3 bytes) 000110 1764106545.978267 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df49): Mirror 3: 0xee0b2961 (stale) Mirror 4: 0xee0b2961 (stale) Mirror 5: 0xee0b2961 (stale) Mirror 6: 0xee0b2961 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df49): Mirror 3: 0xee0b2961 (stale) Mirror 4: 0xee0b2961 (stale) Mirror 5: 0xee0b2961 (stale) Mirror 6: 0xee0b2961 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df49: skipped 000111 1764106546.156525 write 0x02925e thru 0x03121b (0x7fbe bytes) 000112 1764106546.164019 read 0x02b1eb thru 0x03880c (0xd622 bytes) 000113 1764106546.168692 trunc from 0x012f98 to 0x019e95 (0x6efe bytes) 000114 1764106546.171843 punch from 0x037b1f to 0x03df48 (0x642a bytes) 000115 1764106546.175043 punch from 0x016be9 to 0x01c03c (0x5454 bytes) 000117 1764106546.178940 mapwrite 0x01574a thru 0x01806a (0x2921 bytes) 000118 1764106546.196435 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3df49): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df49): Mirror 3: 0xee0b2961 (stale) Mirror 4: 0xee0b2961 (stale) Mirror 5: 0xee0b2961 (stale) Mirror 6: 0xee0b2961 (stale) CRC-32 checksum value for chunk [0, 0x3df49): Mirror 1: 0x5395a58e Mirror 7: 0x5395a58e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df49: skipped 000119 1764106546.600900 trunc from 0x03df49 to 0x03342f (0xab19 bytes) 000120 1764106546.637104 mapwrite 0x01a159 thru 0x028b76 (0xea1e bytes) 000122 1764106546.665494 mapwrite 0x03eed8 thru 0x03ffff (0x1128 bytes) 000123 1764106546.692089 trunc from 0x029e73 to 0x037c65 (0xddf3 bytes) 000124 1764106546.697833 mapwrite 0x021aeb thru 0x02f66b (0xdb81 bytes) 000125 1764106546.719185 fallocate 0x0296ec thru 0x038730 (0xf045 bytes) 000127 1764106546.728705 mapread 0x02b7d7 thru 0x030be9 (0x5413 bytes) 000128 1764106546.729572 read 0x01fe72 thru 0x029f1c (0xa0ab bytes) 000129 1764106546.730217 write 0x01c2a1 thru 0x01f5ba (0x331a bytes) 000130 1764106546.731469 mapwrite 0x0061d2 thru 0x00d96f (0x779e bytes) 000131 1764106546.750651 punch from 0x0276b9 to 0x028e02 (0x174a bytes) 000132 1764106546.754183 mapread 0x035c84 thru 0x03ffff (0xa37c bytes) 000133 1764106546.760927 mapwrite 0x007293 thru 0x00e267 (0x6fd5 bytes) 000134 1764106546.774314 read 0x03357a thru 0x03ffff (0xca86 bytes) 000135 1764106546.774945 mapread 0x01e148 thru 0x0219fc (0x38b5 bytes) 000137 1764106546.776185 write 0x01531c thru 0x019640 (0x4325 bytes) 000138 1764106546.777151 trunc from 0x040000 to 0x00c6a3 (0x3395c bytes) 000139 1764106546.790133 fallocate 0x01dd58 thru 0x01eb32 (0xddb bytes) 000140 1764106546.797789 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 3: 0x530ea0f3 (stale) Mirror 4: 0x530ea0f3 (stale) Mirror 5: 0x530ea0f3 (stale) Mirror 6: 0x530ea0f3 (stale) Mirror 7: 0x530ea0f3 (stale) CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 1: 0xdeef59b4 Mirror 8: 0xdeef59b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6a4: skipped 000141 1764106547.190929 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 3: 0x530ea0f3 (stale) Mirror 4: 0x530ea0f3 (stale) Mirror 5: 0x530ea0f3 (stale) Mirror 6: 0x530ea0f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 3: 0x530ea0f3 (stale) Mirror 4: 0x530ea0f3 (stale) Mirror 5: 0x530ea0f3 (stale) Mirror 6: 0x530ea0f3 (stale) CRC-32 checksum value for chunk [0, 0xc6a4): Mirror 1: 0xdeef59b4 Mirror 8: 0xdeef59b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6a4: skipped 000143 1764106547.503530 trunc from 0x00c6a4 to 0x01793d (0xb29a bytes) 000144 1764106547.530067 trunc from 0x01793e to 0x023cac (0xc36f bytes) 000145 1764106547.541439 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23cad): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23cad): Mirror 3: 0xe18e9629 (stale) Mirror 4: 0xe18e9629 (stale) Mirror 5: 0xe18e9629 (stale) Mirror 6: 0xe18e9629 (stale) Mirror 8: 0xf70e38cb (stale) CRC-32 checksum value for chunk [0, 0x23cad): Mirror 1: 0xf70e38cb Mirror 9: 0xf70e38cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23cad: skipped 000146 1764106547.961014 fallocate 0x00f195 thru 0x01df2d (0xed99 bytes) 000147 1764106547.981804 mapread 0x0080df thru 0x013f5c (0xbe7e bytes) 000148 1764106548.001330 write 0x031049 thru 0x03ffff (0xefb7 bytes) 000149 1764106548.007215 read 0x003495 thru 0x00dd7f (0xa8eb bytes) 000150 1764106548.007989 fallocate 0x02be5d thru 0x0356e8 (0x988c bytes) 000151 1764106548.012558 trunc from 0x040000 to 0x03ee39 (0x11c6 bytes) 000152 1764106548.024753 read 0x022206 thru 0x02cef3 (0xacee bytes) 000153 1764106548.030536 mapwrite 0x02703f thru 0x0294ad (0x246f bytes) 000154 1764106548.042048 mapread 0x02eaeb thru 0x030a3d (0x1f53 bytes) 000155 1764106548.045536 MIRROR_RESYNC 7 mirrors 000156 1764106548.953284 write 0x027f38 thru 0x02c30c (0x43d5 bytes) 000157 1764106548.979254 mapwrite 0x02e497 thru 0x037527 (0x9091 bytes) 000158 1764106549.007785 MIRROR_RESYNC 7 mirrors 000159 1764106549.683635 punch from 0x010c50 to 0x010e47 (0x1f8 bytes) 000161 1764106549.703262 trunc from 0x02f06d to 0x02f5a5 (0x539 bytes) 000162 1764106549.707211 trunc from 0x03ee3a to 0x013b70 (0x2b2c9 bytes) 000163 1764106549.718325 read 0x0123f1 thru 0x013b70 (0x1780 bytes) 000164 1764106549.724926 MIRROR_RESYNC 7 mirrors 000165 1764106550.386729 mapwrite 0x03d74e thru 0x03ffff (0x28b2 bytes) 000166 1764106550.434201 write 0x025199 thru 0x030dd8 (0xbc40 bytes) 000167 1764106550.444721 write 0x0214d3 thru 0x02c56a (0xb098 bytes) 000168 1764106550.446064 punch from 0x01c828 to 0x01c99f (0x178 bytes) 000169 1764106550.451692 mapread 0x02a889 thru 0x02e2a1 (0x3a19 bytes) 000170 1764106550.452543 fallocate 0x025cb7 thru 0x030d27 (0xb071 bytes) 000171 1764106550.458082 trunc from 0x010ccb to 0x01ece7 (0xe01d bytes) 000172 1764106550.461262 read 0x038d41 thru 0x0398d9 (0xb99 bytes) 000174 1764106550.462418 mapread 0x0292bc thru 0x02f2ca (0x600f bytes) 000175 1764106550.463256 trunc from 0x040000 to 0x0009f8 (0x3f607 bytes) 000177 1764106550.484216 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x9f9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f9): Mirror 3: 0xaad8816 (stale) Mirror 4: 0xaad8816 (stale) Mirror 5: 0xaad8816 (stale) Mirror 6: 0xaad8816 (stale) Mirror 8: 0xaad8816 (stale) Mirror 9: 0xaad8816 (stale) CRC-32 checksum value for chunk [0, 0x9f9): Mirror 1: 0xaad8816 Mirror 10: 0xaad8816 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f9: skipped 000178 1764106550.860647 read 0x000502 thru 0x0009f8 (0x4f7 bytes) 000179 1764106550.870733 mapwrite 0x032872 thru 0x03a3f4 (0x7b83 bytes) 000180 1764106550.931986 mapwrite 0x01ec41 thru 0x0252ee (0x66ae bytes) 000181 1764106550.950835 write 0x02049f thru 0x024687 (0x41e9 bytes) 000182 1764106550.951965 read 0x0297d0 thru 0x031b77 (0x83a8 bytes) 000183 1764106550.957391 trunc from 0x005a3f to 0x014b33 (0xf0f5 bytes) 000184 1764106550.960020 mapread 0x038490 thru 0x03a3f4 (0x1f65 bytes) 000185 1764106550.960880 punch from 0x01b6b6 to 0x026325 (0xac70 bytes) 000186 1764106550.967338 trunc from 0x03a3f5 to 0x02a61e (0xfdd6 bytes) 000187 1764106550.977058 trunc from 0x009f90 to 0x00e587 (0x45f8 bytes) 000188 1764106550.979568 trunc from 0x010c9b to 0x01a50f (0x9875 bytes) 000189 1764106550.982130 trunc from 0x02a61f to 0x03321f (0x8c01 bytes) 000190 1764106550.988145 trunc from 0x033220 to 0x01355f (0x1fcc0 bytes) 000191 1764106550.994706 fallocate 0x014191 thru 0x022f89 (0xedf9 bytes) 000192 1764106550.997241 mapread 0x016231 thru 0x022f89 (0xcd59 bytes) 000193 1764106551.034555 read 0x01ea52 thru 0x020ea1 (0x2450 bytes) 000194 1764106551.035232 mapwrite 0x038129 thru 0x03da93 (0x596b bytes) 000195 1764106551.070201 trunc from 0x03da94 to 0x02f500 (0xe593 bytes) 000196 1764106551.079372 write 0x00b259 thru 0x00e38f (0x3137 bytes) 000197 1764106551.080444 read 0x007c5f thru 0x00dfb0 (0x6352 bytes) 000198 1764106551.085142 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f501): Mirror 3: 0x9e5a8344 (stale) Mirror 4: 0x9e5a8344 (stale) Mirror 5: 0x9e5a8344 (stale) Mirror 6: 0x9e5a8344 (stale) Mirror 9: 0x9e5a8344 (stale) Mirror 10: 0xb614b17c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f501): Mirror 3: 0x9e5a8344 (stale) Mirror 4: 0x9e5a8344 (stale) Mirror 5: 0x9e5a8344 (stale) Mirror 6: 0x9e5a8344 (stale) Mirror 9: 0x9e5a8344 (stale) Mirror 10: 0xb614b17c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f501: skipped 000199 1764106551.321428 trunc from 0x01a093 to 0x01d026 (0x2f94 bytes) 000200 1764106551.324806 write 0x034c5e thru 0x03ffff (0xb3a2 bytes) 000201 1764106551.328942 read 0x01a350 thru 0x01af77 (0xc28 bytes) 000202 1764106551.332707 mapread 0x01c076 thru 0x02789c (0xb827 bytes) 000203 1764106551.352114 trunc from 0x040000 to 0x007862 (0x3879d bytes) 000204 1764106551.360791 trunc from 0x007863 to 0x0114e2 (0x9c80 bytes) 000206 1764106551.369188 mapwrite 0x03c2e7 thru 0x03ffff (0x3d19 bytes) 000207 1764106551.388971 fallocate 0x039835 thru 0x03dc24 (0x43f0 bytes) 000208 1764106551.392080 mapwrite 0x00902c thru 0x00c1f8 (0x31cd bytes) 000209 1764106551.404768 fallocate 0x032236 thru 0x0327fd (0x5c8 bytes) 000210 1764106551.408027 mapread 0x001d5c thru 0x0045b9 (0x285e bytes) 000211 1764106551.412080 fallocate 0x02df6f thru 0x0388ee (0xa980 bytes) 000212 1764106551.416365 mapwrite 0x016892 thru 0x01d565 (0x6cd4 bytes) 000213 1764106551.435782 mapwrite 0x001b84 thru 0x00bb1b (0x9f98 bytes) 000214 1764106551.449211 punch from 0x03d9d7 to 0x03ffff (0x2629 bytes) 000215 1764106551.454755 trunc from 0x040000 to 0x023d0e (0x1c2f1 bytes) 000216 1764106551.466675 punch from 0x017f92 to 0x01b6e5 (0x3754 bytes) 000217 1764106551.470370 mapread 0x01acaa thru 0x023d0e (0x9065 bytes) 000218 1764106551.478044 mapwrite 0x025e76 thru 0x02eded (0x8f78 bytes) 000219 1764106551.505618 write 0x02b4b7 thru 0x037df3 (0xc93d bytes) 000220 1764106551.507067 trunc from 0x005d57 to 0x00ea27 (0x8cd1 bytes) 000221 1764106551.510246 mapread 0x025066 thru 0x031245 (0xc1e0 bytes) 000222 1764106551.511171 write 0x00f071 thru 0x0146ef (0x567f bytes) 000224 1764106551.512634 trunc from 0x037df4 to 0x03b873 (0x3a80 bytes) 000225 1764106551.522712 MIRROR_RESYNC 7 mirrors 000226 1764106552.573563 trunc from 0x03b874 to 0x0132d2 (0x285a1 bytes) 000227 1764106552.594236 fallocate 0x03fb4b thru 0x03ffff (0x4b5 bytes) 000228 1764106552.602086 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x132d3): Mirror 4: 0xb1026e04 (stale) Mirror 5: 0xb1026e04 (stale) Mirror 6: 0xb1026e04 (stale) Mirror 9: 0xb1026e04 (stale) Mirror 10: 0xb1026e04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132d3): Mirror 4: 0xb1026e04 (stale) Mirror 5: 0xb1026e04 (stale) Mirror 6: 0xb1026e04 (stale) Mirror 9: 0xb1026e04 (stale) Mirror 10: 0xb1026e04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132d3: skipped 000229 1764106552.791181 punch from 0x00030a to 0x003cfc (0x39f3 bytes) 000230 1764106552.793414 trunc from 0x006ed8 to 0x011899 (0xa9c2 bytes) 000231 1764106552.795903 trunc from 0x0132d3 to 0x00ba2c (0x78a6 bytes) 000232 1764106552.805624 trunc from 0x00ba2d to 0x019cfe (0xe2d2 bytes) 000233 1764106552.815512 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19cff): Mirror 4: 0x72a98191 (stale) Mirror 6: 0x72a98191 (stale) Mirror 9: 0x72a98191 (stale) Mirror 10: 0x72a98191 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cff): Mirror 4: 0x72a98191 (stale) Mirror 6: 0x72a98191 (stale) Mirror 9: 0x72a98191 (stale) Mirror 10: 0x72a98191 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cff: skipped 000234 1764106552.984319 read 0x00bc9a thru 0x00cc3f (0xfa6 bytes) 000235 1764106552.987612 fallocate 0x005abf thru 0x009f28 (0x446a bytes) 000236 1764106552.992324 trunc from 0x019cff to 0x00ffde (0x9d20 bytes) 000237 1764106553.004621 MIRROR_RESYNC 5 mirrors 000238 1764106553.464570 read 0x0068b5 thru 0x00ff89 (0x96d5 bytes) 000239 1764106553.469960 write 0x020b45 thru 0x0240bd (0x3579 bytes) 000240 1764106553.488799 punch from 0x019554 to 0x01da41 (0x44ee bytes) 000241 1764106553.491704 trunc from 0x0240be to 0x0169f6 (0xd6c7 bytes) 000242 1764106553.505091 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x169f7): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169f7): Mirror 4: 0x2ed8e306 (stale) Mirror 6: 0x2ed8e306 (stale) Mirror 9: 0x2ed8e306 (stale) Mirror 10: 0x2ed8e306 (stale) CRC-32 checksum value for chunk [0, 0x169f7): Mirror 1: 0x2ed8e306 Mirror 11: 0x2ed8e306 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169f7: skipped 000243 1764106553.912790 trunc from 0x0169f7 to 0x0381cf (0x217d9 bytes) 000244 1764106553.943768 trunc from 0x0381d0 to 0x02e734 (0x9a9b bytes) 000245 1764106553.955094 mapwrite 0x031ad2 thru 0x0356e2 (0x3c11 bytes) 000246 1764106553.991791 mapwrite 0x00a598 thru 0x016c8b (0xc6f4 bytes) 000247 1764106554.019161 mapwrite 0x0216d4 thru 0x02a5a3 (0x8ed0 bytes) 000248 1764106554.038843 punch from 0x029e9b to 0x02cff3 (0x3159 bytes) 000249 1764106554.042042 trunc from 0x0356e3 to 0x02dae2 (0x7c00 bytes) 000250 1764106554.050048 read 0x02c1b6 thru 0x02dae2 (0x192d bytes) 000252 1764106554.056480 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dae3): Mirror 6: 0xc023163f (stale) Mirror 9: 0xc023163f (stale) Mirror 10: 0xc023163f (stale) Mirror 11: 0xc023163f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dae3): Mirror 6: 0xc023163f (stale) Mirror 9: 0xc023163f (stale) Mirror 10: 0xc023163f (stale) Mirror 11: 0xc023163f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dae3: skipped 000254 1764106554.255549 mapread 0x010389 thru 0x010749 (0x3c1 bytes) 000255 1764106554.259251 fallocate 0x031ff0 thru 0x03d518 (0xb529 bytes) 000256 1764106554.262783 trunc from 0x03d519 to 0x003b6b (0x399ad bytes) 000257 1764106554.275198 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6c): Mirror 6: 0x55d48491 (stale) Mirror 9: 0x55d48491 (stale) Mirror 10: 0x55d48491 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6c): Mirror 6: 0x55d48491 (stale) Mirror 9: 0x55d48491 (stale) Mirror 10: 0x55d48491 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6c: skipped 000258 1764106554.404448 trunc from 0x003b6c to 0x0315c8 (0x2da5d bytes) 000259 1764106554.416292 read 0x02dd67 thru 0x0315c8 (0x3862 bytes) 000260 1764106554.421147 read 0x003208 thru 0x00751b (0x4314 bytes) 000261 1764106554.429714 write 0x01e046 thru 0x01f052 (0x100d bytes) 000262 1764106554.430622 write 0x01c2d0 thru 0x0290b7 (0xcde8 bytes) 000263 1764106554.431893 mapread 0x01379f thru 0x01d3b8 (0x9c1a bytes) 000264 1764106554.433221 trunc from 0x00f546 to 0x0156fa (0x61b5 bytes) 000265 1764106554.437814 read 0x027135 thru 0x0315c8 (0xa494 bytes) 000266 1764106554.438448 read 0x0080b4 thru 0x017d28 (0xfc75 bytes) 000267 1764106554.445034 mapread 0x001152 thru 0x0025c3 (0x1472 bytes) 000270 1764106554.446911 trunc from 0x00de61 to 0x019159 (0xb2f9 bytes) 000271 1764106554.455171 MIRROR_RESYNC 4 mirrors 000272 1764106554.886717 trunc from 0x019ca5 to 0x027c4a (0xdfa6 bytes) 000273 1764106554.902606 mapwrite 0x009a37 thru 0x00e4be (0x4a88 bytes) 000274 1764106554.925652 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x315c9): Mirror 9: 0xa3fcd8eb (stale) Mirror 10: 0xa3fcd8eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x315c9): Mirror 9: 0xa3fcd8eb (stale) Mirror 10: 0xa3fcd8eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x315c9: skipped 000275 1764106555.063563 mapread 0x0071ac thru 0x00e442 (0x7297 bytes) 000276 1764106555.073650 mapread 0x003f6f thru 0x007a31 (0x3ac3 bytes) 000277 1764106555.074367 write 0x031f14 thru 0x035202 (0x32ef bytes) 000278 1764106555.084000 MIRROR_RESYNC 3 mirrors 000279 1764106555.525892 mapwrite 0x03ac6b thru 0x03ffff (0x5395 bytes) 000280 1764106555.577726 fallocate 0x01488a thru 0x017f6f (0x36e6 bytes) 000282 1764106555.581848 read 0x03503f thru 0x035e43 (0xe05 bytes) 000283 1764106555.582477 write 0x01ed00 thru 0x027cbf (0x8fc0 bytes) 000286 1764106555.592112 trunc from 0x03eea8 to 0x03ffff (0x1158 bytes) 000287 1764106555.597686 MIRROR_RESYNC 3 mirrors 000289 1764106556.226466 fallocate 0x0120e1 thru 0x018749 (0x6669 bytes) 000290 1764106556.252849 mapwrite 0x017925 thru 0x01d23a (0x5916 bytes) 000291 1764106556.279809 punch from 0x013321 to 0x01c04a (0x8d2a bytes) 000292 1764106556.285440 punch from 0x03024b to 0x0382c6 (0x807c bytes) 000293 1764106556.290111 mapread 0x02602e thru 0x0260d8 (0xab bytes) 000295 1764106556.300074 mapwrite 0x0270d5 thru 0x034c2a (0xdb56 bytes) 000296 1764106556.320032 trunc from 0x040000 to 0x00a641 (0x359be bytes) 000297 1764106556.336328 trunc from 0x0086a9 to 0x00a641 (0x1f99 bytes) 000298 1764106556.339996 punch from 0x003dd5 to 0x00a641 (0x686d bytes) 000299 1764106556.344649 mapread 0x002c0a thru 0x00a4a4 (0x789b bytes) 000300 1764106556.351310 punch from 0x009c77 to 0x00a347 (0x6d1 bytes) 000301 1764106556.354770 trunc from 0x005471 to 0x00a641 (0x51d1 bytes) 000302 1764106556.358455 write 0x032de1 thru 0x03ba16 (0x8c36 bytes) 000303 1764106556.360837 write 0x03e4f7 thru 0x03ffff (0x1b09 bytes) 000304 1764106556.362453 mapread 0x0345ca thru 0x03d413 (0x8e4a bytes) 000305 1764106556.367947 trunc from 0x040000 to 0x036d61 (0x929e bytes) 000306 1764106556.379034 fallocate 0x009e0b thru 0x00bed5 (0x20cb bytes) 000307 1764106556.382172 mapread 0x0265fd thru 0x02dbc0 (0x75c4 bytes) 000308 1764106556.388135 write 0x02fc2a thru 0x038561 (0x8938 bytes) 000309 1764106556.392846 mapread 0x01864f thru 0x02115b (0x8b0d bytes) 000310 1764106556.399072 fallocate 0x0188a1 thru 0x02419d (0xb8fd bytes) 000311 1764106556.405126 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38562): Mirror 10: 0xe8f7b5e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38562): Mirror 10: 0xe8f7b5e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38562: skipped 000312 1764106556.548865 write 0x0208a1 thru 0x02918a (0x88ea bytes) 000313 1764106556.558328 mapwrite 0x019772 thru 0x0278ae (0xe13d bytes) 000314 1764106556.577963 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38562): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38562): Mirror 10: 0xe8f7b5e4 (stale) CRC-32 checksum value for chunk [0, 0x38562): Mirror 1: 0xb299a337 Mirror 11: 0xb299a337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38562: skipped 000315 1764106556.997755 mapwrite 0x00eb70 thru 0x012622 (0x3ab3 bytes) 000316 1764106557.068692 read 0x00d0a0 thru 0x0143a0 (0x7301 bytes) 000318 1764106557.071650 punch from 0x031c83 to 0x038561 (0x68df bytes) 000319 1764106557.076147 fallocate 0x0256e6 thru 0x030f1d (0xb838 bytes) 000320 1764106557.081082 trunc from 0x038562 to 0x02fa7e (0x8ae3 bytes) 000321 1764106557.103178 read 0x0081cc thru 0x00f698 (0x74cd bytes) 000322 1764106557.103916 fallocate 0x01eaa2 thru 0x029c91 (0xb1f0 bytes) 000323 1764106557.107671 mapwrite 0x036fdf thru 0x03e718 (0x773a bytes) 000324 1764106557.141348 mapwrite 0x022115 thru 0x0262c2 (0x41ae bytes) 000325 1764106557.156555 trunc from 0x03e719 to 0x00f784 (0x2ef94 bytes) 000327 1764106557.170133 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf785): Mirror 11: 0x1a8814eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf785): Mirror 11: 0x1a8814eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf785: skipped 000329 1764106557.292028 read 0x000ee5 thru 0x00e41a (0xd536 bytes) 000330 1764106557.297660 mapread 0x006eaf thru 0x00c826 (0x5978 bytes) 000331 1764106557.299138 read 0x006c67 thru 0x00f784 (0x8b1e bytes) 000333 1764106557.310559 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf785): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf785): Mirror 11: 0x1a8814eb (stale) CRC-32 checksum value for chunk [0, 0xf785): Mirror 1: 0xa851e5e3 Mirror 12: 0xa851e5e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf785: skipped 000334 1764106557.719295 write 0x027b5e thru 0x030647 (0x8aea bytes) 000335 1764106557.747471 trunc from 0x00a33b to 0x017563 (0xd229 bytes) 000336 1764106557.751735 mapread 0x0296e7 thru 0x02e42c (0x4d46 bytes) 000337 1764106557.752897 read 0x002dcd thru 0x00cd97 (0x9fcb bytes) 000339 1764106557.759335 write 0x035297 thru 0x035fc3 (0xd2d bytes) 000340 1764106557.763336 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35fc4): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fc4): Mirror 11: 0xe62f33e4 (stale) Mirror 12: 0xd53156b4 (stale) CRC-32 checksum value for chunk [0, 0x35fc4): Mirror 1: 0xf162aab2 Mirror 13: 0xf162aab2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fc4: skipped 000341 1764106558.192281 fallocate 0x02d058 thru 0x035a35 (0x89de bytes) 000342 1764106558.213917 punch from 0x02f2ce to 0x035fc3 (0x6cf6 bytes) 000343 1764106558.217318 punch from 0x01b1c3 to 0x027cbf (0xcafd bytes) 000344 1764106558.220352 punch from 0x010cae to 0x01b2c0 (0xa613 bytes) 000345 1764106558.224483 fallocate 0x002ce1 thru 0x00ce13 (0xa133 bytes) 000346 1764106558.228384 mapread 0x00cf6b thru 0x0135b0 (0x6646 bytes) 000347 1764106558.241240 read 0x02d288 thru 0x035fc3 (0x8d3c bytes) 000349 1764106558.247322 trunc from 0x035fc4 to 0x037873 (0x18b0 bytes) 000350 1764106558.283043 mapread 0x02b25e thru 0x031c08 (0x69ab bytes) 000351 1764106558.288849 punch from 0x0324f0 to 0x035063 (0x2b74 bytes) 000352 1764106558.293783 trunc from 0x037874 to 0x03b23e (0x39cb bytes) 000353 1764106558.307094 punch from 0x00a513 to 0x00a900 (0x3ee bytes) 000354 1764106558.310947 mapread 0x01f704 thru 0x02007c (0x979 bytes) 000355 1764106558.317510 write 0x0043a0 thru 0x010b4f (0xc7b0 bytes) 000356 1764106558.324637 fallocate 0x01d1ee thru 0x01e83f (0x1652 bytes) 000357 1764106558.330070 fallocate 0x0268a1 thru 0x02bcc9 (0x5429 bytes) 000358 1764106558.333835 trunc from 0x00fc09 to 0x01cca6 (0xd09e bytes) 000359 1764106558.345540 punch from 0x020d16 to 0x02a830 (0x9b1b bytes) 000360 1764106558.350076 trunc from 0x0350d3 to 0x03a076 (0x4fa4 bytes) 000361 1764106558.354440 trunc from 0x03b23f to 0x020918 (0x1a926 bytes) 000362 1764106558.366498 write 0x021ed7 thru 0x022709 (0x833 bytes) 000363 1764106558.367610 punch from 0x019c26 to 0x022709 (0x8ae4 bytes) 000364 1764106558.380645 MIRROR_RESYNC 4 mirrors 000365 1764106558.784414 read 0x00b74b thru 0x019a86 (0xe33c bytes) 000366 1764106558.790521 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) 000367 1764106558.937686 mapread 0x005f76 thru 0x011141 (0xb1cc bytes) 000368 1764106558.954519 write 0x013ecd thru 0x01e407 (0xa53b bytes) 000369 1764106558.981893 trunc from 0x01d09f to 0x02189e (0x4800 bytes) 000370 1764106558.992665 MIRROR_RESYNC 3 mirrors 000371 1764106559.339315 trunc from 0x01325f to 0x020167 (0xcf09 bytes) 000372 1764106559.353059 trunc from 0x00af12 to 0x00d140 (0x222f bytes) 000373 1764106559.356392 trunc from 0x01eb01 to 0x022709 (0x3c09 bytes) 000374 1764106559.359210 fallocate 0x032550 thru 0x038112 (0x5bc3 bytes) 000375 1764106559.362078 trunc from 0x03269e to 0x038112 (0x5a75 bytes) 000376 1764106559.364442 punch from 0x01a135 to 0x025bfd (0xbac9 bytes) 000377 1764106559.366844 fallocate 0x01ede8 thru 0x02cbb2 (0xddcb bytes) 000378 1764106559.368967 fallocate 0x021ea3 thru 0x031cef (0xfe4d bytes) 000380 1764106559.373898 MIRROR_RESYNC 3 mirrors 000381 1764106559.808393 trunc from 0x006fda to 0x011984 (0xa9ab bytes) 000382 1764106559.833017 mapread 0x028d8e thru 0x03677a (0xd9ed bytes) 000383 1764106559.857510 punch from 0x020fbd to 0x030d2a (0xfd6e bytes) 000384 1764106559.862132 mapread 0x001455 thru 0x003d40 (0x28ec bytes) 000385 1764106559.870184 mapread 0x0279a4 thru 0x02f103 (0x7760 bytes) 000386 1764106559.876104 punch from 0x0094ad to 0x011da1 (0x88f5 bytes) 000387 1764106559.879532 read 0x0080e4 thru 0x00b310 (0x322d bytes) 000388 1764106559.883647 mapread 0x01e723 thru 0x02de42 (0xf720 bytes) 000390 1764106559.889035 read 0x034ef4 thru 0x038112 (0x321f bytes) 000391 1764106559.889662 trunc from 0x038113 to 0x02c87c (0xb896 bytes) 000392 1764106559.905255 mapread 0x018415 thru 0x023031 (0xac1d bytes) 000393 1764106559.906235 mapread 0x005dac thru 0x0065c8 (0x81d bytes) 000394 1764106559.906965 write 0x0014bc thru 0x0021c4 (0xd09 bytes) 000395 1764106559.907825 fallocate 0x01e2fd thru 0x02451d (0x6221 bytes) 000396 1764106559.913408 mapwrite 0x0227d5 thru 0x025035 (0x2861 bytes) 000397 1764106559.927422 punch from 0x011f01 to 0x01a222 (0x8322 bytes) 000398 1764106559.931058 write 0x03a318 thru 0x03ffff (0x5ce8 bytes) 000399 1764106559.932165 trunc from 0x040000 to 0x008ceb (0x37314 bytes) 000400 1764106559.942029 punch from 0x001a66 to 0x004b14 (0x30af bytes) 000401 1764106559.945907 trunc from 0x008cec to 0x02ff30 (0x27245 bytes) 000402 1764106559.955955 fallocate 0x0189ca thru 0x027f82 (0xf5b9 bytes) 000403 1764106559.960656 write 0x030a75 thru 0x03ffff (0xf58b bytes) 000404 1764106559.962184 fallocate 0x001b02 thru 0x00d13a (0xb639 bytes) 000405 1764106559.966026 trunc from 0x026915 to 0x02a8a3 (0x3f8f bytes) 000406 1764106559.970046 fallocate 0x00bf8a thru 0x016406 (0xa47d bytes) 000407 1764106559.973426 trunc from 0x007283 to 0x013e65 (0xcbe3 bytes) 000408 1764106559.977547 mapread 0x01a55b thru 0x01b307 (0xdad bytes) 000409 1764106559.986973 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd217ba86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd217ba86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1764106560.146951 punch from 0x009728 to 0x01507d (0xb956 bytes) 000411 1764106560.155950 MIRROR_RESYNC 2 mirrors 000412 1764106560.473439 trunc from 0x023e36 to 0x033289 (0xf454 bytes) 000413 1764106560.490793 write 0x0260c4 thru 0x02db29 (0x7a66 bytes) 000414 1764106560.498494 write 0x02df9d thru 0x033f57 (0x5fbb bytes) 000415 1764106560.503906 read 0x026e40 thru 0x0330e2 (0xc2a3 bytes) 000416 1764106560.504667 trunc from 0x0332eb to 0x03f054 (0xbd6a bytes) 000417 1764106560.515924 trunc from 0x040000 to 0x00f956 (0x306a9 bytes) 000418 1764106560.531199 fallocate 0x02fc3a thru 0x03e023 (0xe3ea bytes) 000419 1764106560.535680 read 0x005d71 thru 0x00f956 (0x9be6 bytes) 000420 1764106560.542561 trunc from 0x00c9ce to 0x00f956 (0x2f89 bytes) 000421 1764106560.548176 fallocate 0x00315f thru 0x00e2e3 (0xb185 bytes) 000422 1764106560.552177 punch from 0x0017c8 to 0x00373e (0x1f77 bytes) 000423 1764106560.556439 write 0x039dba thru 0x03ffff (0x6246 bytes) 000424 1764106560.557948 fallocate 0x031201 thru 0x0377a7 (0x65a7 bytes) 000425 1764106560.563802 fallocate 0x014e76 thru 0x01691c (0x1aa7 bytes) 000426 1764106560.569543 trunc from 0x037788 to 0x03d393 (0x5c0c bytes) 000428 1764106560.583177 fallocate 0x00ba59 thru 0x00e76a (0x2d12 bytes) 000429 1764106560.587174 mapread 0x037758 thru 0x03ffff (0x88a8 bytes) 000430 1764106560.593935 read 0x03b7d5 thru 0x03ffff (0x482b bytes) 000431 1764106560.594470 write 0x02cd07 thru 0x03cae5 (0xfddf bytes) 000432 1764106560.595526 trunc from 0x033f62 to 0x03a48c (0x652b bytes) 000433 1764106560.604661 mapwrite 0x018144 thru 0x0217b9 (0x9676 bytes) 000434 1764106560.617922 punch from 0x01f330 to 0x0239a9 (0x467a bytes) 000435 1764106560.621500 fallocate 0x01874a thru 0x026129 (0xd9e0 bytes) 000436 1764106560.625528 trunc from 0x0163e9 to 0x01a0e2 (0x3cfa bytes) 000437 1764106560.630231 trunc from 0x040000 to 0x01e0a5 (0x21f5a bytes) 000438 1764106560.640444 mapwrite 0x017dbe thru 0x022675 (0xa8b8 bytes) 000439 1764106560.674007 trunc from 0x01b691 to 0x022675 (0x6fe5 bytes) 000440 1764106560.679048 punch from 0x01510f to 0x022675 (0xd567 bytes) 000442 1764106560.683178 mapread 0x01ca45 thru 0x022675 (0x5c31 bytes) 000443 1764106560.705670 mapwrite 0x022469 thru 0x0246a0 (0x2238 bytes) 000444 1764106560.729479 write 0x00162b thru 0x007717 (0x60ed bytes) 000445 1764106560.730970 trunc from 0x0246a1 to 0x01c257 (0x8449 bytes) 000446 1764106560.746156 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) 000447 1764106560.848666 punch from 0x00e93e to 0x01050a (0x1bcd bytes) 000448 1764106560.853511 mapread 0x01350c thru 0x01c257 (0x8d4c bytes) 000449 1764106560.865282 write 0x0075e8 thru 0x014c2d (0xd646 bytes) 000450 1764106560.874506 fallocate 0x02f856 thru 0x03f28d (0xfa38 bytes) 000452 1764106560.879052 read 0x024c6e thru 0x0271c1 (0x2554 bytes) 000453 1764106560.883504 write 0x03c779 thru 0x03ffff (0x3887 bytes) 000454 1764106560.891599 fallocate 0x02e463 thru 0x03179a (0x3338 bytes) 000455 1764106560.895895 trunc from 0x040000 to 0x0068d8 (0x39727 bytes) 000456 1764106560.909403 trunc from 0x0068d9 to 0x02e000 (0x27728 bytes) 000457 1764106560.923477 mapwrite 0x011bb4 thru 0x0120ca (0x517 bytes) 000458 1764106560.944099 trunc from 0x01f62a to 0x024e4d (0x5824 bytes) 000459 1764106560.948548 mapwrite 0x03fab7 thru 0x03ffff (0x549 bytes) 000460 1764106560.977737 read 0x015ff0 thru 0x0256bd (0xf6ce bytes) 000461 1764106560.986015 trunc from 0x0183c8 to 0x01fc2a (0x7863 bytes) 000463 1764106560.990507 read 0x013099 thru 0x01d4c3 (0xa42b bytes) 000465 1764106561.002311 fallocate 0x0191c3 thru 0x01e351 (0x518f bytes) 000466 1764106561.008021 trunc from 0x00b509 to 0x00faaf (0x45a7 bytes) 000469 1764106561.012770 fallocate 0x0302d5 thru 0x032d22 (0x2a4e bytes) 000471 1764106561.017612 trunc from 0x01bb0a to 0x01ef13 (0x340a bytes) 000472 1764106561.022069 read 0x0309d8 thru 0x038c83 (0x82ac bytes) 000473 1764106561.022803 write 0x007b73 thru 0x00992b (0x1db9 bytes) 000474 1764106561.023933 mapwrite 0x013f0d thru 0x018d03 (0x4df7 bytes) 000475 1764106561.036051 read 0x0283bb thru 0x03732b (0xef71 bytes) 000476 1764106561.036812 punch from 0x00d38a to 0x01b15a (0xddd1 bytes) 000477 1764106561.041623 mapwrite 0x03f32a thru 0x03f8db (0x5b2 bytes) 000478 1764106561.050484 punch from 0x0274c7 to 0x0303fb (0x8f35 bytes) 000480 1764106561.054877 punch from 0x0050d6 to 0x007e0a (0x2d35 bytes) 000481 1764106561.059655 trunc from 0x012ea7 to 0x016caa (0x3e04 bytes) 000482 1764106561.063634 trunc from 0x040000 to 0x03aa6e (0x5591 bytes) 000483 1764106561.075023 mapread 0x010ca6 thru 0x02048c (0xf7e7 bytes) 000484 1764106561.082072 trunc from 0x03aa6f to 0x037409 (0x3665 bytes) 000485 1764106561.096932 MIRROR_EXTEND to 2 mirrors 000486 1764106561.454049 write 0x014ccb thru 0x022624 (0xd95a bytes) 000488 1764106561.478565 read 0x02c314 thru 0x035edb (0x9bc8 bytes) 000490 1764106561.483808 mapwrite 0x02bc96 thru 0x032026 (0x6391 bytes) 000491 1764106561.503841 read 0x023231 thru 0x032e26 (0xfbf6 bytes) 000492 1764106561.510068 trunc from 0x03740a to 0x007dc6 (0x2f643 bytes) 000494 1764106561.545196 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7dc7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7dc7): Mirror 2: 0x9f34eb4b (stale) CRC-32 checksum value for chunk [0, 0x7dc7): Mirror 1: 0x9f34eb4b Mirror 3: 0x9f34eb4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7dc7: skipped 000495 1764106561.958332 mapwrite 0x01da1a thru 0x02abbe (0xd1a5 bytes) 000496 1764106562.037906 write 0x025d8c thru 0x02fb7b (0x9df0 bytes) 000497 1764106562.043462 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fb7c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb7c): Mirror 2: 0xaa3b7e79 (stale) Mirror 3: 0x7f1b0e1e (stale) CRC-32 checksum value for chunk [0, 0x2fb7c): Mirror 1: 0xb8a35d2a Mirror 4: 0xb8a35d2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb7c: skipped 000498 1764106562.438413 MIRROR_RESYNC 4 mirrors 000499 1764106562.809522 read 0x01fe27 thru 0x024b28 (0x4d02 bytes) 000500 1764106562.814227 write 0x0307ff thru 0x03e0f7 (0xd8f9 bytes) 000501 1764106562.830722 mapwrite 0x016242 thru 0x016246 (0x5 bytes) 000502 1764106562.844956 mapread 0x012cb7 thru 0x0164ea (0x3834 bytes) 000503 1764106562.854455 fallocate 0x011717 thru 0x012466 (0xd50 bytes) 000504 1764106562.857737 fallocate 0x00d377 thru 0x01becd (0xeb57 bytes) 000505 1764106562.860924 write 0x029483 thru 0x02f310 (0x5e8e bytes) 000506 1764106562.866134 read 0x003a76 thru 0x00f49d (0xba28 bytes) 000507 1764106562.874742 MIRROR_RESYNC 4 mirrors 000508 1764106563.478922 MIRROR_RESYNC 4 mirrors 000509 1764106563.656397 read 0x02f3c8 thru 0x035bd1 (0x680a bytes) 000510 1764106563.665399 trunc from 0x035aff to 0x035e84 (0x386 bytes) 000511 1764106563.682782 write 0x039b08 thru 0x03f405 (0x58fe bytes) 000512 1764106563.687258 trunc from 0x03f406 to 0x017a1d (0x279e8 bytes) 000513 1764106563.706674 fallocate 0x0002a3 thru 0x00d506 (0xd264 bytes) 000514 1764106563.712571 mapread 0x005fe8 thru 0x0082fa (0x2313 bytes) 000518 1764106563.721483 trunc from 0x017a1e to 0x02f259 (0x1783c bytes) 000519 1764106563.731139 trunc from 0x02f25a to 0x01a805 (0x14a54 bytes) 000520 1764106563.742558 read 0x006da0 thru 0x00faac (0x8d0d bytes) 000521 1764106563.743191 write 0x03b8d3 thru 0x03ffff (0x472d bytes) 000523 1764106563.744807 trunc from 0x0307af to 0x03e356 (0xdba8 bytes) 000526 1764106563.754865 trunc from 0x040000 to 0x02e541 (0x11abe bytes) 000527 1764106563.766775 mapread 0x005dc1 thru 0x00646e (0x6ae bytes) 000528 1764106563.767509 trunc from 0x02e542 to 0x0371d2 (0x8c91 bytes) 000529 1764106563.778758 fallocate 0x03950c thru 0x03ffff (0x6af4 bytes) 000530 1764106563.788295 MIRROR_RESYNC 4 mirrors 000531 1764106564.283875 punch from 0x036876 to 0x0371d2 (0x95d bytes) 000534 1764106564.301958 fallocate 0x00397c thru 0x0081f9 (0x487e bytes) 000535 1764106564.306146 trunc from 0x024dde to 0x033663 (0xe886 bytes) 000536 1764106564.310892 mapwrite 0x001ee0 thru 0x00b383 (0x94a4 bytes) 000537 1764106564.339702 trunc from 0x0371d3 to 0x03cc60 (0x5a8e bytes) 000538 1764106564.353666 fallocate 0x013c32 thru 0x015207 (0x15d6 bytes) 000539 1764106564.357207 mapread 0x01b354 thru 0x01ded1 (0x2b7e bytes) 000540 1764106564.363834 punch from 0x030b59 to 0x032057 (0x14ff bytes) 000541 1764106564.367260 trunc from 0x03cc61 to 0x015bb1 (0x270af bytes) 000542 1764106564.375346 write 0x017022 thru 0x022440 (0xb41f bytes) 000543 1764106564.376838 write 0x017f0a thru 0x01d7cc (0x58c3 bytes) 000544 1764106564.377821 read 0x018015 thru 0x022440 (0xa42c bytes) 000545 1764106564.378314 write 0x012a29 thru 0x01f52d (0xcb05 bytes) 000546 1764106564.384502 punch from 0x0032fa to 0x0110bc (0xddc3 bytes) 000548 1764106564.388859 punch from 0x01277d to 0x01e48b (0xbd0f bytes) 000550 1764106564.396869 mapwrite 0x00bcac thru 0x00fe80 (0x41d5 bytes) 000551 1764106564.410176 mapwrite 0x03292b thru 0x037ca1 (0x5377 bytes) 000552 1764106564.436275 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37ca2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ca2): Mirror 2: 0xc25862df (stale) Mirror 3: 0xc25862df (stale) Mirror 4: 0xc25862df (stale) CRC-32 checksum value for chunk [0, 0x37ca2): Mirror 1: 0xcef7b972 Mirror 5: 0xcef7b972 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ca2: skipped 000553 1764106564.820833 read 0x002268 thru 0x008911 (0x66aa bytes) 000554 1764106564.825703 punch from 0x010d23 to 0x013057 (0x2335 bytes) 000555 1764106564.843949 punch from 0x036572 to 0x037ca1 (0x1730 bytes) 000556 1764106564.846782 trunc from 0x037ca2 to 0x0080a1 (0x2fc00 bytes) 000557 1764106564.867483 punch from 0x000bc7 to 0x0080a1 (0x74db bytes) 000558 1764106564.873219 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x80a2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x80a2): Mirror 2: 0x2fe61ef3 (stale) Mirror 3: 0x2fe61ef3 (stale) Mirror 4: 0x2fe61ef3 (stale) Mirror 5: 0x10a2ee87 (stale) CRC-32 checksum value for chunk [0, 0x80a2): Mirror 1: 0xce0924f5 Mirror 6: 0xce0924f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x80a2: skipped 000559 1764106565.246782 mapread 0x0049d3 thru 0x0080a1 (0x36cf bytes) 000560 1764106565.266939 trunc from 0x001e4c to 0x0080a1 (0x6256 bytes) 000561 1764106565.283386 read 0x003edd thru 0x0080a1 (0x41c5 bytes) 000562 1764106565.287634 read 0x005f5d thru 0x0080a1 (0x2145 bytes) 000564 1764106565.288693 read 0x005735 thru 0x0080a1 (0x296d bytes) 000565 1764106565.289242 write 0x03ae9a thru 0x03ffff (0x5166 bytes) 000566 1764106565.292619 trunc from 0x006fbe to 0x00f7b9 (0x87fc bytes) 000567 1764106565.297351 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4dc44816 (stale) Mirror 3: 0x4dc44816 (stale) Mirror 4: 0x4dc44816 (stale) Mirror 5: 0xd4ee5d48 (stale) Mirror 6: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91c77ce4 Mirror 7: 0x91c77ce4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000568 1764106565.668215 punch from 0x001f71 to 0x00ac35 (0x8cc5 bytes) 000569 1764106565.686198 mapread 0x003952 thru 0x00ec5c (0xb30b bytes) 000570 1764106565.695877 fallocate 0x0168f4 thru 0x020e55 (0xa562 bytes) 000571 1764106565.698602 fallocate 0x02089e thru 0x023859 (0x2fbc bytes) 000572 1764106565.701227 read 0x027ac1 thru 0x0285cc (0xb0c bytes) 000573 1764106565.704676 fallocate 0x03b3e4 thru 0x03ffff (0x4c1c bytes) 000575 1764106565.710440 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4dc44816 (stale) Mirror 3: 0x4dc44816 (stale) Mirror 5: 0xd4ee5d48 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x91c77ce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4dc44816 (stale) Mirror 3: 0x4dc44816 (stale) Mirror 5: 0xd4ee5d48 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x91c77ce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1764106565.902675 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4dc44816 (stale) Mirror 3: 0x4dc44816 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x91c77ce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4dc44816 (stale) Mirror 3: 0x4dc44816 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x91c77ce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000578 1764106566.069967 mapread 0x02ad2a thru 0x02b59d (0x874 bytes) 000579 1764106566.076415 mapwrite 0x0259c8 thru 0x032faf (0xd5e8 bytes) 000580 1764106566.101731 trunc from 0x040000 to 0x011058 (0x2efa7 bytes) 000581 1764106566.125001 trunc from 0x00de48 to 0x011058 (0x3211 bytes) 000583 1764106566.128935 fallocate 0x0321e6 thru 0x0355b6 (0x33d1 bytes) 000584 1764106566.136155 MIRROR_RESYNC 5 mirrors 000585 1764106566.561905 mapread 0x034e3d thru 0x0355b6 (0x77a bytes) 000586 1764106566.568810 write 0x0091a9 thru 0x0135ac (0xa404 bytes) 000587 1764106566.593248 mapread 0x032f14 thru 0x033a87 (0xb74 bytes) 000588 1764106566.598519 trunc from 0x02d6e5 to 0x0315b7 (0x3ed3 bytes) 000589 1764106566.601931 mapread 0x00930e thru 0x00bc5a (0x294d bytes) 000590 1764106566.603037 fallocate 0x00503f thru 0x00b69d (0x665f bytes) 000591 1764106566.606372 write 0x034a19 thru 0x037d77 (0x335f bytes) 000592 1764106566.607617 trunc from 0x00cbed to 0x0111ae (0x45c2 bytes) 000593 1764106566.619300 punch from 0x02e7a3 to 0x030cd0 (0x252e bytes) 000594 1764106566.622697 trunc from 0x037d78 to 0x027f52 (0xfe25 bytes) 000595 1764106566.634248 read 0x019a31 thru 0x024569 (0xab39 bytes) 000596 1764106566.640085 read 0x01965c thru 0x01a387 (0xd2c bytes) 000597 1764106566.640682 write 0x0066e2 thru 0x00fe19 (0x9738 bytes) 000598 1764106566.646784 punch from 0x004c81 to 0x00e2cd (0x964d bytes) 000600 1764106566.655480 mapwrite 0x006b1a thru 0x009a7e (0x2f65 bytes) 000601 1764106566.672491 punch from 0x013dd0 to 0x0158ba (0x1aeb bytes) 000602 1764106566.675260 write 0x00e9ff thru 0x015996 (0x6f98 bytes) 000603 1764106566.678376 read 0x00e480 thru 0x01647f (0x8000 bytes) 000604 1764106566.678959 punch from 0x01b21a to 0x021c68 (0x6a4f bytes) 000605 1764106566.681844 punch from 0x01b095 to 0x025daf (0xad1b bytes) 000606 1764106566.684772 trunc from 0x027f53 to 0x038af8 (0x10ba6 bytes) 000607 1764106566.695194 write 0x011e18 thru 0x01e97d (0xcb66 bytes) 000608 1764106566.701844 mapread 0x01fef5 thru 0x02a630 (0xa73c bytes) 000609 1764106566.708848 write 0x01d535 thru 0x025f00 (0x89cc bytes) 000611 1764106566.710434 trunc from 0x00ede5 to 0x017b7a (0x8d96 bytes) 000612 1764106566.719352 fallocate 0x02754b thru 0x029176 (0x1c2c bytes) 000613 1764106566.723071 punch from 0x037f2e to 0x038af8 (0xbcb bytes) 000615 1764106566.726329 mapread 0x00490d thru 0x00a18b (0x587f bytes) 000617 1764106566.727774 fallocate 0x00c77a thru 0x00e846 (0x20cd bytes) 000618 1764106566.731848 mapwrite 0x00a629 thru 0x013596 (0x8f6e bytes) 000619 1764106566.758923 mapwrite 0x03d109 thru 0x03ffff (0x2ef7 bytes) 000620 1764106566.781625 MIRROR_RESYNC 5 mirrors 000622 1764106567.343096 mapwrite 0x03e799 thru 0x03ffff (0x1867 bytes) 000623 1764106567.393703 punch from 0x02e010 to 0x039950 (0xb941 bytes) 000624 1764106567.396505 fallocate 0x03022d thru 0x032f9b (0x2d6f bytes) 000626 1764106567.400839 trunc from 0x02d10c to 0x036fa1 (0x9e96 bytes) 000627 1764106567.404505 punch from 0x00092f to 0x002579 (0x1c4b bytes) 000628 1764106567.407787 trunc from 0x040000 to 0x021c28 (0x1e3d7 bytes) 000629 1764106567.421462 trunc from 0x01de57 to 0x01ed1a (0xec4 bytes) 000631 1764106567.425542 punch from 0x01a85e to 0x01d7e6 (0x2f89 bytes) 000632 1764106567.432308 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21c29): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c29): Mirror 2: 0x5137815e (stale) Mirror 3: 0x5137815e (stale) Mirror 6: 0x5137815e (stale) Mirror 7: 0x5137815e (stale) CRC-32 checksum value for chunk [0, 0x21c29): Mirror 1: 0x15bf15d8 Mirror 8: 0x15bf15d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c29: skipped 000633 1764106567.836089 MIRROR_RESYNC 6 mirrors 000634 1764106568.388380 trunc from 0x021c29 to 0x0252a6 (0x367e bytes) 000635 1764106568.406374 trunc from 0x000061 to 0x009eaf (0x9e4f bytes) 000636 1764106568.409625 read 0x00ccbf thru 0x0146c2 (0x7a04 bytes) 000637 1764106568.414397 write 0x031c2b thru 0x03bb87 (0x9f5d bytes) 000638 1764106568.418025 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb88): Mirror 3: 0x20f67472 (stale) Mirror 6: 0x20f67472 (stale) Mirror 7: 0x20f67472 (stale) Mirror 8: 0x20f67472 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb88): Mirror 3: 0x20f67472 (stale) Mirror 6: 0x20f67472 (stale) Mirror 7: 0x20f67472 (stale) Mirror 8: 0x20f67472 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb88: skipped 000639 1764106568.588088 mapread 0x02b1e1 thru 0x031e1c (0x6c3c bytes) 000640 1764106568.598366 read 0x00b947 thru 0x0161d6 (0xa890 bytes) 000641 1764106568.604288 read 0x01ad06 thru 0x01e61a (0x3915 bytes) 000643 1764106568.610602 fallocate 0x0100f0 thru 0x0148c9 (0x47da bytes) 000644 1764106568.614080 read 0x025fd7 thru 0x032bde (0xcc08 bytes) 000645 1764106568.614536 read 0x0332bb thru 0x03bb87 (0x88cd bytes) 000646 1764106568.614903 mapwrite 0x007865 thru 0x00bb26 (0x42c2 bytes) 000647 1764106568.624007 trunc from 0x022174 to 0x0309dc (0xe869 bytes) 000648 1764106568.626442 write 0x005fff thru 0x006eb4 (0xeb6 bytes) 000649 1764106568.629528 mapwrite 0x007352 thru 0x008304 (0xfb3 bytes) 000650 1764106568.635774 write 0x03529a thru 0x03df75 (0x8cdc bytes) 000651 1764106568.637132 trunc from 0x03ac0e to 0x03df75 (0x3368 bytes) 000652 1764106568.642693 trunc from 0x03df76 to 0x00aafd (0x33478 bytes) 000653 1764106568.649248 read 0x008521 thru 0x00aafd (0x25dd bytes) 000654 1764106568.651896 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaafe): Mirror 6: 0x8597aa82 (stale) Mirror 7: 0x8597aa82 (stale) Mirror 8: 0x8597aa82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaafe): Mirror 6: 0x8597aa82 (stale) Mirror 7: 0x8597aa82 (stale) Mirror 8: 0x8597aa82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaafe: skipped 000655 1764106568.772849 fallocate 0x018bbe thru 0x01e716 (0x5b59 bytes) 000656 1764106568.775624 trunc from 0x00a302 to 0x00aafd (0x7fc bytes) 000657 1764106568.778452 punch from 0x002578 to 0x0030d5 (0xb5e bytes) 000658 1764106568.782800 trunc from 0x00aafe to 0x031757 (0x26c5a bytes) 000659 1764106568.792259 mapread 0x011904 thru 0x01fd67 (0xe464 bytes) 000660 1764106568.804218 trunc from 0x013828 to 0x01db44 (0xa31d bytes) 000661 1764106568.807162 trunc from 0x0039cc to 0x013925 (0xff5a bytes) 000663 1764106568.812886 MIRROR_RESYNC 4 mirrors 000664 1764106569.058459 mapread 0x02a67f thru 0x02f1d9 (0x4b5b bytes) 000665 1764106569.064570 mapread 0x0271c3 thru 0x027cd7 (0xb15 bytes) 000666 1764106569.066508 MIRROR_RESYNC 4 mirrors 000667 1764106569.224059 write 0x038aa5 thru 0x03ffff (0x755b bytes) 000668 1764106569.238242 mapread 0x01e4d2 thru 0x028a9c (0xa5cb bytes) 000669 1764106569.245783 read 0x0038c9 thru 0x012312 (0xea4a bytes) 000670 1764106569.250523 write 0x000115 thru 0x004030 (0x3f1c bytes) 000672 1764106569.253834 write 0x037488 thru 0x03b91c (0x4495 bytes) 000673 1764106569.259553 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1764106569.390532 trunc from 0x040000 to 0x028027 (0x17fd8 bytes) 000675 1764106569.409233 mapwrite 0x023855 thru 0x024f0b (0x16b7 bytes) 000676 1764106569.424277 read 0x01959a thru 0x01d4ee (0x3f55 bytes) 000677 1764106569.428617 fallocate 0x03c8bf thru 0x03e857 (0x1f99 bytes) 000678 1764106569.431995 mapwrite 0x013a4b thru 0x01e91f (0xaed5 bytes) 000679 1764106569.450029 write 0x032a6a thru 0x035175 (0x270c bytes) 000680 1764106569.458893 write 0x008b0d thru 0x017dee (0xf2e2 bytes) 000681 1764106569.464193 trunc from 0x036849 to 0x03a20d (0x39c5 bytes) 000682 1764106569.469949 mapread 0x0099dd thru 0x00b129 (0x174d bytes) 000683 1764106569.470809 trunc from 0x029097 to 0x02f2bc (0x6226 bytes) 000685 1764106569.476238 fallocate 0x017792 thru 0x0179fc (0x26b bytes) 000686 1764106569.479366 fallocate 0x0088b3 thru 0x0173b1 (0xeaff bytes) 000687 1764106569.482761 read 0x01d9ee thru 0x0222f7 (0x490a bytes) 000688 1764106569.483413 fallocate 0x0000c1 thru 0x00e0ef (0xe02f bytes) 000689 1764106569.486930 punch from 0x00a0e8 to 0x019e5a (0xfd73 bytes) 000690 1764106569.493992 mapwrite 0x032eba thru 0x037e29 (0x4f70 bytes) 000691 1764106569.509762 mapwrite 0x03d435 thru 0x03fca5 (0x2871 bytes) 000692 1764106569.527807 trunc from 0x03fca6 to 0x03be96 (0x3e0f bytes) 000693 1764106569.536347 mapwrite 0x029d73 thru 0x03953f (0xf7cd bytes) 000694 1764106569.547628 punch from 0x017cae to 0x018e6b (0x11be bytes) 000695 1764106569.550182 write 0x0344ec thru 0x03fff7 (0xbb0c bytes) 000696 1764106569.551249 punch from 0x010e57 to 0x013db1 (0x2f5b bytes) 000697 1764106569.554160 trunc from 0x039afc to 0x03fff7 (0x64fc bytes) 000698 1764106569.559737 read 0x01525e thru 0x01b03e (0x5de1 bytes) 000699 1764106569.563306 trunc from 0x03fff8 to 0x012bf2 (0x2d405 bytes) 000700 1764106569.569469 read 0x006013 thru 0x012bf2 (0xcbe0 bytes) 000701 1764106569.570058 punch from 0x0068cd to 0x012bf2 (0xc326 bytes) 000704 1764106569.572703 write 0x00a6d9 thru 0x0165b0 (0xbed8 bytes) 000705 1764106569.576640 punch from 0x0002d9 to 0x00fa8d (0xf7b5 bytes) 000706 1764106569.582187 trunc from 0x0165b1 to 0x003545 (0x1306b bytes) 000707 1764106569.588250 trunc from 0x003546 to 0x034fdc (0x31a97 bytes) 000708 1764106569.593181 trunc from 0x034fdd to 0x00e56f (0x26a6d bytes) 000709 1764106569.597766 punch from 0x00d79c to 0x00e56f (0xdd4 bytes) 000710 1764106569.599407 fallocate 0x00f2df thru 0x01b7e2 (0xc504 bytes) 000711 1764106569.601636 read 0x004c52 thru 0x00728a (0x2639 bytes) 000712 1764106569.605828 mapread 0x000677 thru 0x004716 (0x40a0 bytes) 000713 1764106569.606510 trunc from 0x0042c3 to 0x005dbe (0x1afc bytes) 000714 1764106569.609478 fallocate 0x026e4e thru 0x031f5c (0xb10f bytes) 000715 1764106569.612569 fallocate 0x02bc9a thru 0x03803a (0xc3a1 bytes) 000716 1764106569.614838 trunc from 0x00e570 to 0x0074a7 (0x70c8 bytes) 000717 1764106569.620769 read 0x006e85 thru 0x0074a7 (0x623 bytes) 000718 1764106569.621185 mapread 0x0013c8 thru 0x0074a7 (0x60e0 bytes) 000719 1764106569.624115 trunc from 0x0074a8 to 0x034ef9 (0x2da52 bytes) 000720 1764106569.630329 trunc from 0x034efa to 0x0181c8 (0x1cd31 bytes) 000721 1764106569.635603 mapwrite 0x026217 thru 0x02ed7f (0x8b69 bytes) 000722 1764106569.655421 write 0x030fab thru 0x03f0d3 (0xe129 bytes) 000724 1764106569.657742 trunc from 0x007957 to 0x014320 (0xc9ca bytes) 000726 1764106569.660810 write 0x017f23 thru 0x027e6b (0xff49 bytes) 000727 1764106569.662377 trunc from 0x03f0d4 to 0x0229a5 (0x1c72e bytes) 000729 1764106569.669801 read 0x021cf6 thru 0x0224ad (0x7b8 bytes) 000730 1764106569.670321 write 0x0177a1 thru 0x0219fd (0xa25d bytes) 000731 1764106569.671666 write 0x02c48e thru 0x034a5b (0x85ce bytes) 000732 1764106569.672732 trunc from 0x034a5c to 0x0161a1 (0x1e8ba bytes) 000733 1764106569.679017 mapread 0x00fe33 thru 0x0161a1 (0x636f bytes) 000734 1764106569.683245 punch from 0x000e92 to 0x00183e (0x9ad bytes) 000735 1764106569.686172 punch from 0x00cfe6 to 0x0161a1 (0x91bc bytes) 000736 1764106569.689177 trunc from 0x008301 to 0x014af5 (0xc7f5 bytes) 000737 1764106569.692273 fallocate 0x032ea7 thru 0x033b1c (0xc76 bytes) 000738 1764106569.695482 write 0x0303b5 thru 0x034c35 (0x4881 bytes) 000740 1764106569.700339 trunc from 0x031729 to 0x034849 (0x3121 bytes) 000741 1764106569.706420 mapread 0x01b002 thru 0x029c36 (0xec35 bytes) 000744 1764106569.708101 read 0x0275b4 thru 0x0306f8 (0x9145 bytes) 000745 1764106569.708699 fallocate 0x03a4a2 thru 0x03b7b9 (0x1318 bytes) 000746 1764106569.711852 read 0x01e244 thru 0x02bf11 (0xdcce bytes) 000747 1764106569.714840 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34c36): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c36): Mirror 7: 0x6cb1c7df (stale) Mirror 8: 0x6cb1c7df (stale) CRC-32 checksum value for chunk [0, 0x34c36): Mirror 1: 0xbc0aa3e1 Mirror 9: 0xbc0aa3e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c36: skipped 000748 1764106570.014950 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34c36): Mirror 7: 0x6cb1c7df (stale) Mirror 8: 0x6cb1c7df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c36): Mirror 7: 0x6cb1c7df (stale) Mirror 8: 0x6cb1c7df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c36: skipped 000749 1764106570.121442 mapread 0x024de5 thru 0x03277b (0xd997 bytes) 000751 1764106570.131906 mapwrite 0x0350a9 thru 0x03ffff (0xaf57 bytes) 000752 1764106570.163950 mapread 0x0395bf thru 0x03ffff (0x6a41 bytes) 000753 1764106570.164729 punch from 0x000126 to 0x008778 (0x8653 bytes) 000755 1764106570.167735 punch from 0x0261b9 to 0x034ea0 (0xece8 bytes) 000756 1764106570.170378 punch from 0x00a664 to 0x0109f9 (0x6396 bytes) 000757 1764106570.172818 mapwrite 0x03ac2e thru 0x03ffff (0x53d2 bytes) 000758 1764106570.182143 trunc from 0x018a4f to 0x0224d7 (0x9a89 bytes) 000759 1764106570.185174 mapwrite 0x0139cc thru 0x01511b (0x1750 bytes) 000760 1764106570.196111 write 0x00a47e thru 0x0107c3 (0x6346 bytes) 000761 1764106570.200308 punch from 0x01fb80 to 0x02aa4c (0xaecd bytes) 000762 1764106570.203041 mapwrite 0x0164bd thru 0x0258aa (0xf3ee bytes) 000763 1764106570.223239 punch from 0x02769a to 0x02dd97 (0x66fe bytes) 000764 1764106570.227531 trunc from 0x017614 to 0x023b71 (0xc55e bytes) 000765 1764106570.234362 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000766 1764106570.352662 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) 000767 1764106570.399404 mapwrite 0x01626d thru 0x024f62 (0xecf6 bytes) 000770 1764106570.423070 mapread 0x0226c1 thru 0x02ce10 (0xa750 bytes) 000771 1764106570.427720 write 0x03a498 thru 0x03ffff (0x5b68 bytes) 000772 1764106570.435410 mapread 0x01a64c thru 0x023703 (0x90b8 bytes) 000774 1764106570.436845 trunc from 0x040000 to 0x013429 (0x2cbd6 bytes) 000775 1764106570.443737 punch from 0x0124bb to 0x013429 (0xf6f bytes) 000776 1764106570.449058 MIRROR_EXTEND to 2 mirrors 000777 1764106570.679550 write 0x024128 thru 0x025694 (0x156d bytes) 000778 1764106570.704715 fallocate 0x018fa8 thru 0x026878 (0xd8d1 bytes) 000780 1764106570.708553 write 0x01bd18 thru 0x028c7f (0xcf68 bytes) 000781 1764106570.713390 punch from 0x011520 to 0x019f76 (0x8a57 bytes) 000782 1764106570.718101 read 0x014c9a thru 0x01e265 (0x95cc bytes) 000783 1764106570.723319 trunc from 0x028c80 to 0x002efa (0x25d85 bytes) 000784 1764106570.735958 mapread 0x001342 thru 0x002efa (0x1bb9 bytes) 000785 1764106570.743246 fallocate 0x03d9d0 thru 0x03ffff (0x2630 bytes) 000786 1764106570.747275 trunc from 0x029c88 to 0x02ee7a (0x51f3 bytes) 000788 1764106570.751378 mapread 0x035d65 thru 0x03ffff (0xa29b bytes) 000789 1764106570.760861 write 0x00c1aa thru 0x01513e (0x8f95 bytes) 000790 1764106570.770680 write 0x02111e thru 0x0225c7 (0x14aa bytes) 000791 1764106570.776396 fallocate 0x03f2cd thru 0x03ffff (0xd33 bytes) 000793 1764106570.779754 trunc from 0x040000 to 0x01bea2 (0x2415d bytes) 000794 1764106570.790434 punch from 0x00805c to 0x015515 (0xd4ba bytes) 000795 1764106570.796094 write 0x02143c thru 0x0229f9 (0x15be bytes) 000796 1764106570.797169 fallocate 0x02a79a thru 0x02b451 (0xcb8 bytes) 000797 1764106570.801396 mapread 0x00cfba thru 0x019eac (0xcef3 bytes) 000798 1764106570.810121 fallocate 0x013ed2 thru 0x02360f (0xf73e bytes) 000799 1764106570.815776 fallocate 0x023208 thru 0x02bde3 (0x8bdc bytes) 000802 1764106570.820450 fallocate 0x02d04e thru 0x036bee (0x9ba1 bytes) 000803 1764106570.824539 mapread 0x018083 thru 0x0203e8 (0x8366 bytes) 000804 1764106570.832184 trunc from 0x036bef to 0x016faf (0x1fc3f bytes) 000805 1764106570.842393 mapwrite 0x028a8d thru 0x0374b9 (0xea2d bytes) 000806 1764106570.880628 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) 000807 1764106570.942047 write 0x00633b thru 0x0128af (0xc575 bytes) 000808 1764106570.950013 trunc from 0x0066df to 0x00dd4c (0x766e bytes) 000810 1764106570.960199 mapread 0x034f05 thru 0x0374b9 (0x25b5 bytes) 000811 1764106570.967770 read 0x0276ef thru 0x032f83 (0xb895 bytes) 000812 1764106570.974015 punch from 0x02b66f to 0x0374b9 (0xbe4b bytes) 000813 1764106570.977445 trunc from 0x0374ba to 0x032c92 (0x4827 bytes) 000814 1764106570.990421 read 0x01a344 thru 0x028cfd (0xe9ba bytes) 000815 1764106570.999458 mapread 0x032581 thru 0x032c92 (0x712 bytes) 000816 1764106571.000458 read 0x0027b4 thru 0x0092b5 (0x6b02 bytes) 000817 1764106571.001083 trunc from 0x01a62a to 0x02968c (0xf063 bytes) 000819 1764106571.005387 trunc from 0x012e03 to 0x0152ee (0x24ec bytes) 000820 1764106571.008397 mapwrite 0x025910 thru 0x026091 (0x782 bytes) 000821 1764106571.020034 trunc from 0x001526 to 0x00a207 (0x8ce2 bytes) 000822 1764106571.023211 mapwrite 0x00e71d thru 0x019762 (0xb046 bytes) 000824 1764106571.032772 mapread 0x02f542 thru 0x032c92 (0x3751 bytes) 000825 1764106571.033383 mapwrite 0x039949 thru 0x03bb71 (0x2229 bytes) 000827 1764106571.052604 trunc from 0x03a70e to 0x03bb71 (0x1464 bytes) 000830 1764106571.056739 mapwrite 0x02ec77 thru 0x02f108 (0x492 bytes) 000831 1764106571.064304 punch from 0x0328e5 to 0x03bb71 (0x928d bytes) 000832 1764106571.067459 punch from 0x01273f to 0x01d109 (0xa9cb bytes) 000833 1764106571.072841 fallocate 0x0183d1 thru 0x025a09 (0xd639 bytes) 000834 1764106571.078246 MIRROR_EXTEND to 2 mirrors 000835 1764106571.398812 trunc from 0x029b16 to 0x038334 (0xe81f bytes) 000836 1764106571.418226 mapwrite 0x02611b thru 0x031e51 (0xbd37 bytes) 000838 1764106571.448555 fallocate 0x01a72f thru 0x022cbd (0x858f bytes) 000839 1764106571.451661 mapwrite 0x02def7 thru 0x038c05 (0xad0f bytes) 000840 1764106571.463227 mapwrite 0x02a7ec thru 0x02eed3 (0x46e8 bytes) 000841 1764106571.472106 fallocate 0x00b537 thru 0x00e4a6 (0x2f70 bytes) 000842 1764106571.475118 trunc from 0x019880 to 0x01f3bf (0x5b40 bytes) 000843 1764106571.478013 read 0x00e1d7 thru 0x012a1e (0x4848 bytes) 000844 1764106571.485752 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 2: 0x79778b4 (stale) CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 1: 0x3b9395e8 Mirror 3: 0x3b9395e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb72: skipped 000845 1764106571.793205 mapwrite 0x0241d8 thru 0x025bc4 (0x19ed bytes) 000846 1764106571.840874 trunc from 0x03bb72 to 0x016a27 (0x2514a bytes) 000847 1764106571.860276 write 0x02054b thru 0x028695 (0x814b bytes) 000848 1764106571.861388 trunc from 0x011261 to 0x01b330 (0xa0d0 bytes) 000849 1764106571.864011 trunc from 0x028696 to 0x00534a (0x2334b bytes) 000850 1764106571.873020 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x534b): Mirror 2: 0xc536ca2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x534b): Mirror 2: 0xc536ca2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x534b: skipped 000851 1764106571.983303 fallocate 0x0271d3 thru 0x02aebe (0x3cec bytes) 000852 1764106571.985469 trunc from 0x02aebf to 0x015f1d (0x14fa1 bytes) 000853 1764106571.994041 read 0x00c808 thru 0x013c15 (0x740e bytes) 000854 1764106571.997850 mapread 0x0042b7 thru 0x00a62e (0x6378 bytes) 000855 1764106572.006496 mapwrite 0x030165 thru 0x036b80 (0x6a1c bytes) 000856 1764106572.029383 fallocate 0x014e90 thru 0x01935d (0x44ce bytes) 000857 1764106572.033010 mapwrite 0x007ef9 thru 0x01793b (0xfa43 bytes) 000858 1764106572.052814 MIRROR_RESYNC 2 mirrors 000859 1764106572.341040 trunc from 0x036a07 to 0x036b80 (0x17a bytes) 000860 1764106572.358299 write 0x03dee1 thru 0x03ffff (0x211f bytes) 000861 1764106572.359521 write 0x01f3a3 thru 0x020945 (0x15a3 bytes) 000862 1764106572.365841 mapwrite 0x03c3c2 thru 0x03ffff (0x3c3e bytes) 000863 1764106572.379991 mapwrite 0x01649e thru 0x01b27f (0x4de2 bytes) 000864 1764106572.395665 read 0x01fb85 thru 0x0245d2 (0x4a4e bytes) 000865 1764106572.400113 mapread 0x015b57 thru 0x0229e3 (0xce8d bytes) 000866 1764106572.401079 mapwrite 0x03504a thru 0x03ffff (0xafb6 bytes) 000867 1764106572.411678 read 0x02e0d6 thru 0x0300b6 (0x1fe1 bytes) 000869 1764106572.417067 mapread 0x008ddb thru 0x01171a (0x8940 bytes) 000870 1764106572.418006 mapwrite 0x00e035 thru 0x012569 (0x4535 bytes) 000871 1764106572.427954 trunc from 0x009319 to 0x0139db (0xa6c3 bytes) 000872 1764106572.431227 write 0x00693f thru 0x00af5d (0x461f bytes) 000873 1764106572.434906 trunc from 0x015b6a to 0x01ed5b (0x91f2 bytes) 000874 1764106572.437678 mapwrite 0x035db1 thru 0x037f24 (0x2174 bytes) 000876 1764106572.446794 trunc from 0x01ea62 to 0x025567 (0x6b06 bytes) 000877 1764106572.450222 read 0x01e983 thru 0x02e0f8 (0xf776 bytes) 000878 1764106572.455248 mapread 0x022283 thru 0x02608a (0x3e08 bytes) 000880 1764106572.456472 fallocate 0x00204e thru 0x00c968 (0xa91b bytes) 000881 1764106572.460254 mapread 0x02dc5a thru 0x03bd00 (0xe0a7 bytes) 000882 1764106572.461020 read 0x0197cb thru 0x020d74 (0x75aa bytes) 000883 1764106572.461514 write 0x01e611 thru 0x020096 (0x1a86 bytes) 000884 1764106572.462209 punch from 0x009ca3 to 0x018bc2 (0xef20 bytes) 000885 1764106572.466503 punch from 0x013e96 to 0x020cea (0xce55 bytes) 000886 1764106572.471879 trunc from 0x040000 to 0x011d45 (0x2e2ba bytes) 000888 1764106572.489647 trunc from 0x00be20 to 0x011d45 (0x5f26 bytes) 000889 1764106572.492886 fallocate 0x0306e8 thru 0x03fd4e (0xf667 bytes) 000890 1764106572.495964 fallocate 0x00ccef thru 0x014408 (0x771a bytes) 000891 1764106572.499393 punch from 0x0158b2 to 0x019b0e (0x425d bytes) 000892 1764106572.502255 trunc from 0x002343 to 0x0081de (0x5e9c bytes) 000893 1764106572.505154 trunc from 0x024bf5 to 0x0292f2 (0x46fe bytes) 000894 1764106572.507760 read 0x0316d5 thru 0x03ae06 (0x9732 bytes) 000895 1764106572.513631 read 0x0224a3 thru 0x027e67 (0x59c5 bytes) 000896 1764106572.514180 mapwrite 0x03f07f thru 0x03ffff (0xf81 bytes) 000897 1764106572.530408 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x660ca46a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbf451402 Mirror 3: 0xbf451402 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1764106572.886927 mapread 0x016f41 thru 0x018852 (0x1912 bytes) 000899 1764106572.910303 write 0x000918 thru 0x007aec (0x71d5 bytes) 000900 1764106572.937123 read 0x027e60 thru 0x037d0e (0xfeaf bytes) 000902 1764106572.942287 mapwrite 0x01cfb6 thru 0x02aaeb (0xdb36 bytes) 000903 1764106572.970939 MIRROR_RESYNC 3 mirrors 000904 1764106573.355286 MIRROR_RESYNC 3 mirrors 000905 1764106573.554315 mapwrite 0x0007b9 thru 0x003c43 (0x348b bytes) 000906 1764106573.604764 read 0x01e7f3 thru 0x023fa3 (0x57b1 bytes) 000907 1764106573.608620 punch from 0x006520 to 0x007b4d (0x162e bytes) 000908 1764106573.611697 mapread 0x03ebad thru 0x03ffff (0x1453 bytes) 000909 1764106573.616290 read 0x039341 thru 0x03e20f (0x4ecf bytes) 000911 1764106573.617273 write 0x0352f2 thru 0x03ffff (0xad0e bytes) 000912 1764106573.622507 fallocate 0x0076c3 thru 0x010c80 (0x95be bytes) 000913 1764106573.625515 punch from 0x00f0a9 to 0x012cc3 (0x3c1b bytes) 000914 1764106573.628226 mapwrite 0x01d7d0 thru 0x01f360 (0x1b91 bytes) 000915 1764106573.637215 punch from 0x003047 to 0x009a50 (0x6a0a bytes) 000916 1764106573.640650 punch from 0x002c04 to 0x008ff7 (0x63f4 bytes) 000917 1764106573.644181 mapread 0x0028bb thru 0x00d781 (0xaec7 bytes) 000918 1764106573.663851 punch from 0x00b6cd to 0x01aa8f (0xf3c3 bytes) 000919 1764106573.669951 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2759b8b0 (stale) Mirror 3: 0x2759b8b0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf38017af Mirror 4: 0xf38017af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000920 1764106574.120851 trunc from 0x040000 to 0x00e165 (0x31e9a bytes) 000921 1764106574.160423 trunc from 0x00e166 to 0x00fa40 (0x18db bytes) 000922 1764106574.170814 mapread 0x0088ad thru 0x00fa40 (0x7194 bytes) 000923 1764106574.179699 punch from 0x00a663 to 0x00fa40 (0x53de bytes) 000924 1764106574.183063 write 0x01e543 thru 0x01f236 (0xcf4 bytes) 000925 1764106574.183947 trunc from 0x01bd4c to 0x01bf97 (0x24c bytes) 000926 1764106574.187054 trunc from 0x01f237 to 0x01b294 (0x3fa2 bytes) 000927 1764106574.197417 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b295): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b295): Mirror 2: 0x848e2703 (stale) Mirror 3: 0x848e2703 (stale) Mirror 4: 0x865f448e (stale) CRC-32 checksum value for chunk [0, 0x1b295): Mirror 1: 0x865f448e Mirror 5: 0x865f448e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b295: skipped 000928 1764106574.704526 mapwrite 0x02f7fb thru 0x030c18 (0x141e bytes) 000930 1764106574.769180 mapwrite 0x02c2a8 thru 0x02c68f (0x3e8 bytes) 000931 1764106574.783015 fallocate 0x0384a4 thru 0x03eb38 (0x6695 bytes) 000932 1764106574.789507 MIRROR_RESYNC 5 mirrors 000933 1764106575.548158 trunc from 0x03eb39 to 0x03df9f (0xb99 bytes) 000934 1764106575.572812 read 0x023980 thru 0x025064 (0x16e5 bytes) 000935 1764106575.576287 fallocate 0x026835 thru 0x0272c1 (0xa8d bytes) 000937 1764106575.580044 mapwrite 0x0050b9 thru 0x0138f9 (0xe841 bytes) 000938 1764106575.610855 write 0x003330 thru 0x009c4b (0x691c bytes) 000939 1764106575.618957 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x452:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfa0): Mirror 3: 0x50f1cec7 (stale) Mirror 4: 0x50f1cec7 (stale) Mirror 5: 0x50f1cec7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfa0): Mirror 3: 0x50f1cec7 (stale) Mirror 4: 0x50f1cec7 (stale) Mirror 5: 0x50f1cec7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfa0: skipped 000941 1764106575.793064 trunc from 0x03dfa0 to 0x001c49 (0x3c356 bytes) 000942 1764106575.807158 trunc from 0x0019aa to 0x001c49 (0x2a0 bytes) 000943 1764106575.810241 trunc from 0x000bd6 to 0x001c49 (0x1074 bytes) 000944 1764106575.818661 trunc from 0x000358 to 0x001c49 (0x18f2 bytes) 000945 1764106575.822030 trunc from 0x001c4a to 0x000d93 (0xeb6 bytes) 000946 1764106575.837168 trunc from 0x000d94 to 0x01dfba (0x1d227 bytes) 000947 1764106575.852526 punch from 0x00ddef to 0x01320d (0x541f bytes) 000948 1764106575.855780 trunc from 0x01dfbb to 0x03af3b (0x1cf81 bytes) 000949 1764106575.864991 mapwrite 0x03ca2b thru 0x03ffff (0x35d5 bytes) 000950 1764106575.888516 mapwrite 0x03de3a thru 0x03ffff (0x21c6 bytes) 000951 1764106575.897120 mapread 0x01c85b thru 0x01e891 (0x2037 bytes) 000952 1764106575.902379 mapread 0x019f47 thru 0x024184 (0xa23e bytes) 000953 1764106575.903304 read 0x02c8e3 thru 0x033e00 (0x751e bytes) 000954 1764106575.907915 mapread 0x01d944 thru 0x02c18d (0xe84a bytes) 000955 1764106575.913360 trunc from 0x040000 to 0x0159a0 (0x2a65f bytes) 000956 1764106575.930140 trunc from 0x0159a1 to 0x038629 (0x22c89 bytes) 000957 1764106575.940298 fallocate 0x01bd74 thru 0x022382 (0x660f bytes) 000958 1764106575.944777 mapread 0x027eb0 thru 0x029185 (0x12d6 bytes) 000959 1764106575.950512 write 0x018cac thru 0x01dd28 (0x507d bytes) 000960 1764106575.956666 fallocate 0x001d27 thru 0x00bd8f (0xa069 bytes) 000961 1764106575.960442 trunc from 0x0263c7 to 0x02c7fb (0x6435 bytes) 000962 1764106575.963946 write 0x005da7 thru 0x007c3f (0x1e99 bytes) 000963 1764106575.971329 trunc from 0x03862a to 0x009b62 (0x2eac7 bytes) 000965 1764106575.985963 mapread 0x009759 thru 0x009b62 (0x40a bytes) 000966 1764106575.986848 punch from 0x001b4d to 0x009b62 (0x8016 bytes) 000967 1764106576.000525 mapread 0x002e88 thru 0x009b62 (0x6cdb bytes) 000968 1764106576.007368 write 0x008b7b thru 0x01178e (0x8c14 bytes) 000969 1764106576.009470 read 0x006ac0 thru 0x01178e (0xaccf bytes) 000970 1764106576.010117 fallocate 0x0194bc thru 0x027c21 (0xe766 bytes) 000971 1764106576.014042 fallocate 0x028694 thru 0x034b71 (0xc4de bytes) 000972 1764106576.019187 fallocate 0x013b6b thru 0x0231ab (0xf641 bytes) 000974 1764106576.023991 read 0x022a4a thru 0x0231ab (0x762 bytes) 000975 1764106576.030190 mapwrite 0x00c6c9 thru 0x00d5f4 (0xf2c bytes) 000976 1764106576.050147 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x231ac): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x231ac): Mirror 3: 0x43013b37 (stale) Mirror 4: 0x43013b37 (stale) Mirror 5: 0x43013b37 (stale) CRC-32 checksum value for chunk [0, 0x231ac): Mirror 1: 0xe76f2d38 Mirror 6: 0xe76f2d38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x231ac: skipped 000977 1764106576.433224 trunc from 0x0231ac to 0x001c7b (0x21530 bytes) 000978 1764106576.472323 punch from 0x001a65 to 0x001c7b (0x217 bytes) 000979 1764106576.475362 read 0x001aef thru 0x001c7b (0x18d bytes) 000980 1764106576.479036 trunc from 0x00074a to 0x001c7b (0x1532 bytes) 000981 1764106576.483101 mapread 0x000a1a thru 0x001c7b (0x1262 bytes) 000982 1764106576.493524 trunc from 0x001c7c to 0x01c9ba (0x1ad3f bytes) 000983 1764106576.508474 fallocate 0x02605d thru 0x028c1f (0x2bc3 bytes) 000984 1764106576.512880 punch from 0x01226c to 0x01b555 (0x92ea bytes) 000986 1764106576.516663 write 0x0104cb thru 0x01239b (0x1ed1 bytes) 000987 1764106576.525623 read 0x01ec3e thru 0x028a81 (0x9e44 bytes) 000988 1764106576.531309 fallocate 0x0053d6 thru 0x00fae2 (0xa70d bytes) 000989 1764106576.534837 write 0x038864 thru 0x03ffff (0x779c bytes) 000990 1764106576.536102 read 0x0272ef thru 0x02fc3a (0x894c bytes) 000991 1764106576.543828 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4c930a9 (stale) Mirror 4: 0xe4c930a9 (stale) Mirror 5: 0xe4c930a9 (stale) Mirror 6: 0x24e0df67 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x31b3f986 Mirror 7: 0x31b3f986 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1764106576.980887 mapwrite 0x0167ac thru 0x01bbfe (0x5453 bytes) 000993 1764106577.068475 MIRROR_RESYNC 6 mirrors 000995 1764106577.772367 trunc from 0x01af40 to 0x02472d (0x97ee bytes) 000996 1764106577.789789 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe006fc63 (stale) Mirror 4: 0xe006fc63 (stale) Mirror 5: 0xe006fc63 (stale) Mirror 6: 0xe006fc63 (stale) Mirror 7: 0xe006fc63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8159b03c Mirror 8: 0x8159b03c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1764106578.197466 punch from 0x037f16 to 0x03fd68 (0x7e53 bytes) 000998 1764106578.218366 trunc from 0x040000 to 0x0052e1 (0x3ad1e bytes) 000999 1764106578.228706 mapwrite 0x03f31f thru 0x03ffff (0xce1 bytes) 001000 1764106578.248055 MIRROR_RESYNC 7 mirrors All operations completed A-OK! PASS 70a (39s) == sanity-flr test 71: check mirror extend parent fid ==== 16:36:20 (1764106580) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x48d:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 16:36:24 (1764106584) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3794752 .. Extending file size to 5797856 .. failed Extending file size to 7843008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8506624 .. Extending file size to 8234272 .. Extending file size to 8439488 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8590240 .. Extending file size to 8488832 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8883456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8862912 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8907616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8962656 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60702 60703 60706 60707 60708 failed 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:37:31 (1764106651) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1698720 .. Extending file size to 5867520 .. failed Extending file size to 8748480 .. Extending file size to 3246656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8841856 .. Extending file size to 4032736 .. Extending file size to 6394656 .. Extending file size to 9003008 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9090464 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 64097 64098 64099 64102 64104 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) 192.168.201.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:38:36 (1764106716) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2568 1268736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21284 3585736 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15648 3591372 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36932 7177108 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2568 1268736 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21284 3585736 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15648 3591372 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36932 7177108 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0489742 s, 42.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0679751 s, 30.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf9f270f0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:38:43 (1764106723) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0488281 s, 42.9 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 16:38:46 (1764106726) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:38:49 (1764106729) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423759 s, 49.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:38:52 (1764106732) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0356909 s, 58.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:38:56 (1764106736) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0457287 s, 45.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:39:21 (1764106761) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381952 s, 27.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5ee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5ee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5ef:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:39:56 (1764106796) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0561952 s, 37.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:40:32 (1764106832) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg146-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg146-server: Pool lustre.test_204f created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0374899 s, 55.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg146-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:41:21 (1764106881) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:41:24 (1764106884) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:41:27 (1764106887) oleg146-server: Pool lustre.test_206 created Adding targets to pool oleg146-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg146-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg146-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg146-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:41:42 (1764106902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284877 s, 36.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137918 s, 76.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x93c725e3 Mirror 3: 0x93c725e3 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:41:47 (1764106907) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:41:48 (1764106908) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:41:49 (1764106909) Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0169589 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19f0000 Read this (4 bytes): BBBB 192.168.201.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg146-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:42:12 (1764106932) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0224605 s, 365 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:42:21 (1764106941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10174 s, 10.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-25 16:42:24.000000000 -0500 Modify: 2025-11-25 16:42:24.000000000 -0500 Change: 2025-11-25 16:42:27.000000000 -0500 Birth: 2025-11-25 16:42:23.000000000 -0500 PASS 210a (18s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:42:40 (1764106960) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111075 s, 9.4 MB/s oleg146-server: debugfs 1.47.3-wc1 (8-Aug-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg146-server: debugfs 1.47.3-wc1 (8-Aug-2025) oleg146-server: O/280000401/d29/bd: File not found by ext2_lookup PASS 210b (31s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:43:11 (1764106991) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.291369 s, 141 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.029623 s, 138 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 25 16:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 25 16:43 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:43:22 (1764107002) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:43:31 (1764107011) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1012 sec =========== 16:43:38 (1764107018) === sanity-flr: start cleanup 16:43:40 (1764107020) === === sanity-flr: finish cleanup 16:43:47 (1764107027) ===