-----============= acceptance-small: sanity-flr ============----- Mon Dec 22 09:21:28 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg205-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:21:49 (1766413309) === oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff941c85613800.idle_timeout=debug osc.lustre-OST0001-osc-ffff941c85613800.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:22:20 (1766413340) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7566] == sanity-flr test 0a: lfs mirror create with -N option == 09:22:22 (1766413342) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:22:36 (1766413356) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:22:39 (1766413359) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:22:42 (1766413362) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:23:05 (1766413385) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:23:08 (1766413388) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:23:12 (1766413392) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff941c85613800.stats=clear osc.lustre-OST0001-osc-ffff941c85613800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:23:20 (1766413400) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:23:30 (1766413410) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:23:38 (1766413418) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:23:41 (1766413421) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:23:50 (1766413430) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:23:58 (1766413438) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:24:01 (1766413441) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:24:09 (1766413449) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:24:20 (1766413460) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:24:28 (1766413468) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.60186 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180507 s, 5.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:24:40 (1766413480) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.152151 s, 27.6 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:24:47 (1766413487) Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (58s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:25:45 (1766413545) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.400903 s, 18.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:26:19 (1766413579) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (64s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:27:23 (1766413643) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.58677 s, 14.6 MB/s -rw-r--r-- 1 root root 52428800 Dec 22 09:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.796s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.000s user 0m0.000s sys 0m0.166s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.536s user 0m0.002s sys 0m0.231s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff941c85613800.ost_server_uuid in FULL state after 0 sec PASS 33b (71s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:28:34 (1766413714) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:28:37 (1766413717) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:28:40 (1766413720) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:28:42 (1766413722) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 09:28:49 (1766413729) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.398255 s, 31.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:29:00 (1766413740) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650162 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486991 s, 21.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:29:08 (1766413748) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:29:15 (1766413755) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00181327 s, 565 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012464 s, 822 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00195728 s, 523 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105681 s, 969 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 09:29:27 (1766413767) create test files with size 15733408 .. 9c19de15db158ee6c23cbf4dc597acb6 - 1b52b05c89e7b07c9debad872b0e8439 - e4baefca81fd72c45ed25851c96991e2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 09:29:49 (1766413789) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:30:03 (1766413803) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:30:11 (1766413811) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137961 s, 7.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237969 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 09:30:19 (1766413819) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23321 s, 9.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.462712 s, 9.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261239 s, 8.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 09:30:38 (1766413838) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:30:42 (1766413842) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:30:46 (1766413846) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334018 s, 31.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 09:30:55 (1766413855) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:30:58 (1766413858) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:31:06 (1766413866) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237972 s, 44.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:31:16 (1766413876) llite.lustre-ffff941c82bfb000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.251365 s, 41.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff941ca01a7800.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:31:28 (1766413888) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:32:07 (1766413927) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:32:45 (1766413965) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:32:47 (1766413967) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.222737 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.120163 s, 26.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdaf21411 (stale) Mirror 2: 0xdaf21411 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: d5987e8fd1284b410d9ce4cb105e602f - mirror 1 checksum: 32893bf7e50d92693524f3c0207554d5 - mirror 2 checksum: d5987e8fd1284b410d9ce4cb105e602f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 22 09:32:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 22 09:32:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d5987e8fd1284b410d9ce4cb105e602f - mirror 1 checksum: 32893bf7e50d92693524f3c0207554d5 - mirror 2 checksum: 32893bf7e50d92693524f3c0207554d5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 22 09:32:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 22 09:32:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:32:57 (1766413977) Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server 09:33:03 (1766413983) shut down facet: mds1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover mds1 to oleg205-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 09:33:25 (1766414005) targets are mounted 09:33:25 (1766414005) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369335 s, 28.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (41s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:33:38 (1766414018) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:33:41 (1766414021) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:33:44 (1766414024) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00813299 s, 2.0 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:33:50 (1766414030) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:33:53 (1766414033) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:33:55 (1766414035) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:34:18 (1766414058) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:34:46 (1766414086) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:35:10 (1766414110) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:35:35 (1766414135) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 09:35:42 (1766414142) mirror create pid 52274 mirror split pid 52275 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7638: 52275 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52274 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:36:48 (1766414208) Seed set to 4210 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1766414211.040160 mapwrite 0x036acc thru 0x03ffff (0x9534 bytes) 000004 1766414211.182431 mapwrite 0x001002 thru 0x00287f (0x187e bytes) 000009 1766414211.269631 read 0x01ef59 thru 0x028d7c (0x9e24 bytes) 000012 1766414211.292007 write 0x018b5e thru 0x021a4a (0x8eed bytes) 000013 1766414211.311733 mapread 0x001977 thru 0x00dbb8 (0xc242 bytes) 000015 1766414211.313061 mapwrite 0x016245 thru 0x01b285 (0x5041 bytes) 000016 1766414211.397384 write 0x00ba97 thru 0x010ede (0x5448 bytes) truncating to largest ever: 0xbce3 000017 1766414211.404676 trunc from 0x040000 to 0x00bce2 (0x3431d bytes) 000021 1766414211.426755 mapread 0x0042e9 thru 0x00bce2 (0x79fa bytes) 000024 1766414211.432392 mapwrite 0x016bb7 thru 0x01a466 (0x38b0 bytes) 000028 1766414211.521317 mapread 0x013156 thru 0x01a466 (0x7311 bytes) 000030 1766414211.524906 read 0x00241c thru 0x007b00 (0x56e5 bytes) 000031 1766414211.525467 write 0x00c18f thru 0x012225 (0x6097 bytes) 000032 1766414211.538329 mapread 0x00602e thru 0x014df7 (0xedca bytes) 000034 1766414211.540591 read 0x0040e2 thru 0x007d62 (0x3c81 bytes) 000036 1766414211.542342 trunc from 0x01a467 to 0x0094aa (0x10fbc bytes) 000039 1766414211.572585 trunc from 0x0094ab to 0x00aad0 (0x1626 bytes) 000040 1766414211.591586 mapread 0x00470d thru 0x00aad0 (0x63c4 bytes) 000042 1766414211.599787 mapwrite 0x00bf2d thru 0x019443 (0xd517 bytes) 000043 1766414211.721243 write 0x03577d thru 0x03dd44 (0x85c8 bytes) 000045 1766414211.723869 read 0x02ae31 thru 0x02ded5 (0x30a5 bytes) 000047 1766414211.737119 mapwrite 0x02892a thru 0x037caf (0xf386 bytes) truncating to largest ever: 0x2f4c2 000051 1766414211.805079 trunc from 0x03dd45 to 0x02f4c1 (0xe883 bytes) 000052 1766414211.832173 read 0x005125 thru 0x012e37 (0xdd13 bytes) 000053 1766414211.833026 trunc from 0x02f4c2 to 0x018404 (0x170bd bytes) 000054 1766414211.859017 read 0x012a56 thru 0x018404 (0x59af bytes) 000056 1766414211.860962 read 0x007f58 thru 0x00de88 (0x5f31 bytes) 000057 1766414211.861555 mapread 0x012816 thru 0x018404 (0x5bef bytes) 000059 1766414211.862851 write 0x02686c thru 0x02e963 (0x80f8 bytes) 000060 1766414211.864225 mapwrite 0x02eaf7 thru 0x030ab5 (0x1fbf bytes) 000062 1766414211.944632 mapread 0x020e33 thru 0x02594a (0x4b18 bytes) 000068 1766414211.947928 mapread 0x0299d4 thru 0x030ab5 (0x70e2 bytes) 000069 1766414211.948930 mapwrite 0x0069cd thru 0x0105a7 (0x9bdb bytes) 000070 1766414212.006024 read 0x02b436 thru 0x030ab5 (0x5680 bytes) 000072 1766414212.008178 mapread 0x030366 thru 0x030ab5 (0x750 bytes) truncating to largest ever: 0x336fb 000073 1766414212.010697 trunc from 0x030ab6 to 0x0336fa (0x2c45 bytes) 000076 1766414212.037071 MIRROR_EXTEND to 2 mirrors 000077 1766414212.709318 mapread 0x021d53 thru 0x02c826 (0xaad4 bytes) 000079 1766414212.830918 MIRROR_EXTEND to 3 mirrors 000080 1766414213.511815 mapread 0x023fdc thru 0x02b2f7 (0x731c bytes) 000082 1766414213.537968 write 0x02bec6 thru 0x03a3fd (0xe538 bytes) 000084 1766414213.608963 mapread 0x0336a3 thru 0x03a3fd (0x6d5b bytes) 000085 1766414213.609928 read 0x017941 thru 0x018b31 (0x11f1 bytes) 000086 1766414213.620084 mapread 0x023138 thru 0x02a974 (0x783d bytes) truncating to largest ever: 0x37bc2 000087 1766414213.640748 trunc from 0x03a3fe to 0x037bc1 (0x283c bytes) 000088 1766414213.662305 read 0x008615 thru 0x009f35 (0x1921 bytes) 000089 1766414213.675936 mapread 0x01f958 thru 0x02485a (0x4f03 bytes) 000090 1766414213.676997 trunc from 0x037bc2 to 0x005aa1 (0x32120 bytes) 000092 1766414213.698776 mapread 0x002172 thru 0x005aa1 (0x3930 bytes) 000093 1766414213.699804 write 0x00eb7a thru 0x0134c0 (0x4947 bytes) 000094 1766414213.701234 mapwrite 0x00939d thru 0x013bae (0xa812 bytes) 000095 1766414213.810281 mapread 0x0014fc thru 0x009983 (0x8488 bytes) 000100 1766414213.836860 read 0x011320 thru 0x013bae (0x288f bytes) 000101 1766414213.837499 mapread 0x00a20b thru 0x00a491 (0x287 bytes) truncating to largest ever: 0x398f0 000102 1766414213.839958 trunc from 0x013baf to 0x0398ef (0x25d41 bytes) 000104 1766414213.862939 trunc from 0x0398f0 to 0x031f17 (0x79d8 bytes) 000105 1766414213.878092 mapread 0x031671 thru 0x031f17 (0x8a7 bytes) 000106 1766414213.888103 write 0x005bd4 thru 0x00d6d6 (0x7b03 bytes) 000107 1766414213.891565 read 0x00518e thru 0x00a70a (0x557d bytes) 000109 1766414213.900727 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31f18): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f18): Mirror 2: 0x2c93be85 (stale) Mirror 3: 0x2c93be85 (stale) CRC-32 checksum value for chunk [0, 0x31f18): Mirror 1: 0x5e2f1e66 Mirror 4: 0x5e2f1e66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f18: skipped 000110 1766414214.642063 mapwrite 0x03cce1 thru 0x03f32f (0x264f bytes) 000111 1766414214.812363 mapread 0x0281bf thru 0x02e61e (0x6460 bytes) 000112 1766414214.828410 write 0x023f17 thru 0x02a879 (0x6963 bytes) 000114 1766414214.831409 mapread 0x02eda9 thru 0x03dff3 (0xf24b bytes) 000115 1766414214.840812 write 0x019bae thru 0x028696 (0xeae9 bytes) 000117 1766414214.851172 mapwrite 0x035bb0 thru 0x03ffff (0xa450 bytes) 000120 1766414214.914201 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65f99a66 (stale) Mirror 4: 0xc1d9b5d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65f99a66 (stale) Mirror 4: 0xc1d9b5d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000121 1766414215.214779 mapwrite 0x02feb8 thru 0x0348ce (0x4a17 bytes) 000123 1766414215.288790 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65f99a66 (stale) Mirror 4: 0xc1d9b5d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f8b2977 Mirror 5: 0x6f8b2977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000124 1766414216.019033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65f99a66 (stale) Mirror 4: 0xc1d9b5d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f8b2977 Mirror 5: 0x6f8b2977 Mirror 6: 0x6f8b2977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000127 1766414216.879782 mapread 0x00a9ec thru 0x017dc0 (0xd3d5 bytes) 000128 1766414216.925578 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc1d9b5d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc1d9b5d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f8b2977 Mirror 5: 0x6f8b2977 Mirror 6: 0x6f8b2977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000129 1766414217.386698 read 0x03a959 thru 0x03ffff (0x56a7 bytes) 000130 1766414217.400369 MIRROR_RESYNC 4 mirrors 000134 1766414218.071864 mapread 0x036a2a thru 0x03e287 (0x785e bytes) 000135 1766414218.095894 write 0x02c74c thru 0x02d84c (0x1101 bytes) 000136 1766414218.176570 read 0x01fbaa thru 0x021049 (0x14a0 bytes) 000137 1766414218.182204 trunc from 0x040000 to 0x0002bb (0x3fd44 bytes) 000139 1766414218.223052 trunc from 0x0002bc to 0x011109 (0x10e4e bytes) 000140 1766414218.237976 read 0x00af13 thru 0x011109 (0x61f7 bytes) 000141 1766414218.247414 mapwrite 0x003c48 thru 0x006dba (0x3173 bytes) 000142 1766414218.298803 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1110a): Mirror 5: 0xcbae2a62 (stale) Mirror 6: 0xcbae2a62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1110a): Mirror 5: 0xcbae2a62 (stale) Mirror 6: 0xcbae2a62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1110a: skipped 000143 1766414218.632993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1110a): Mirror 6: 0xcbae2a62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1110a): Mirror 6: 0xcbae2a62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1110a: skipped 000146 1766414218.939566 write 0x026cc4 thru 0x02b2bc (0x45f9 bytes) 000147 1766414218.941307 trunc from 0x02b2bd to 0x026db2 (0x450a bytes) 000148 1766414218.965693 trunc from 0x026db3 to 0x0228b7 (0x44fb bytes) 000153 1766414219.005857 read 0x01a083 thru 0x02168c (0x760a bytes) 000156 1766414219.014488 mapwrite 0x02d1ff thru 0x03ba72 (0xe874 bytes) 000158 1766414219.156187 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba73): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba73): Mirror 6: 0x1447a6d1 (stale) CRC-32 checksum value for chunk [0, 0x3ba73): Mirror 1: 0x6b7f3569 Mirror 7: 0x6b7f3569 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba73: skipped 000159 1766414219.933902 mapwrite 0x01b17d thru 0x02542a (0xa2ae bytes) 000161 1766414220.161036 mapread 0x02ea87 thru 0x0318bb (0x2e35 bytes) 000167 1766414220.175138 read 0x025ecf thru 0x027c74 (0x1da6 bytes) 000169 1766414220.177471 read 0x02bb70 thru 0x02f1d9 (0x366a bytes) 000172 1766414220.179637 mapread 0x036d08 thru 0x03ba72 (0x4d6b bytes) 000173 1766414220.204823 trunc from 0x03ba73 to 0x00345b (0x38617 bytes) 000174 1766414220.262158 write 0x01ad26 thru 0x02a4a4 (0xf77f bytes) 000175 1766414220.263936 mapread 0x015d85 thru 0x01ea01 (0x8c7d bytes) 000177 1766414220.298986 mapwrite 0x01fcbc thru 0x02a500 (0xa845 bytes) 000178 1766414220.411884 trunc from 0x02a501 to 0x008fb3 (0x2154d bytes) 000180 1766414220.444030 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8fb4): Mirror 6: 0xe1fd0967 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8fb4): Mirror 6: 0xe1fd0967 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8fb4: skipped 000181 1766414220.829268 trunc from 0x008fb4 to 0x02c149 (0x23196 bytes) 000182 1766414220.850524 mapwrite 0x02558a thru 0x02e5fa (0x9071 bytes) 000187 1766414220.981212 read 0x01e9fb thru 0x02db8d (0xf193 bytes) 000189 1766414220.987976 mapread 0x00920c thru 0x009443 (0x238 bytes) 000195 1766414221.004807 mapread 0x006076 thru 0x00637d (0x308 bytes) 000200 1766414221.010937 trunc from 0x02e5fb to 0x011349 (0x1d2b1 bytes) 000201 1766414221.041380 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000202 1766414221.197258 write 0x003ba5 thru 0x00d34b (0x97a7 bytes) 000203 1766414221.216282 trunc from 0x01134a to 0x008d02 (0x8647 bytes) 000204 1766414221.239057 mapwrite 0x03d1d2 thru 0x03ffff (0x2e2e bytes) 000205 1766414221.344682 trunc from 0x040000 to 0x004180 (0x3be7f bytes) 000207 1766414221.376107 MIRROR_EXTEND to 2 mirrors 000209 1766414222.134333 trunc from 0x004181 to 0x027db8 (0x23c38 bytes) 000212 1766414222.222373 read 0x00c4da thru 0x019499 (0xcfc0 bytes) 000213 1766414222.234087 mapwrite 0x01c311 thru 0x024558 (0x8248 bytes) 000214 1766414222.299615 trunc from 0x027db9 to 0x02411b (0x3c9d bytes) 000215 1766414222.322828 trunc from 0x02411c to 0x02399f (0x77c bytes) 000217 1766414222.350597 mapread 0x010c67 thru 0x012960 (0x1cfa bytes) 000219 1766414222.352372 mapread 0x00d501 thru 0x00e287 (0xd87 bytes) truncating to largest ever: 0x3e75e 000221 1766414222.354728 trunc from 0x0239a0 to 0x03e75d (0x1adbe bytes) 000222 1766414222.375661 mapread 0x00f629 thru 0x0177b5 (0x818d bytes) 000224 1766414222.377104 mapwrite 0x03e475 thru 0x03f4d5 (0x1061 bytes) 000227 1766414222.445050 write 0x01b70f thru 0x01d208 (0x1afa bytes) 000228 1766414222.446127 read 0x026c62 thru 0x02f0cb (0x846a bytes) 000229 1766414222.464343 write 0x02b002 thru 0x0399d4 (0xe9d3 bytes) 000231 1766414222.466186 write 0x00de3b thru 0x016882 (0x8a48 bytes) 000232 1766414222.469420 mapwrite 0x035f38 thru 0x03ffff (0xa0c8 bytes) 000234 1766414222.575972 MIRROR_RESYNC 2 mirrors 000236 1766414223.333579 mapread 0x00697f thru 0x00bb3f (0x51c1 bytes) 000237 1766414223.390844 write 0x00f328 thru 0x0134ad (0x4186 bytes) 000240 1766414223.511915 write 0x0359f7 thru 0x03ffff (0xa609 bytes) 000242 1766414223.540762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb8b56cf6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x234fb1fb Mirror 3: 0x234fb1fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000243 1766414224.566516 write 0x027bf4 thru 0x0319dd (0x9dea bytes) 000245 1766414224.698373 mapread 0x019d8d thru 0x024adc (0xad50 bytes) 000246 1766414224.715862 write 0x021bc3 thru 0x02bfc5 (0xa403 bytes) 000247 1766414224.721155 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x234fb1fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x234fb1fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1766414225.087310 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x234fb1fb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85173750 Mirror 4: 0x85173750 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1766414225.985977 mapread 0x02aae0 thru 0x03651a (0xba3b bytes) 000250 1766414226.084853 MIRROR_RESYNC 3 mirrors 000256 1766414226.864358 read 0x02221a thru 0x030df9 (0xebe0 bytes) 000259 1766414226.879594 mapwrite 0x02dbbc thru 0x02dd18 (0x15d bytes) 000260 1766414227.057538 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85173750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85173750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000261 1766414227.384811 trunc from 0x040000 to 0x036ac5 (0x953a bytes) truncating to largest ever: 0x3ffb9 000262 1766414227.407787 trunc from 0x036ac6 to 0x03ffb8 (0x94f3 bytes) 000264 1766414227.426261 mapread 0x0363eb thru 0x03ffb8 (0x9bce bytes) 000266 1766414227.449131 trunc from 0x03ffb9 to 0x009105 (0x36eb3 bytes) 000268 1766414227.471300 mapread 0x005472 thru 0x009105 (0x3c94 bytes) 000270 1766414227.482605 trunc from 0x009106 to 0x0260c6 (0x1cfc1 bytes) 000272 1766414227.505019 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000273 1766414227.710733 mapwrite 0x03aec0 thru 0x03f4cd (0x460e bytes) 000275 1766414227.878420 trunc from 0x03f4ce to 0x00eddf (0x306ee bytes) 000278 1766414227.915396 read 0x00686d thru 0x007dd7 (0x156b bytes) 000280 1766414227.928128 mapwrite 0x00809c thru 0x016605 (0xe56a bytes) 000282 1766414228.063686 write 0x000987 thru 0x007d9a (0x7414 bytes) 000285 1766414228.078883 read 0x0056f7 thru 0x00e2a4 (0x8bae bytes) 000286 1766414228.081716 trunc from 0x016606 to 0x026eab (0x108a6 bytes) 000289 1766414228.125001 mapwrite 0x012059 thru 0x0178e1 (0x5889 bytes) 000290 1766414228.256180 mapwrite 0x02b884 thru 0x030a0f (0x518c bytes) 000291 1766414228.350050 mapwrite 0x02395c thru 0x0266c9 (0x2d6e bytes) 000292 1766414228.392230 write 0x01f354 thru 0x022e38 (0x3ae5 bytes) 000293 1766414228.393259 mapwrite 0x014048 thru 0x016c70 (0x2c29 bytes) 000294 1766414228.453387 write 0x00f3ee thru 0x012e50 (0x3a63 bytes) 000295 1766414228.454591 mapread 0x02415d thru 0x0262f3 (0x2197 bytes) 000297 1766414228.456475 read 0x00fba8 thru 0x00fd8f (0x1e8 bytes) 000298 1766414228.457037 trunc from 0x030a10 to 0x03c640 (0xbc31 bytes) 000299 1766414228.479982 mapread 0x021971 thru 0x02bfdf (0xa66f bytes) 000303 1766414228.481904 write 0x020a45 thru 0x02437f (0x393b bytes) 000304 1766414228.482720 read 0x030d5a thru 0x0360dc (0x5383 bytes) 000305 1766414228.492069 trunc from 0x03c641 to 0x009af3 (0x32b4d bytes) 000308 1766414228.526382 MIRROR_EXTEND to 2 mirrors 000309 1766414229.153691 write 0x003ea2 thru 0x00db50 (0x9caf bytes) 000311 1766414229.257228 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000313 1766414229.425269 read 0x00b6e5 thru 0x00db50 (0x246c bytes) 000316 1766414229.441966 mapread 0x0040dc thru 0x00db50 (0x9a75 bytes) 000317 1766414229.474009 mapwrite 0x03fc4f thru 0x03ffff (0x3b1 bytes) 000319 1766414229.634957 trunc from 0x040000 to 0x03aa2e (0x55d1 bytes) 000320 1766414229.650344 mapread 0x02ea09 thru 0x038f49 (0xa541 bytes) 000323 1766414229.671875 MIRROR_EXTEND to 2 mirrors 000324 1766414230.354511 mapwrite 0x01380d thru 0x01ab8b (0x737f bytes) 000325 1766414230.527518 read 0x01dd83 thru 0x02c19a (0xe418 bytes) 000326 1766414230.539637 trunc from 0x03aa2f to 0x031551 (0x94dd bytes) 000327 1766414230.556949 write 0x003a79 thru 0x005420 (0x19a8 bytes) 000330 1766414230.569881 write 0x012680 thru 0x01eae8 (0xc469 bytes) 000333 1766414230.572834 read 0x031515 thru 0x031551 (0x3d bytes) 000334 1766414230.589536 read 0x00ea18 thru 0x012b70 (0x4159 bytes) 000336 1766414230.591755 read 0x00cf48 thru 0x01c7cd (0xf886 bytes) 000338 1766414230.597092 mapwrite 0x02d212 thru 0x03a36d (0xd15c bytes) 000339 1766414230.705359 mapwrite 0x035594 thru 0x035cc6 (0x733 bytes) 000341 1766414230.748357 mapwrite 0x034cf3 thru 0x03ffff (0xb30d bytes) 000342 1766414230.826096 trunc from 0x040000 to 0x02e05b (0x11fa4 bytes) 000343 1766414230.848146 write 0x039bf3 thru 0x03ffff (0x640d bytes) 000344 1766414230.849269 read 0x03e05d thru 0x03ffff (0x1fa3 bytes) 000345 1766414230.849849 trunc from 0x040000 to 0x023b50 (0x1c4af bytes) 000346 1766414230.880323 MIRROR_RESYNC 2 mirrors 000350 1766414231.574699 write 0x03f7cb thru 0x03ffff (0x835 bytes) 000351 1766414231.659372 mapread 0x00b9f7 thru 0x01135a (0x5964 bytes) 000358 1766414231.684933 mapwrite 0x039e56 thru 0x03ffff (0x61aa bytes) 000359 1766414231.742652 write 0x01def1 thru 0x02ca37 (0xeb47 bytes) 000360 1766414231.774284 trunc from 0x040000 to 0x0110db (0x2ef24 bytes) 000361 1766414231.806832 write 0x023808 thru 0x02f4c1 (0xbcba bytes) 000362 1766414231.809001 mapread 0x00f98c thru 0x015379 (0x59ee bytes) 000363 1766414231.819623 write 0x02c472 thru 0x032514 (0x60a3 bytes) 000364 1766414231.821062 read 0x003f86 thru 0x008cfb (0x4d76 bytes) 000365 1766414231.832575 MIRROR_RESYNC 2 mirrors 000366 1766414232.418709 trunc from 0x032515 to 0x01e3a8 (0x1416c bytes) 000367 1766414232.504273 mapwrite 0x00f921 thru 0x01b88e (0xbf6e bytes) 000368 1766414232.600748 mapwrite 0x015eb1 thru 0x020f66 (0xb0b6 bytes) 000369 1766414232.690853 write 0x039d72 thru 0x03ffff (0x628e bytes) 000370 1766414232.692094 read 0x03a364 thru 0x03eca8 (0x4945 bytes) 000371 1766414232.692690 write 0x016e3c thru 0x025ae7 (0xecac bytes) 000373 1766414232.698330 mapread 0x031923 thru 0x03d4ce (0xbbac bytes) 000374 1766414232.709375 read 0x00b761 thru 0x011ab2 (0x6352 bytes) 000375 1766414232.710827 mapwrite 0x00526c thru 0x011fb8 (0xcd4d bytes) 000376 1766414232.780544 write 0x0210da thru 0x027229 (0x6150 bytes) 000377 1766414232.781600 mapread 0x028556 thru 0x02b27b (0x2d26 bytes) 000378 1766414232.782372 read 0x0310c2 thru 0x038cce (0x7c0d bytes) 000379 1766414232.782927 read 0x014653 thru 0x01497e (0x32c bytes) 000383 1766414232.785523 trunc from 0x040000 to 0x0141c1 (0x2be3e bytes) 000384 1766414232.810708 mapwrite 0x005ca9 thru 0x012597 (0xc8ef bytes) 000386 1766414232.882128 trunc from 0x0141c2 to 0x03ae41 (0x26c80 bytes) 000390 1766414232.906519 trunc from 0x03ae42 to 0x0373b1 (0x3a90 bytes) 000391 1766414232.925897 mapwrite 0x019634 thru 0x01bdad (0x277a bytes) 000392 1766414232.985062 trunc from 0x0373b2 to 0x016d09 (0x206a8 bytes) 000393 1766414233.007688 write 0x028a97 thru 0x030c3b (0x81a5 bytes) 000394 1766414233.008928 mapread 0x010107 thru 0x0159e5 (0x58df bytes) 000395 1766414233.017022 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30c3c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c3c): Mirror 2: 0x65179d09 (stale) CRC-32 checksum value for chunk [0, 0x30c3c): Mirror 1: 0x6d8d025a Mirror 3: 0x6d8d025a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c3c: skipped 000396 1766414233.826158 write 0x03223a thru 0x037f60 (0x5d27 bytes) 000400 1766414233.904678 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f61): Mirror 3: 0x2fc9137b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f61): Mirror 3: 0x2fc9137b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f61: skipped 000405 1766414234.224875 trunc from 0x037f61 to 0x02c188 (0xbdd8 bytes) 000407 1766414234.267165 trunc from 0x02c189 to 0x012d02 (0x19486 bytes) 000408 1766414234.282540 mapread 0x00b893 thru 0x01169c (0x5e0a bytes) 000409 1766414234.317688 MIRROR_RESYNC 2 mirrors 000410 1766414234.908375 read 0x01236d thru 0x012d02 (0x996 bytes) 000411 1766414234.916554 trunc from 0x012d03 to 0x02c18a (0x19488 bytes) 000412 1766414235.050957 trunc from 0x02c18b to 0x007767 (0x24a23 bytes) 000413 1766414235.086710 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7768): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7768): Mirror 3: 0x9505e17b (stale) CRC-32 checksum value for chunk [0, 0x7768): Mirror 1: 0x9505e17b Mirror 4: 0x9505e17b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7768: skipped 000414 1766414235.982114 trunc from 0x007768 to 0x027ba7 (0x20440 bytes) 000415 1766414236.083351 write 0x028986 thru 0x02e071 (0x56ec bytes) 000416 1766414236.089675 MIRROR_RESYNC 3 mirrors 000418 1766414236.980325 read 0x00e8b4 thru 0x01574d (0x6e9a bytes) 000419 1766414236.997337 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000421 1766414237.336162 read 0x0167e2 thru 0x018489 (0x1ca8 bytes) 000422 1766414237.348487 read 0x004344 thru 0x010ac5 (0xc782 bytes) 000423 1766414237.359336 read 0x01527b thru 0x01617b (0xf01 bytes) 000424 1766414237.359953 read 0x01e8d9 thru 0x0276ed (0x8e15 bytes) 000425 1766414237.372556 MIRROR_EXTEND to 3 mirrors 000427 1766414238.059059 trunc from 0x02e072 to 0x00589a (0x287d7 bytes) 000428 1766414238.161311 mapwrite 0x03ed58 thru 0x03ffff (0x12a8 bytes) 000430 1766414238.258655 trunc from 0x040000 to 0x01028b (0x2fd74 bytes) 000431 1766414238.287342 trunc from 0x01028c to 0x02eceb (0x1ea60 bytes) 000433 1766414238.317917 mapwrite 0x01d737 thru 0x0209c6 (0x3290 bytes) 000434 1766414238.397815 mapread 0x0264b3 thru 0x029811 (0x335f bytes) 000435 1766414238.429841 mapwrite 0x0229d8 thru 0x0269a2 (0x3fcb bytes) 000439 1766414238.507890 read 0x0192de thru 0x027348 (0xe06b bytes) 000442 1766414238.518402 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 4: 0x943f8031 (stale) Mirror 5: 0x943f8031 (stale) CRC-32 checksum value for chunk [0, 0x2ecec): Mirror 1: 0xac60fdf6 Mirror 6: 0xac60fdf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecec: skipped 000443 1766414239.346613 mapread 0x01e0fc thru 0x02ab8d (0xca92 bytes) 000444 1766414239.414635 mapwrite 0x0078ce thru 0x0078ef (0x22 bytes) 000445 1766414239.650067 mapwrite 0x021cd3 thru 0x0257a7 (0x3ad5 bytes) 000449 1766414239.750258 trunc from 0x02ecec to 0x03019d (0x14b2 bytes) 000450 1766414239.821989 read 0x00cca5 thru 0x00fc00 (0x2f5c bytes) 000452 1766414239.830624 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3019e): Mirror 5: 0xc65ef5ab (stale) Mirror 6: 0xc5701c54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3019e): Mirror 5: 0xc65ef5ab (stale) Mirror 6: 0xc5701c54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3019e: skipped 000455 1766414240.169762 write 0x024fd3 thru 0x02dc9e (0x8ccc bytes) 000456 1766414240.179889 mapwrite 0x021af6 thru 0x0227f1 (0xcfc bytes) 000457 1766414240.231737 write 0x0284e1 thru 0x02e115 (0x5c35 bytes) 000458 1766414240.242409 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3019e): Mirror 5: 0xc65ef5ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3019e): Mirror 5: 0xc65ef5ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3019e: skipped 000461 1766414240.566922 mapread 0x02d0ee thru 0x03019d (0x30b0 bytes) 000463 1766414240.591659 mapread 0x023ea1 thru 0x027f6a (0x40ca bytes) 000464 1766414240.599043 mapread 0x01886e thru 0x022921 (0xa0b4 bytes) 000467 1766414240.609980 mapwrite 0x02614e thru 0x0281f1 (0x20a4 bytes) 000468 1766414240.679399 trunc from 0x03019e to 0x00bb23 (0x2467a bytes) 000469 1766414240.710120 write 0x0068ef thru 0x00ae4f (0x4561 bytes) 000470 1766414240.721717 read 0x0006b6 thru 0x00bb23 (0xb46e bytes) 000473 1766414240.729184 read 0x000853 thru 0x005f7d (0x572b bytes) 000474 1766414240.729883 mapread 0x00a4fc thru 0x00ad86 (0x88b bytes) 000475 1766414240.731005 write 0x00919b thru 0x017539 (0xe39f bytes) 000479 1766414240.736710 read 0x0052c7 thru 0x00aafb (0x5835 bytes) 000482 1766414240.738399 mapread 0x01015c thru 0x0102ae (0x153 bytes) 000485 1766414240.746720 read 0x016b5e thru 0x017539 (0x9dc bytes) 000487 1766414240.759236 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000489 1766414240.948015 MIRROR_EXTEND to 2 mirrors 000490 1766414241.584213 write 0x013350 thru 0x01610e (0x2dbf bytes) 000491 1766414241.672227 trunc from 0x01753a to 0x03d0bc (0x25b83 bytes) 000492 1766414241.695342 trunc from 0x03d0bd to 0x028fad (0x1410f bytes) 000493 1766414241.717793 trunc from 0x028fae to 0x014338 (0x14c75 bytes) 000497 1766414241.740992 trunc from 0x014339 to 0x002833 (0x11b05 bytes) 000498 1766414241.759981 mapread 0x001b14 thru 0x002833 (0xd20 bytes) 000499 1766414241.770679 mapwrite 0x035904 thru 0x037505 (0x1c02 bytes) 000500 1766414241.886839 read 0x02d434 thru 0x037505 (0xa0d2 bytes) 000502 1766414241.893811 mapread 0x019893 thru 0x01d5f3 (0x3d61 bytes) 000506 1766414241.913086 mapread 0x030577 thru 0x030cd0 (0x75a bytes) 000508 1766414241.914607 mapwrite 0x02a9bc thru 0x035b1d (0xb162 bytes) 000509 1766414241.982686 MIRROR_RESYNC 2 mirrors 000510 1766414242.577846 read 0x00c80a thru 0x01c42d (0xfc24 bytes) 000511 1766414242.583510 read 0x01a8b6 thru 0x0232b6 (0x8a01 bytes) 000513 1766414242.594358 mapread 0x011a83 thru 0x0124f3 (0xa71 bytes) 000514 1766414242.594925 mapwrite 0x0022bd thru 0x0034c5 (0x1209 bytes) 000519 1766414242.749591 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) 000520 1766414242.902166 write 0x02287a thru 0x031036 (0xe7bd bytes) 000521 1766414242.920670 read 0x028d66 thru 0x02addc (0x2077 bytes) 000522 1766414242.921200 write 0x0342b8 thru 0x0383ba (0x4103 bytes) 000525 1766414242.954056 write 0x02e97d thru 0x0380be (0x9742 bytes) 000527 1766414242.958895 mapread 0x010c54 thru 0x01c938 (0xbce5 bytes) 000528 1766414243.002604 mapread 0x02fde6 thru 0x038071 (0x828c bytes) 000531 1766414243.004273 read 0x010f15 thru 0x013f86 (0x3072 bytes) 000532 1766414243.004850 write 0x01a022 thru 0x01a658 (0x637 bytes) 000537 1766414243.014773 mapwrite 0x02ac0c thru 0x030420 (0x5815 bytes) 000538 1766414243.087359 mapwrite 0x02acdf thru 0x031a65 (0x6d87 bytes) 000540 1766414243.133219 write 0x030bda thru 0x03f6ad (0xead4 bytes) 000541 1766414243.134647 write 0x0092ba thru 0x00c166 (0x2ead bytes) 000542 1766414243.135407 write 0x029c2b thru 0x03782c (0xdc02 bytes) 000543 1766414243.137291 mapwrite 0x0142d7 thru 0x01ea07 (0xa731 bytes) 000544 1766414243.193323 read 0x03ba67 thru 0x03c8cc (0xe66 bytes) 000545 1766414243.193981 mapwrite 0x01329a thru 0x01db97 (0xa8fe bytes) 000547 1766414243.256130 MIRROR_EXTEND to 2 mirrors 000548 1766414243.872062 mapread 0x02a5bf thru 0x039bd2 (0xf614 bytes) 000549 1766414243.928851 MIRROR_EXTEND to 3 mirrors 000551 1766414244.462378 mapread 0x009fd5 thru 0x019585 (0xf5b1 bytes) 000553 1766414244.509426 read 0x0103a6 thru 0x010c96 (0x8f1 bytes) 000554 1766414244.509761 mapread 0x03c5a5 thru 0x03f6ad (0x3109 bytes) 000556 1766414244.520524 read 0x0097c5 thru 0x0197b1 (0xffed bytes) 000558 1766414244.521405 write 0x028c21 thru 0x0351d6 (0xc5b6 bytes) 000560 1766414244.618218 write 0x0240fb thru 0x02aedf (0x6de5 bytes) 000561 1766414244.628438 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f6ae): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6ae): Mirror 2: 0xb2823095 (stale) Mirror 3: 0xb2823095 (stale) CRC-32 checksum value for chunk [0, 0x3f6ae): Mirror 1: 0x5c48ed86 Mirror 4: 0x5c48ed86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6ae: skipped 000562 1766414245.497927 mapread 0x01f80d thru 0x02c5c9 (0xcdbd bytes) 000563 1766414245.581816 trunc from 0x03f6ae to 0x038f5f (0x674e bytes) 000564 1766414245.675710 mapwrite 0x02c801 thru 0x036578 (0x9d78 bytes) 000565 1766414245.747292 read 0x0176ff thru 0x01f3cd (0x7ccf bytes) 000567 1766414245.758676 read 0x00ab8b thru 0x014153 (0x95c9 bytes) 000568 1766414245.767126 write 0x02c5de thru 0x037bac (0xb5cf bytes) 000569 1766414245.768303 trunc from 0x038f60 to 0x028d71 (0x101ee bytes) 000572 1766414245.798587 mapwrite 0x022e64 thru 0x0312d3 (0xe470 bytes) 000573 1766414245.954826 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x312d4): Mirror 3: 0x4c0c1fc1 (stale) Mirror 4: 0xf3696749 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312d4): Mirror 3: 0x4c0c1fc1 (stale) Mirror 4: 0xf3696749 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312d4: skipped 000574 1766414246.342408 read 0x01f6bf thru 0x02cd81 (0xd6c3 bytes) 000578 1766414246.359381 mapwrite 0x01635c thru 0x022bb5 (0xc85a bytes) 000579 1766414246.443615 mapread 0x02924b thru 0x02b66a (0x2420 bytes) 000581 1766414246.449615 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x312d4): Mirror 4: 0xf3696749 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312d4): Mirror 4: 0xf3696749 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312d4: skipped 000582 1766414246.762073 mapwrite 0x02c455 thru 0x033d8e (0x793a bytes) 000583 1766414246.878747 read 0x01ecbf thru 0x02556a (0x68ac bytes) 000585 1766414246.889795 mapread 0x0184d8 thru 0x025244 (0xcd6d bytes) 000586 1766414246.898157 trunc from 0x033d8f to 0x035f46 (0x21b8 bytes) 000588 1766414246.921572 write 0x020bed thru 0x02d117 (0xc52b bytes) 000589 1766414246.922781 mapread 0x0179d5 thru 0x023727 (0xbd53 bytes) 000591 1766414246.924711 mapread 0x02cc32 thru 0x030d48 (0x4117 bytes) 000592 1766414246.926331 mapwrite 0x0281da thru 0x030c6c (0x8a93 bytes) 000593 1766414247.003708 read 0x02e42f thru 0x0341b1 (0x5d83 bytes) 000594 1766414247.013153 mapwrite 0x031908 thru 0x03ffff (0xe6f8 bytes) 000595 1766414247.088599 read 0x002f8e thru 0x004c45 (0x1cb8 bytes) 000596 1766414247.093762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d6261f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7240c9ce Mirror 5: 0x7240c9ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1766414247.997709 read 0x00a337 thru 0x00eb25 (0x47ef bytes) 000599 1766414248.025981 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d6261f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d6261f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1766414248.421829 write 0x020e63 thru 0x0241ab (0x3349 bytes) 000601 1766414248.456694 trunc from 0x040000 to 0x02571e (0x1a8e1 bytes) 000604 1766414248.508876 read 0x009305 thru 0x01524e (0xbf4a bytes) 000605 1766414248.525074 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2571f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2571f): Mirror 4: 0xd18ad7ed (stale) CRC-32 checksum value for chunk [0, 0x2571f): Mirror 1: 0x284aaba7 Mirror 5: 0x284aaba7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2571f: skipped 000606 1766414249.177843 mapwrite 0x02bb8f thru 0x038d42 (0xd1b4 bytes) 000607 1766414249.433748 mapread 0x010892 thru 0x01ae20 (0xa58f bytes) 000609 1766414249.461222 trunc from 0x038d43 to 0x00f03f (0x29d03 bytes) 000610 1766414249.490976 mapwrite 0x002dea thru 0x005072 (0x2289 bytes) 000612 1766414249.558385 trunc from 0x00f040 to 0x000a6e (0xe5d1 bytes) 000613 1766414249.609097 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa6f): Mirror 5: 0x84542e71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa6f): Mirror 5: 0x84542e71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa6f: skipped 000615 1766414249.927021 write 0x021f7d thru 0x02b6bf (0x9743 bytes) 000616 1766414249.929199 mapwrite 0x031a1e thru 0x03c474 (0xaa57 bytes) 000618 1766414250.079254 read 0x014dc4 thru 0x01fe0c (0xb049 bytes) 000619 1766414250.088865 trunc from 0x03c475 to 0x039915 (0x2b5f bytes) 000620 1766414250.138177 MIRROR_RESYNC 2 mirrors 000622 1766414250.896991 mapread 0x00156a thru 0x00de8f (0xc926 bytes) 000624 1766414250.926037 mapread 0x034c06 thru 0x039915 (0x4d10 bytes) 000626 1766414250.949934 read 0x031473 thru 0x0379f8 (0x6586 bytes) 000627 1766414250.951795 mapwrite 0x018b63 thru 0x01fe03 (0x72a1 bytes) 000628 1766414251.159172 read 0x01b539 thru 0x02420a (0x8cd2 bytes) 000630 1766414251.186984 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39916): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39916): Mirror 5: 0xe02f6012 (stale) CRC-32 checksum value for chunk [0, 0x39916): Mirror 1: 0x9c970433 Mirror 6: 0x9c970433 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39916: skipped 000631 1766414252.051884 mapread 0x007668 thru 0x014fef (0xd988 bytes) 000633 1766414252.117693 read 0x028ad5 thru 0x029e95 (0x13c1 bytes) 000634 1766414252.134583 write 0x01a9c6 thru 0x01b82d (0xe68 bytes) 000636 1766414252.231341 mapread 0x00d40a thru 0x00def4 (0xaeb bytes) 000637 1766414252.249646 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39916): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39916): Mirror 5: 0xe02f6012 (stale) Mirror 6: 0x9c970433 (stale) CRC-32 checksum value for chunk [0, 0x39916): Mirror 1: 0xc994b15a Mirror 7: 0xc994b15a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39916: skipped 000638 1766414253.010392 write 0x021b92 thru 0x030a98 (0xef07 bytes) 000640 1766414253.115781 trunc from 0x039916 to 0x01cc61 (0x1ccb4 bytes) 000643 1766414253.162058 write 0x034085 thru 0x03ffff (0xbf7b bytes) 000644 1766414253.164694 mapwrite 0x0343d8 thru 0x03fcdc (0xb905 bytes) 000645 1766414253.274248 MIRROR_RESYNC 4 mirrors 000647 1766414254.174386 MIRROR_RESYNC 4 mirrors 000649 1766414254.818989 read 0x0154b0 thru 0x020823 (0xb374 bytes) 000650 1766414254.826677 mapread 0x0388e5 thru 0x03b79e (0x2eba bytes) 000651 1766414254.842742 write 0x03e368 thru 0x03ffff (0x1c98 bytes) 000652 1766414254.915523 MIRROR_RESYNC 4 mirrors 000654 1766414255.798930 mapwrite 0x01d40d thru 0x0280b6 (0xacaa bytes) 000658 1766414256.008091 MIRROR_RESYNC 4 mirrors 000661 1766414256.946187 write 0x02727d thru 0x02bc0f (0x4993 bytes) 000662 1766414257.015722 mapwrite 0x004968 thru 0x00d6b5 (0x8d4e bytes) 000663 1766414257.076529 mapwrite 0x00239e thru 0x00a88f (0x84f2 bytes) 000664 1766414257.119681 read 0x02da05 thru 0x030303 (0x28ff bytes) 000668 1766414257.127238 mapread 0x02be70 thru 0x030dea (0x4f7b bytes) 000669 1766414257.140894 mapread 0x00845b thru 0x013160 (0xad06 bytes) 000671 1766414257.156031 mapwrite 0x037456 thru 0x03b3e4 (0x3f8f bytes) 000674 1766414257.213152 mapread 0x017425 thru 0x01ba5c (0x4638 bytes) 000675 1766414257.229088 MIRROR_RESYNC 4 mirrors 000677 1766414258.117669 read 0x0261c5 thru 0x033865 (0xd6a1 bytes) 000679 1766414258.126197 trunc from 0x040000 to 0x038458 (0x7ba7 bytes) 000681 1766414258.250797 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38459): Mirror 6: 0x693ebcb6 (stale) Mirror 7: 0x693ebcb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38459): Mirror 6: 0x693ebcb6 (stale) Mirror 7: 0x693ebcb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38459: skipped 000686 1766414258.666500 read 0x0279ef thru 0x032823 (0xae35 bytes) 000687 1766414258.682199 mapwrite 0x03a5b7 thru 0x03ffff (0x5a49 bytes) 000689 1766414258.845632 read 0x01a62b thru 0x01db80 (0x3556 bytes) 000690 1766414258.879902 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x599da35a (stale) Mirror 7: 0x599da35a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x220edd6 Mirror 8: 0x220edd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000691 1766414259.795633 write 0x0294a9 thru 0x02c80b (0x3363 bytes) 000692 1766414259.934607 mapwrite 0x035711 thru 0x03ffff (0xa8ef bytes) 000693 1766414260.074691 MIRROR_RESYNC 4 mirrors 000696 1766414260.908429 trunc from 0x040000 to 0x03355b (0xcaa4 bytes) 000697 1766414261.008597 trunc from 0x03355c to 0x00fb84 (0x239d7 bytes) 000698 1766414261.060035 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xfb85): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfb85): Mirror 6: 0xfa6677d6 (stale) Mirror 7: 0xfa6677d6 (stale) Mirror 8: 0xfa6677d6 (stale) CRC-32 checksum value for chunk [0, 0xfb85): Mirror 1: 0xfa6677d6 Mirror 9: 0xfa6677d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfb85: skipped 000699 1766414262.196947 mapwrite 0x0313e9 thru 0x03f586 (0xe19e bytes) 000703 1766414262.469126 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f587): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f587): Mirror 6: 0xe6a232be (stale) Mirror 7: 0xe6a232be (stale) Mirror 8: 0xe6a232be (stale) Mirror 9: 0xdae1f608 (stale) CRC-32 checksum value for chunk [0, 0x3f587): Mirror 1: 0x66c1b7f8 Mirror 10: 0x66c1b7f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f587: skipped 000707 1766414263.397725 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f587): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f587): Mirror 6: 0xe6a232be (stale) Mirror 7: 0xe6a232be (stale) Mirror 8: 0xe6a232be (stale) Mirror 9: 0xdae1f608 (stale) CRC-32 checksum value for chunk [0, 0x3f587): Mirror 1: 0x66c1b7f8 Mirror 10: 0x66c1b7f8 Mirror 11: 0x66c1b7f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f587: skipped 000709 1766414264.459705 mapread 0x0284af thru 0x02d72f (0x5281 bytes) 000711 1766414264.483847 trunc from 0x03f587 to 0x03a30f (0x5277 bytes) 000714 1766414264.657655 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: 0xa2a8bb30 (stale) Mirror 7: 0xa2a8bb30 (stale) Mirror 8: 0xa2a8bb30 (stale) Mirror 9: 0xf2bfe0ca (stale) Mirror 10: 0x3ebcab29 (stale) Mirror 11: 0x3ebcab29 (stale) CRC-32 checksum value for chunk [0, 0x3a310): Mirror 1: 0x3ebcab29 Mirror 12: 0x3ebcab29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a310: skipped 000715 1766414265.774072 mapwrite 0x00036d thru 0x00bf4b (0xbbdf bytes) 000718 1766414266.045927 read 0x031ea7 thru 0x0392bc (0x7416 bytes) 000719 1766414266.060056 mapwrite 0x029853 thru 0x02d72d (0x3edb bytes) 000721 1766414266.161347 mapread 0x030a55 thru 0x03589a (0x4e46 bytes) 000722 1766414266.173730 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: 0xa2a8bb30 (stale) Mirror 7: 0xa2a8bb30 (stale) Mirror 9: 0xf2bfe0ca (stale) Mirror 10: 0x3ebcab29 (stale) Mirror 11: 0x3ebcab29 (stale) Mirror 12: 0x3ebcab29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: 0xa2a8bb30 (stale) Mirror 7: 0xa2a8bb30 (stale) Mirror 9: 0xf2bfe0ca (stale) Mirror 10: 0x3ebcab29 (stale) Mirror 11: 0x3ebcab29 (stale) Mirror 12: 0x3ebcab29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a310: skipped 000723 1766414266.671215 read 0x0071ad thru 0x007e23 (0xc77 bytes) 000724 1766414266.679816 mapread 0x02111c thru 0x030c5e (0xfb43 bytes) 000726 1766414266.717175 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a310): Mirror 6: 0xa2a8bb30 (stale) Mirror 7: 0xa2a8bb30 (stale) Mirror 9: 0xf2bfe0ca (stale) Mirror 10: 0x3ebcab29 (stale) Mirror 11: 0x3ebcab29 (stale) Mirror 12: 0x3ebcab29 (stale) CRC-32 checksum value for chunk [0, 0x3a310): Mirror 1: 0xdece63aa Mirror 13: 0xdece63aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a310: skipped 000727 1766414267.821035 mapwrite 0x033fed thru 0x035347 (0x135b bytes) 000729 1766414268.136768 trunc from 0x03a310 to 0x0390b0 (0x125f bytes) 000730 1766414268.185286 mapwrite 0x024407 thru 0x02c940 (0x853a bytes) 000731 1766414268.318323 write 0x000b9d thru 0x00277b (0x1bdf bytes) 000733 1766414268.342155 write 0x02c5fc thru 0x032d97 (0x679c bytes) 000734 1766414268.375894 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x390b1): Mirror 7: 0x6a085486 (stale) Mirror 9: 0xa7b56ca (stale) Mirror 10: 0xbe1719da (stale) Mirror 11: 0xbe1719da (stale) Mirror 12: 0xbe1719da (stale) Mirror 13: 0xe3e1fcd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x390b1): Mirror 7: 0x6a085486 (stale) Mirror 9: 0xa7b56ca (stale) Mirror 10: 0xbe1719da (stale) Mirror 11: 0xbe1719da (stale) Mirror 12: 0xbe1719da (stale) Mirror 13: 0xe3e1fcd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x390b1: skipped 000736 1766414269.033399 write 0x026e71 thru 0x02e8d3 (0x7a63 bytes) 000737 1766414269.047022 write 0x021239 thru 0x028b69 (0x7931 bytes) 000738 1766414269.058202 write 0x00f86a thru 0x0165d1 (0x6d68 bytes) 000739 1766414269.078400 trunc from 0x0390b1 to 0x00c273 (0x2ce3d bytes) 000740 1766414269.122104 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc274): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc274): Mirror 7: 0x8d669d0e (stale) Mirror 9: 0x8d669d0e (stale) Mirror 10: 0x8d669d0e (stale) Mirror 11: 0x8d669d0e (stale) Mirror 12: 0x8d669d0e (stale) Mirror 13: 0xc194d2d4 (stale) CRC-32 checksum value for chunk [0, 0xc274): Mirror 1: 0x50fd199f Mirror 14: 0x50fd199f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc274: skipped 000741 1766414270.203041 MIRROR_RESYNC 8 mirrors 000742 1766414271.609043 write 0x02591f thru 0x025c71 (0x353 bytes) 000744 1766414271.717927 write 0x009852 thru 0x0118e9 (0x8098 bytes) 000745 1766414271.753439 read 0x01d6e7 thru 0x022635 (0x4f4f bytes) 000746 1766414271.770753 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x25c72): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c72): Mirror 7: 0xe10d0e (stale) Mirror 9: 0xe10d0e (stale) Mirror 10: 0xe10d0e (stale) Mirror 11: 0xe10d0e (stale) Mirror 12: 0xe10d0e (stale) Mirror 13: 0xe10d0e (stale) Mirror 14: 0xe10d0e (stale) CRC-32 checksum value for chunk [0, 0x25c72): Mirror 1: 0x62f1921e Mirror 15: 0x62f1921e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c72: skipped 000749 1766414272.900720 read 0x024a3f thru 0x025c71 (0x1233 bytes) 000750 1766414272.919018 trunc from 0x025c72 to 0x02fdfe (0xa18d bytes) 000752 1766414273.040955 write 0x023e7f thru 0x02ce39 (0x8fbb bytes) 000753 1766414273.062904 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: 0xb37cf0a6 (stale) Mirror 9: 0xb37cf0a6 (stale) Mirror 10: 0xb37cf0a6 (stale) Mirror 11: 0xb37cf0a6 (stale) Mirror 12: 0xb37cf0a6 (stale) Mirror 13: 0xb37cf0a6 (stale) Mirror 14: 0xb37cf0a6 (stale) Mirror 15: 0xdf6a26b3 (stale) CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 1: 0x1d1d04d2 Mirror 16: 0x1d1d04d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdff: skipped 000756 1766414274.235844 mapread 0x00629d thru 0x00b239 (0x4f9d bytes) 000758 1766414274.287882 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: 0xb37cf0a6 (stale) Mirror 9: 0xb37cf0a6 (stale) Mirror 10: 0xb37cf0a6 (stale) Mirror 11: 0xb37cf0a6 (stale) Mirror 12: 0xb37cf0a6 (stale) Mirror 14: 0xb37cf0a6 (stale) Mirror 15: 0xdf6a26b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: 0xb37cf0a6 (stale) Mirror 9: 0xb37cf0a6 (stale) Mirror 10: 0xb37cf0a6 (stale) Mirror 11: 0xb37cf0a6 (stale) Mirror 12: 0xb37cf0a6 (stale) Mirror 14: 0xb37cf0a6 (stale) Mirror 15: 0xdf6a26b3 (stale) CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 1: 0x1d1d04d2 Mirror 16: 0x1d1d04d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdff: skipped 000759 1766414275.173344 read 0x0026d4 thru 0x00dc7b (0xb5a8 bytes) 000760 1766414275.232800 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 7: 0xb37cf0a6 (stale) Mirror 9: 0xb37cf0a6 (stale) Mirror 10: 0xb37cf0a6 (stale) Mirror 11: 0xb37cf0a6 (stale) Mirror 12: 0xb37cf0a6 (stale) Mirror 14: 0xb37cf0a6 (stale) Mirror 15: 0xdf6a26b3 (stale) CRC-32 checksum value for chunk [0, 0x2fdff): Mirror 1: 0x1d1d04d2 Mirror 16: 0x1d1d04d2 Mirror 17: 0x1d1d04d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdff: skipped 000761 1766414276.322661 write 0x039422 thru 0x03ffff (0x6bde bytes) 000762 1766414276.417092 mapwrite 0x0213b0 thru 0x02a444 (0x9095 bytes) 000763 1766414276.520942 trunc from 0x040000 to 0x02efa1 (0x1105e bytes) 000764 1766414276.546732 write 0x0136c9 thru 0x022f00 (0xf838 bytes) 000766 1766414276.565845 mapread 0x01aa1c thru 0x02256a (0x7b4f bytes) 000767 1766414276.566872 mapwrite 0x020c0e thru 0x02dc38 (0xd02b bytes) 000769 1766414276.651304 mapwrite 0x01df8b thru 0x02cb3d (0xebb3 bytes) 000770 1766414276.707887 read 0x01f75c thru 0x02ba3f (0xc2e4 bytes) 000771 1766414276.710119 mapread 0x0073d0 thru 0x00e434 (0x7065 bytes) 000773 1766414276.730763 mapwrite 0x0313b7 thru 0x03eba2 (0xd7ec bytes) 000774 1766414276.887295 mapwrite 0x0249ae thru 0x02694e (0x1fa1 bytes) 000775 1766414276.942914 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eba3): Mirror 7: 0xe5e0c07c (stale) Mirror 9: 0xe5e0c07c (stale) Mirror 10: 0xe5e0c07c (stale) Mirror 11: 0xe5e0c07c (stale) Mirror 14: 0xe5e0c07c (stale) Mirror 15: 0x2a067d0f (stale) Mirror 16: 0x6398b7d3 (stale) Mirror 17: 0x6398b7d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eba3): Mirror 7: 0xe5e0c07c (stale) Mirror 9: 0xe5e0c07c (stale) Mirror 10: 0xe5e0c07c (stale) Mirror 11: 0xe5e0c07c (stale) Mirror 14: 0xe5e0c07c (stale) Mirror 15: 0x2a067d0f (stale) Mirror 16: 0x6398b7d3 (stale) Mirror 17: 0x6398b7d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eba3: skipped 000776 1766414277.544307 write 0x009e75 thru 0x00f63f (0x57cb bytes) 000777 1766414277.560041 read 0x020972 thru 0x027c6c (0x72fb bytes) 000778 1766414277.567998 write 0x02dc87 thru 0x0329ce (0x4d48 bytes) 000780 1766414277.579919 write 0x005381 thru 0x0058be (0x53e bytes) 000782 1766414277.593333 write 0x00fbcb thru 0x0179ae (0x7de4 bytes) 000786 1766414277.617053 write 0x03ac95 thru 0x03ffff (0x536b bytes) 000790 1766414277.656388 mapread 0x025705 thru 0x030425 (0xad21 bytes) 000792 1766414277.658228 mapread 0x0067bd thru 0x01557b (0xedbf bytes) 000793 1766414277.659255 mapread 0x0157f3 thru 0x018038 (0x2846 bytes) 000794 1766414277.671001 read 0x014141 thru 0x016341 (0x2201 bytes) 000796 1766414277.674554 mapread 0x0288b4 thru 0x0323f3 (0x9b40 bytes) 000797 1766414277.678920 read 0x00f2d0 thru 0x01d6f2 (0xe423 bytes) 000798 1766414277.682146 read 0x0249e7 thru 0x02faa2 (0xb0bc bytes) 000800 1766414277.690367 write 0x00b50c thru 0x01486f (0x9364 bytes) 000801 1766414277.700674 MIRROR_RESYNC 9 mirrors 000802 1766414279.363084 read 0x039264 thru 0x03eddc (0x5b79 bytes) 000806 1766414279.378567 read 0x0158cc thru 0x021546 (0xbc7b bytes) 000807 1766414279.389519 mapwrite 0x01d5be thru 0x028e3d (0xb880 bytes) 000809 1766414279.614567 mapwrite 0x014257 thru 0x01cfc6 (0x8d70 bytes) 000810 1766414279.697403 mapread 0x03b79d thru 0x03d16e (0x19d2 bytes) 000814 1766414279.733967 read 0x03c7a4 thru 0x03ffff (0x385c bytes) 000815 1766414279.744747 mapread 0x00d814 thru 0x019968 (0xc155 bytes) 000817 1766414279.746732 read 0x021b67 thru 0x0245d2 (0x2a6c bytes) 000818 1766414279.747481 trunc from 0x040000 to 0x02b2ad (0x14d52 bytes) 000819 1766414279.779900 trunc from 0x02b2ae to 0x02ee70 (0x3bc3 bytes) 000821 1766414279.806495 mapwrite 0x0220d5 thru 0x0227c3 (0x6ef bytes) 000824 1766414279.870844 write 0x011e60 thru 0x0122d2 (0x473 bytes) 000825 1766414279.872532 mapwrite 0x010bb8 thru 0x01e95a (0xdda3 bytes) 000826 1766414279.923177 read 0x0253e8 thru 0x02ee70 (0x9a89 bytes) 000828 1766414279.930128 mapread 0x0090cd thru 0x00af74 (0x1ea8 bytes) 000829 1766414279.936816 MIRROR_RESYNC 9 mirrors 000830 1766414281.360598 trunc from 0x02ee71 to 0x00861a (0x26856 bytes) 000832 1766414281.468675 write 0x0368a2 thru 0x038312 (0x1a71 bytes) 000836 1766414281.472046 mapread 0x0291f1 thru 0x038312 (0xf122 bytes) 000837 1766414281.510514 mapwrite 0x02e3ff thru 0x03201d (0x3c1f bytes) 000839 1766414281.612861 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38313): Mirror 9: 0x574bca91 (stale) Mirror 10: 0x574bca91 (stale) Mirror 11: 0x574bca91 (stale) Mirror 14: 0x574bca91 (stale) Mirror 15: 0x574bca91 (stale) Mirror 16: 0x574bca91 (stale) Mirror 17: 0x574bca91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38313): Mirror 9: 0x574bca91 (stale) Mirror 10: 0x574bca91 (stale) Mirror 11: 0x574bca91 (stale) Mirror 14: 0x574bca91 (stale) Mirror 15: 0x574bca91 (stale) Mirror 16: 0x574bca91 (stale) Mirror 17: 0x574bca91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38313: skipped 000840 1766414282.268871 write 0x034985 thru 0x03b6f4 (0x6d70 bytes) 000843 1766414282.284688 read 0x039310 thru 0x03b6f4 (0x23e5 bytes) 000844 1766414282.288055 write 0x00e7d2 thru 0x015baa (0x73d9 bytes) 000847 1766414282.317542 mapwrite 0x009991 thru 0x015fdb (0xc64b bytes) 000848 1766414282.382372 mapread 0x004847 thru 0x00cf12 (0x86cc bytes) 000850 1766414282.384684 mapread 0x039755 thru 0x03b6f4 (0x1fa0 bytes) 000851 1766414282.386552 mapwrite 0x02be6c thru 0x030543 (0x46d8 bytes) 000853 1766414282.475300 mapwrite 0x00af87 thru 0x01460c (0x9686 bytes) 000854 1766414282.529667 read 0x030fd6 thru 0x038bf1 (0x7c1c bytes) 000855 1766414282.531554 read 0x0065cc thru 0x00961d (0x3052 bytes) 000857 1766414282.533349 trunc from 0x03b6f5 to 0x019f66 (0x2178e bytes) 000858 1766414282.566014 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19f67): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f67): Mirror 9: 0xfb18e285 (stale) Mirror 10: 0xfb18e285 (stale) Mirror 11: 0xfb18e285 (stale) Mirror 14: 0xfb18e285 (stale) Mirror 15: 0xfb18e285 (stale) Mirror 16: 0xfb18e285 (stale) Mirror 17: 0xfb18e285 (stale) CRC-32 checksum value for chunk [0, 0x19f67): Mirror 1: 0xcf3c590b Mirror 18: 0xcf3c590b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f67: skipped 000859 1766414283.689988 mapread 0x00d689 thru 0x017879 (0xa1f1 bytes) 000860 1766414283.745375 read 0x003941 thru 0x00d150 (0x9810 bytes) 000862 1766414283.776207 mapread 0x0032cb thru 0x00e1c9 (0xaeff bytes) 000864 1766414283.811559 trunc from 0x019f67 to 0x03bf2f (0x21fc9 bytes) 000865 1766414283.909929 MIRROR_RESYNC 9 mirrors 000868 1766414285.924595 trunc from 0x03bf30 to 0x03a184 (0x1dab bytes) 000869 1766414286.010976 write 0x01b391 thru 0x0209ae (0x561e bytes) 000871 1766414286.027016 write 0x028050 thru 0x037c9f (0xfc50 bytes) 000874 1766414286.063959 read 0x02c470 thru 0x03a184 (0xdd15 bytes) 000875 1766414286.075710 mapwrite 0x01dd7f thru 0x02d6eb (0xf96d bytes) 000876 1766414286.154205 read 0x02c23c thru 0x02df66 (0x1d2b bytes) 000879 1766414286.155693 write 0x02feb1 thru 0x038427 (0x8577 bytes) 000880 1766414286.156734 trunc from 0x03a185 to 0x02034e (0x19e36 bytes) 000882 1766414286.195670 trunc from 0x02034f to 0x02383e (0x34f0 bytes) 000885 1766414286.214330 write 0x00b423 thru 0x00ba4a (0x628 bytes) 000886 1766414286.227733 read 0x0165d2 thru 0x01a552 (0x3f81 bytes) 000893 1766414286.234006 read 0x00c23d thru 0x016099 (0x9e5d bytes) 000894 1766414286.234631 write 0x039679 thru 0x03e509 (0x4e91 bytes) 000896 1766414286.235994 read 0x01634e thru 0x022c28 (0xc8db bytes) 000897 1766414286.243656 write 0x005bdb thru 0x0115c0 (0xb9e6 bytes) 000898 1766414286.245262 mapread 0x03d06a thru 0x03e509 (0x14a0 bytes) 000900 1766414286.254846 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e50a): Mirror 9: 0xd1b6a921 (stale) Mirror 10: 0xd1b6a921 (stale) Mirror 11: 0xd1b6a921 (stale) Mirror 14: 0xd1b6a921 (stale) Mirror 15: 0xd1b6a921 (stale) Mirror 16: 0xd1b6a921 (stale) Mirror 17: 0xd1b6a921 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e50a): Mirror 9: 0xd1b6a921 (stale) Mirror 10: 0xd1b6a921 (stale) Mirror 11: 0xd1b6a921 (stale) Mirror 14: 0xd1b6a921 (stale) Mirror 15: 0xd1b6a921 (stale) Mirror 16: 0xd1b6a921 (stale) Mirror 17: 0xd1b6a921 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e50a: skipped 000901 1766414286.892296 mapwrite 0x03be14 thru 0x03ffff (0x41ec bytes) 000902 1766414286.991394 trunc from 0x040000 to 0x027a71 (0x1858e bytes) 000906 1766414287.023799 read 0x018418 thru 0x02498f (0xc578 bytes) 000908 1766414287.032203 trunc from 0x027a72 to 0x020acc (0x6fa5 bytes) 000909 1766414287.051949 mapread 0x000497 thru 0x00e214 (0xdd7e bytes) 000912 1766414287.064833 mapwrite 0x03ed1e thru 0x03ffff (0x12e2 bytes) 000915 1766414287.151363 mapwrite 0x018b59 thru 0x026a9d (0xdf45 bytes) 000920 1766414287.236635 mapwrite 0x024286 thru 0x028782 (0x44fd bytes) 000921 1766414287.310921 write 0x003bda thru 0x010608 (0xca2f bytes) 000922 1766414287.320364 read 0x00f632 thru 0x019013 (0x99e2 bytes) 000925 1766414287.336826 mapwrite 0x003f5a thru 0x006550 (0x25f7 bytes) 000927 1766414287.431809 trunc from 0x040000 to 0x032ac0 (0xd53f bytes) 000931 1766414287.467927 write 0x000ab5 thru 0x00a050 (0x959c bytes) 000932 1766414287.470302 read 0x02edc4 thru 0x03011b (0x1358 bytes) 000933 1766414287.471073 trunc from 0x032ac1 to 0x02ad45 (0x7d7b bytes) 000934 1766414287.512717 mapread 0x01e14b thru 0x02473e (0x65f4 bytes) 000939 1766414287.521692 trunc from 0x02ad46 to 0x0219b2 (0x9393 bytes) 000940 1766414287.547976 mapwrite 0x02672f thru 0x031e00 (0xb6d2 bytes) 000941 1766414287.666622 write 0x0265ef thru 0x02b75a (0x516c bytes) 000942 1766414287.667641 read 0x028523 thru 0x02df59 (0x5a37 bytes) 000944 1766414287.675627 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x31e01): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e01): Mirror 9: 0xf7885558 (stale) Mirror 10: 0xf7885558 (stale) Mirror 11: 0xf7885558 (stale) Mirror 14: 0xf7885558 (stale) Mirror 15: 0xf7885558 (stale) Mirror 16: 0xf7885558 (stale) Mirror 17: 0xf7885558 (stale) CRC-32 checksum value for chunk [0, 0x31e01): Mirror 1: 0xd926bc84 Mirror 18: 0xd926bc84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e01: skipped 000945 1766414288.680307 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e01): Mirror 9: 0xf7885558 (stale) Mirror 10: 0xf7885558 (stale) Mirror 11: 0xf7885558 (stale) Mirror 15: 0xf7885558 (stale) Mirror 16: 0xf7885558 (stale) Mirror 17: 0xf7885558 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e01): Mirror 9: 0xf7885558 (stale) Mirror 10: 0xf7885558 (stale) Mirror 11: 0xf7885558 (stale) Mirror 15: 0xf7885558 (stale) Mirror 16: 0xf7885558 (stale) Mirror 17: 0xf7885558 (stale) CRC-32 checksum value for chunk [0, 0x31e01): Mirror 1: 0xd926bc84 Mirror 18: 0xd926bc84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e01: skipped 000947 1766414289.518737 MIRROR_RESYNC 8 mirrors 000949 1766414290.842240 mapread 0x027a30 thru 0x0291ec (0x17bd bytes) 000950 1766414290.870120 read 0x01f8e8 thru 0x027376 (0x7a8f bytes) 000951 1766414290.877535 write 0x005312 thru 0x00d8ef (0x85de bytes) 000952 1766414290.982141 trunc from 0x031e01 to 0x0165a9 (0x1b857 bytes) 000953 1766414291.001596 trunc from 0x0165aa to 0x03e920 (0x28377 bytes) 000954 1766414291.026429 mapwrite 0x030c87 thru 0x03aba3 (0x9f1d bytes) 000955 1766414291.163147 trunc from 0x03e921 to 0x0297e2 (0x1513e bytes) 000957 1766414291.203486 mapread 0x018908 thru 0x0237d0 (0xaec9 bytes) 000958 1766414291.243096 mapread 0x0292fd thru 0x0297e2 (0x4e6 bytes) 000960 1766414291.267416 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x297e3): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297e3): Mirror 9: 0x34dcf593 (stale) Mirror 10: 0x34dcf593 (stale) Mirror 11: 0x34dcf593 (stale) Mirror 15: 0x34dcf593 (stale) Mirror 16: 0x34dcf593 (stale) Mirror 17: 0x34dcf593 (stale) Mirror 18: 0x34dcf593 (stale) CRC-32 checksum value for chunk [0, 0x297e3): Mirror 1: 0x75d1ccc9 Mirror 19: 0x75d1ccc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297e3: skipped 000962 1766414292.384713 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x297e3): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297e3): Mirror 9: 0x34dcf593 (stale) Mirror 10: 0x34dcf593 (stale) Mirror 11: 0x34dcf593 (stale) Mirror 15: 0x34dcf593 (stale) Mirror 16: 0x34dcf593 (stale) Mirror 17: 0x34dcf593 (stale) Mirror 18: 0x34dcf593 (stale) CRC-32 checksum value for chunk [0, 0x297e3): Mirror 1: 0x75d1ccc9 Mirror 19: 0x75d1ccc9 Mirror 20: 0x75d1ccc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297e3: skipped 000963 1766414293.720342 mapread 0x018bdf thru 0x01f714 (0x6b36 bytes) 000964 1766414293.782385 read 0x01c917 thru 0x023c8a (0x7374 bytes) 000967 1766414293.853420 read 0x00f334 thru 0x015b99 (0x6866 bytes) 000968 1766414293.886704 read 0x001a10 thru 0x007f3f (0x6530 bytes) 000971 1766414293.933962 read 0x006b35 thru 0x00864a (0x1b16 bytes) 000972 1766414293.954823 mapread 0x027a9e thru 0x0297e2 (0x1d45 bytes) 000973 1766414294.008256 MIRROR_RESYNC 10 mirrors 000974 1766414295.622976 read 0x00247b thru 0x00d0a7 (0xac2d bytes) 000977 1766414295.638753 MIRROR_EXTEND to 11 mirrors 000978 1766414296.501064 mapwrite 0x0157f2 thru 0x01ea45 (0x9254 bytes) 000979 1766414296.771671 trunc from 0x0297e3 to 0x03ca60 (0x1327e bytes) 000982 1766414296.824926 mapwrite 0x017f21 thru 0x01ad75 (0x2e55 bytes) 000983 1766414296.894803 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ca61): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca61): Mirror 9: 0xbe1af24a (stale) Mirror 10: 0xbe1af24a (stale) Mirror 11: 0xbe1af24a (stale) Mirror 15: 0xbe1af24a (stale) Mirror 16: 0xbe1af24a (stale) Mirror 17: 0xbe1af24a (stale) Mirror 18: 0xbe1af24a (stale) Mirror 19: 0xbe1af24a (stale) Mirror 20: 0xbe1af24a (stale) Mirror 21: 0xbe1af24a (stale) CRC-32 checksum value for chunk [0, 0x3ca61): Mirror 1: 0xbd636a13 Mirror 22: 0xbd636a13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca61: skipped 000985 1766414298.110960 write 0x000f31 thru 0x00fafa (0xebca bytes) 000987 1766414298.220726 write 0x0019d8 thru 0x002353 (0x97c bytes) 000988 1766414298.221534 read 0x01b3e6 thru 0x029027 (0xdc42 bytes) 000989 1766414298.229424 read 0x005746 thru 0x01056d (0xae28 bytes) 000993 1766414298.242801 mapwrite 0x00367a thru 0x005025 (0x19ac bytes) 000996 1766414298.289565 write 0x0009d5 thru 0x009383 (0x89af bytes) 000997 1766414298.292138 mapread 0x000dfe thru 0x004c43 (0x3e46 bytes) 000998 1766414298.293940 mapread 0x00021a thru 0x009cce (0x9ab5 bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 09:38:23 (1766414303) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:38:26 (1766414306) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271452 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168214 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:38:34 (1766414314) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4971552 .. Extending file size to 1119616 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1240704 .. Extending file size to 7310880 .. failed Extending file size to 7642336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7682784 .. Extending file size to 8515872 .. Extending file size to 8062528 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8237888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8822144 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9350048 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9187840 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55452 55454 55455 55457 55458 done 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:39:55 (1766414395) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 562880 .. Extending file size to 1707296 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4750848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3860832 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8610528 .. Extending file size to 5812288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8430624 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9082240 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9191040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57345 57346 57348 57349 57350 done 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:41:12 (1766414472) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0695056 s, 30.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.41979 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x60c94e2b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:41:30 (1766414490) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0687934 s, 30.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:41:37 (1766414497) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:41:44 (1766414504) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250442 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x282:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:41:52 (1766414512) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287012 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:42:00 (1766414520) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22977 s, 9.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:42:35 (1766414555) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931903 s, 11.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:43:36 (1766414616) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.321446 s, 6.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:44:37 (1766414677) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. Creating new pool test_204f oleg205-server: Pool lustre.test_204f created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224499 s, 9.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg205-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:45:55 (1766414755) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:46:02 (1766414762) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:46:12 (1766414772) oleg205-server: Pool lustre.test_206 created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg205-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:46:36 (1766414796) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576346 s, 18.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198861 s, 52.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe0a36d9a Mirror 3: 0xe0a36d9a PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:46:44 (1766414804) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:46:47 (1766414807) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:46:51 (1766414811) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0519555 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1680000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1680000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1680000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a10000 Read this (4 bytes): BBBB 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 209a (30s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:47:20 (1766414840) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0405704 s, 202 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:47:27 (1766414847) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517588 s, 20.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x223 lmm_fid: [0x200000404:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834467 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 09:47:28.000000000 -0500 Modify: 2025-12-22 09:47:28.000000000 -0500 Change: 2025-12-22 09:47:29.000000000 -0500 Birth: 2025-12-22 09:47:29.000000000 -0500 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:47:34 (1766414854) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:47:37 (1766414857) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.521105 s, 78.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0432647 s, 94.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 22 09:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x225:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 22 09:47 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:47:44 (1766414864) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:47:49 (1766414869) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1587 sec =========== 09:47:58 (1766414878) === sanity-flr: start cleanup 09:47:59 (1766414879) === === sanity-flr: finish cleanup 09:48:05 (1766414885) ===