-----============= acceptance-small: sanity-flr ============----- Mon Mar 16 09:36:25 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg334-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:36:31 (1773668191) === oleg334-client.virtnet: executing check_config_client /mnt/lustre oleg334-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg334-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff897fd1d65800.idle_timeout=debug osc.lustre-OST0001-osc-ffff897fd1d65800.idle_timeout=debug disable quota as required oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:36:40 (1773668200) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7559] == sanity-flr test 0a: lfs mirror create with -N option == 09:36:41 (1773668201) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:36:45 (1773668205) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:36:46 (1773668206) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:36:47 (1773668207) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:36:53 (1773668213) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:36:54 (1773668214) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:36:55 (1773668215) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff897fd1d65800.stats=clear osc.lustre-OST0001-osc-ffff897fd1d65800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:36:57 (1773668217) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:37:00 (1773668220) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:37:02 (1773668222) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:37:03 (1773668223) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 09:37:06 (1773668226) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:37:08 (1773668228) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:37:09 (1773668229) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:37:11 (1773668231) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:37:14 (1773668234) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:37:16 (1773668236) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0629279 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252739 s, 41.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:37:18 (1773668238) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.126562 s, 124 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:37:20 (1773668240) Stopping /mnt/lustre-ost1 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg334-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg334-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:37:47 (1773668267) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.118261 s, 53.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg334-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:38:01 (1773668281) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg334-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg334-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:38:26 (1773668306) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15691 s, 45.3 MB/s -rw-r--r-- 1 root root 52428800 Mar 16 09:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.267s user 0m0.000s sys 0m0.141s Stopping /mnt/lustre-ost1 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.268s user 0m0.000s sys 0m0.141s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg334-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg334-server oleg334-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.287s user 0m0.000s sys 0m0.146s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg334-server: oleg334-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg334-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff897fd1d65800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:38:53 (1773668333) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:38:54 (1773668334) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:38:55 (1773668335) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:38:56 (1773668336) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 09:38:58 (1773668338) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.152098 s, 82.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:39:02 (1773668342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110787 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119417 s, 87.8 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:39:04 (1773668344) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:39:07 (1773668347) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000708687 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000690883 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00046213 s, 2.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00054212 s, 1.9 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 09:39:10 (1773668350) create test files with size 15760405 .. 6384b11cd955d1706c2dd8e732d055dc - 331586c073d04e2d9b8a310dfc91b447 - 12cdd8fe34c6a160fd92ca4b524ca601 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg334-client.virtnet /mnt/lustre (opts:) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 09:39:16 (1773668356) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:39:20 (1773668360) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:39:22 (1773668362) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545381 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116881 s, 89.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 09:39:25 (1773668365) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0367733 s, 57.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.113423 s, 37.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0511096 s, 41.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 09:39:30 (1773668370) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 09:39:31 (1773668371) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:39:32 (1773668372) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912076 s, 115 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 09:39:35 (1773668375) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:39:36 (1773668376) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:39:40 (1773668380) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.103642 s, 101 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg334-client.virtnet /mnt/lustre (opts:) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:39:58 (1773668398) llite.lustre-ffff897fc3d97000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0831195 s, 126 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg334-client.virtnet /mnt/lustre (opts:) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff897fc7eb0800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:40:02 (1773668402) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:40:15 (1773668415) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:40:27 (1773668427) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:40:28 (1773668428) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0697463 s, 45.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.055464 s, 56.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe237f486 (stale) Mirror 2: 0xe237f486 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2ab6770d7e5974d7fdcca1ed58fb82fd - mirror 1 checksum: a2b668192c38a8d5f37b7c701afb8792 - mirror 2 checksum: 2ab6770d7e5974d7fdcca1ed58fb82fd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 09:40:28 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 09:40:28 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2ab6770d7e5974d7fdcca1ed58fb82fd - mirror 1 checksum: a2b668192c38a8d5f37b7c701afb8792 - mirror 2 checksum: a2b668192c38a8d5f37b7c701afb8792 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 09:40:28 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 09:40:28 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:40:32 (1773668432) Failing mds1 on oleg334-server Stopping /mnt/lustre-mds1 (opts:) on oleg334-server 09:40:34 (1773668434) shut down facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server Failover mds1 to oleg334-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 09:40:48 (1773668448) targets are mounted 09:40:48 (1773668448) facet_failover done oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207094 s, 50.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (23s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:40:55 (1773668455) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:40:56 (1773668456) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:40:57 (1773668457) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00243368 s, 6.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:40:59 (1773668459) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:41:00 (1773668460) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:41:02 (1773668462) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:41:07 (1773668467) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:41:26 (1773668486) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:41:44 (1773668504) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:42:02 (1773668522) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:42:04 (1773668524) mirror create pid 52268 mirror split pid 52270 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52268 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52270 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:43:10 (1773668590) Seed set to 8591 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1773668591.839351 mapwrite 0x037e3f thru 0x0392e0 (0x14a2 bytes) 000007 1773668591.986887 mapread 0x03194e thru 0x0392e0 (0x7993 bytes) 000008 1773668591.996200 mapread 0x037600 thru 0x0382f7 (0xcf8 bytes) 000011 1773668591.998747 read 0x0201cb thru 0x028f69 (0x8d9f bytes) 000012 1773668592.008824 write 0x00ee68 thru 0x011a26 (0x2bbf bytes) truncating to largest ever: 0x1faf8 000015 1773668592.022214 trunc from 0x0392e1 to 0x01faf7 (0x197e9 bytes) 000017 1773668592.042083 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x338ed 000021 1773668592.490431 trunc from 0x01faf8 to 0x0338ec (0x13df5 bytes) 000024 1773668592.579438 read 0x0051af thru 0x009eb6 (0x4d08 bytes) 000026 1773668592.587542 mapwrite 0x0128b2 thru 0x0146bc (0x1e0b bytes) 000027 1773668592.643928 mapread 0x00ac8f thru 0x013aba (0x8e2c bytes) 000028 1773668592.655069 MIRROR_RESYNC 2 mirrors 000029 1773668593.124415 MIRROR_EXTEND to 3 mirrors 000030 1773668593.676007 write 0x00eb82 thru 0x0145de (0x5a5d bytes) 000031 1773668593.784683 trunc from 0x0338ed to 0x02e631 (0x52bb bytes) 000033 1773668593.928210 write 0x00b106 thru 0x017a9f (0xc99a bytes) 000034 1773668593.963929 mapwrite 0x0330f6 thru 0x03ffff (0xcf0a bytes) 000035 1773668594.114763 mapwrite 0x037b05 thru 0x03ffff (0x84fb bytes) 000036 1773668594.168768 mapwrite 0x00f5fc thru 0x0196cd (0xa0d2 bytes) 000037 1773668594.364505 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8714e6e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8714e6e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000038 1773668594.563443 trunc from 0x040000 to 0x02b210 (0x14def bytes) 000039 1773668594.579854 mapwrite 0x02cb3b thru 0x03b8f4 (0xedba bytes) 000040 1773668594.653328 read 0x016570 thru 0x01e451 (0x7ee2 bytes) 000043 1773668594.667819 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) 000045 1773668594.770352 mapread 0x0214ee thru 0x02920c (0x7d1f bytes) 000046 1773668594.793104 MIRROR_EXTEND to 2 mirrors 000047 1773668596.233675 mapwrite 0x00df58 thru 0x013292 (0x533b bytes) 000050 1773668596.637582 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 2: 0xd1e4d20a (stale) CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 1: 0x67f0ff52 Mirror 3: 0x67f0ff52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8f5: skipped 000052 1773668598.324837 trunc from 0x03b8f5 to 0x01641c (0x254d8 bytes) 000053 1773668598.558260 mapread 0x002620 thru 0x011d84 (0xf765 bytes) 000054 1773668598.657552 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1641d): Mirror 2: 0xc03d30b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1641d): Mirror 2: 0xc03d30b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1641d: skipped 000055 1773668599.418203 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) 000056 1773668599.710299 trunc from 0x01641d to 0x00769f (0xed7d bytes) 000058 1773668599.768216 read 0x004d0e thru 0x00769f (0x2992 bytes) 000059 1773668599.776404 write 0x02485f thru 0x02a637 (0x5dd9 bytes) 000063 1773668599.782564 mapwrite 0x035067 thru 0x03e3ed (0x9387 bytes) 000065 1773668599.891402 trunc from 0x03e3ee to 0x019383 (0x2506a bytes) 000066 1773668599.906407 read 0x0148ab thru 0x0183a9 (0x3aff bytes) 000068 1773668599.920673 mapwrite 0x000734 thru 0x00a310 (0x9bdd bytes) 000069 1773668599.975134 mapread 0x003836 thru 0x009d74 (0x653f bytes) 000070 1773668599.976280 write 0x00a7f3 thru 0x013845 (0x9053 bytes) 000072 1773668599.983410 MIRROR_EXTEND to 2 mirrors 000073 1773668601.228462 mapread 0x005094 thru 0x00fa43 (0xa9b0 bytes) 000074 1773668601.281941 write 0x005c08 thru 0x00fac9 (0x9ec2 bytes) 000075 1773668601.485262 write 0x036a4d thru 0x03ffff (0x95b3 bytes) 000076 1773668601.495526 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) 000077 1773668601.823841 read 0x031223 thru 0x03de91 (0xcc6f bytes) 000078 1773668601.872536 mapwrite 0x01aec8 thru 0x022adf (0x7c18 bytes) 000079 1773668602.050358 mapread 0x0299da thru 0x039464 (0xfa8b bytes) 000083 1773668602.095949 mapread 0x030a11 thru 0x031bc9 (0x11b9 bytes) 000084 1773668602.099681 mapwrite 0x027178 thru 0x028fa4 (0x1e2d bytes) 000086 1773668602.237472 read 0x0063cd thru 0x01456d (0xe1a1 bytes) 000087 1773668602.265077 MIRROR_EXTEND to 2 mirrors 000088 1773668603.103734 trunc from 0x040000 to 0x00c738 (0x338c7 bytes) 000091 1773668603.283974 write 0x039c9e thru 0x039ca0 (0x3 bytes) 000092 1773668603.285260 mapread 0x023d17 thru 0x02ea76 (0xad60 bytes) 000094 1773668603.317985 write 0x03a310 thru 0x03ffff (0x5cf0 bytes) 000095 1773668603.319278 mapread 0x02dd8d thru 0x035026 (0x729a bytes) 000096 1773668603.320742 read 0x023315 thru 0x0326a7 (0xf393 bytes) 000097 1773668603.321718 trunc from 0x040000 to 0x013e8e (0x2c171 bytes) 000099 1773668603.353084 MIRROR_RESYNC 2 mirrors 000100 1773668604.008355 trunc from 0x013e8f to 0x02cee4 (0x19056 bytes) 000102 1773668604.127446 mapread 0x0162b9 thru 0x02235a (0xc0a2 bytes) 000103 1773668604.188722 mapread 0x002d18 thru 0x003744 (0xa2d bytes) 000106 1773668604.213283 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: 0x4010caf7 (stale) CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 1: 0x4010caf7 Mirror 3: 0x4010caf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cee5: skipped 000107 1773668605.469525 write 0x02475d thru 0x0272f9 (0x2b9d bytes) 000110 1773668605.661790 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: 0x4010caf7 (stale) Mirror 3: 0x4010caf7 (stale) CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 1: 0xf70c4c17 Mirror 4: 0xf70c4c17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cee5: skipped 000111 1773668607.382067 mapread 0x007322 thru 0x00a52a (0x3209 bytes) 000112 1773668607.427758 mapread 0x026840 thru 0x028792 (0x1f53 bytes) 000114 1773668607.519760 read 0x00512e thru 0x010bef (0xbac2 bytes) 000118 1773668607.598733 mapread 0x017575 thru 0x022061 (0xaaed bytes) 000119 1773668607.664336 read 0x004736 thru 0x0066c9 (0x1f94 bytes) 000121 1773668607.703710 read 0x000e81 thru 0x00a1f9 (0x9379 bytes) 000122 1773668607.715414 trunc from 0x02cee5 to 0x00e58f (0x1e955 bytes) 000123 1773668607.803030 MIRROR_RESYNC 4 mirrors 000124 1773668609.436236 mapwrite 0x017461 thru 0x01a1e5 (0x2d85 bytes) 000125 1773668609.888148 trunc from 0x01a1e6 to 0x01bc3f (0x1a5a bytes) 000126 1773668609.936174 mapread 0x0177fc thru 0x019d13 (0x2518 bytes) 000127 1773668609.937327 mapwrite 0x00f1e1 thru 0x01effd (0xfe1d bytes) 000128 1773668610.247807 read 0x006714 thru 0x00a67a (0x3f67 bytes) 000129 1773668610.263035 read 0x01bfff thru 0x01db88 (0x1b8a bytes) 000130 1773668610.269592 mapread 0x00b5eb thru 0x010ff3 (0x5a09 bytes) 000132 1773668610.277501 mapwrite 0x0295f0 thru 0x031a70 (0x8481 bytes) 000133 1773668610.568518 read 0x0023b7 thru 0x010c37 (0xe881 bytes) 000135 1773668610.591856 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31a71): Mirror 3: 0xf9767a37 (stale) Mirror 4: 0xf9767a37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a71): Mirror 3: 0xf9767a37 (stale) Mirror 4: 0xf9767a37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a71: skipped 000136 1773668611.211383 write 0x012cf9 thru 0x0198fa (0x6c02 bytes) 000140 1773668611.235144 write 0x022801 thru 0x030fc0 (0xe7c0 bytes) 000141 1773668611.287508 write 0x02b614 thru 0x036610 (0xaffd bytes) 000142 1773668611.324907 MIRROR_RESYNC 3 mirrors 000145 1773668612.598660 write 0x038daf thru 0x03ffff (0x7251 bytes) 000147 1773668612.758808 read 0x032947 thru 0x036626 (0x3ce0 bytes) 000148 1773668612.775514 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8451fa66 (stale) Mirror 4: 0x8451fa66 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4013bd11 Mirror 5: 0x4013bd11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000149 1773668614.180226 write 0x0309d1 thru 0x032e5b (0x248b bytes) 000150 1773668614.344816 read 0x03df1d thru 0x03ffff (0x20e3 bytes) 000151 1773668614.368566 mapread 0x00e6c8 thru 0x01d88f (0xf1c8 bytes) 000152 1773668614.419718 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8451fa66 (stale) Mirror 5: 0x4013bd11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8451fa66 (stale) Mirror 5: 0x4013bd11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000153 1773668614.930479 write 0x0372be thru 0x0386ac (0x13ef bytes) 000154 1773668614.954447 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8451fa66 (stale) Mirror 5: 0x4013bd11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40f3e6fe Mirror 6: 0x40f3e6fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000157 1773668616.520048 trunc from 0x040000 to 0x01f5cf (0x20a30 bytes) 000158 1773668616.663977 read 0x00f2bc thru 0x01728b (0x7fd0 bytes) 000159 1773668616.677869 read 0x002214 thru 0x0079ea (0x57d7 bytes) 000161 1773668616.693569 trunc from 0x01f5d0 to 0x030500 (0x10f31 bytes) 000162 1773668616.744267 MIRROR_RESYNC 4 mirrors 000164 1773668618.430745 read 0x00adda thru 0x0171d0 (0xc3f7 bytes) 000165 1773668618.460898 read 0x002b52 thru 0x00d22c (0xa6db bytes) 000168 1773668618.482364 mapread 0x00d11a thru 0x015638 (0x851f bytes) 000171 1773668618.531212 trunc from 0x030501 to 0x00715c (0x293a4 bytes) 000172 1773668618.766794 write 0x032290 thru 0x0359bf (0x3730 bytes) 000174 1773668618.776581 read 0x010ae4 thru 0x01ed86 (0xe2a3 bytes) 000175 1773668618.804553 mapwrite 0x004076 thru 0x00876d (0x46f8 bytes) 000177 1773668618.980980 read 0x025c83 thru 0x031bbc (0xbf3a bytes) 000180 1773668618.995876 trunc from 0x0359c0 to 0x001f36 (0x33a89 bytes) 000181 1773668619.071025 mapwrite 0x0171e1 thru 0x027078 (0xfe98 bytes) 000182 1773668619.357327 mapread 0x02317e thru 0x027078 (0x3efb bytes) 000184 1773668619.371048 MIRROR_RESYNC 4 mirrors 000185 1773668621.032338 read 0x01e19f thru 0x02222c (0x408e bytes) 000189 1773668621.056247 read 0x023890 thru 0x027078 (0x37e9 bytes) 000190 1773668621.071369 read 0x019fdb thru 0x01e616 (0x463c bytes) 000192 1773668621.072162 mapread 0x027013 thru 0x027078 (0x66 bytes) 000193 1773668621.072843 trunc from 0x027079 to 0x025c14 (0x1464 bytes) 000194 1773668621.246077 mapwrite 0x037513 thru 0x03b430 (0x3f1e bytes) 000195 1773668621.444346 write 0x037389 thru 0x03ffff (0x8c77 bytes) 000197 1773668621.446401 read 0x0088a8 thru 0x01676d (0xdec6 bytes) 000200 1773668621.457745 write 0x02f022 thru 0x034317 (0x52f6 bytes) 000201 1773668621.459118 trunc from 0x040000 to 0x011bac (0x2e453 bytes) 000204 1773668621.501299 mapwrite 0x02290e thru 0x03202b (0xf71e bytes) 000205 1773668621.655994 mapwrite 0x031128 thru 0x03fbf5 (0xeace bytes) 000207 1773668621.810261 mapwrite 0x032c26 thru 0x03ffff (0xd3da bytes) 000208 1773668621.982896 mapread 0x020c33 thru 0x020ec4 (0x292 bytes) 000209 1773668621.998764 mapread 0x01d481 thru 0x02c51a (0xf09a bytes) 000211 1773668622.012015 mapread 0x026468 thru 0x02e467 (0x8000 bytes) 000212 1773668622.013247 write 0x01a227 thru 0x023305 (0x90df bytes) 000214 1773668622.036042 mapwrite 0x037408 thru 0x03ffff (0x8bf8 bytes) 000215 1773668622.198165 write 0x0342b5 thru 0x0361c6 (0x1f12 bytes) 000217 1773668622.199996 read 0x02d06b thru 0x031d7a (0x4d10 bytes) 000219 1773668622.216829 mapread 0x0131f9 thru 0x016dfa (0x3c02 bytes) 000221 1773668622.241126 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf93eae75 (stale) Mirror 6: 0xf93eae75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf93eae75 (stale) Mirror 6: 0xf93eae75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1773668622.771078 write 0x00f1f3 thru 0x01b02c (0xbe3a bytes) 000223 1773668622.786402 trunc from 0x040000 to 0x02cf6a (0x13095 bytes) 000224 1773668622.810189 mapread 0x02293b thru 0x02cf6a (0xa630 bytes) 000225 1773668622.835251 read 0x00fe8e thru 0x01d3dd (0xd550 bytes) 000226 1773668622.836102 write 0x03c929 thru 0x03ffff (0x36d7 bytes) 000228 1773668622.844233 write 0x0280e6 thru 0x032ea7 (0xadc2 bytes) truncating to largest ever: 0x3a41a 000230 1773668622.858077 trunc from 0x040000 to 0x03a419 (0x5be6 bytes) 000231 1773668622.898918 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 4: 0xf732e9f7 (stale) Mirror 6: 0xf732e9f7 (stale) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 1: 0xd935161d Mirror 7: 0xd935161d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000235 1773668624.401131 mapread 0x03a019 thru 0x03a419 (0x401 bytes) 000236 1773668624.466111 mapread 0x01e885 thru 0x026e59 (0x85d5 bytes) 000237 1773668624.525263 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 1: 0xd935161d Mirror 7: 0xd935161d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000238 1773668625.446590 mapread 0x0241e1 thru 0x0248cb (0x6eb bytes) 000239 1773668625.465073 write 0x02cc78 thru 0x035f2f (0x92b8 bytes) 000240 1773668625.581073 read 0x005593 thru 0x010a80 (0xb4ee bytes) 000243 1773668625.598017 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) Mirror 7: 0xd935161d (stale) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 1: 0xc89d7b48 Mirror 8: 0xc89d7b48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000244 1773668626.487753 trunc from 0x03a41a to 0x029081 (0x11398 bytes) 000245 1773668626.683430 read 0x024be5 thru 0x029081 (0x449d bytes) 000246 1773668626.711781 mapread 0x018475 thru 0x01d32d (0x4eb9 bytes) 000247 1773668626.750203 MIRROR_RESYNC 4 mirrors 000250 1773668628.098870 write 0x024bd2 thru 0x027215 (0x2644 bytes) 000251 1773668628.203934 write 0x03dfea thru 0x03ffff (0x2016 bytes) 000253 1773668628.210558 mapwrite 0x005ef8 thru 0x01084f (0xa958 bytes) 000256 1773668628.304444 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1afd1a96 (stale) Mirror 7: 0x1afd1a96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1afd1a96 (stale) Mirror 7: 0x1afd1a96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000257 1773668628.719745 mapread 0x027926 thru 0x035071 (0xd74c bytes) 000258 1773668628.747444 mapwrite 0x00ffb7 thru 0x017b8e (0x7bd8 bytes) 000259 1773668628.812221 write 0x00ec8a thru 0x0174f8 (0x886f bytes) 000261 1773668628.824776 write 0x0082de thru 0x01544f (0xd172 bytes) 000264 1773668628.829747 write 0x01e581 thru 0x02df19 (0xf999 bytes) 000266 1773668628.846992 trunc from 0x040000 to 0x021ecc (0x1e133 bytes) 000269 1773668628.875223 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 6: 0x3704bd9b (stale) Mirror 7: 0x3704bd9b (stale) CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 1: 0x3d78f205 Mirror 8: 0x3d78f205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ecd: skipped 000270 1773668629.673082 mapwrite 0x007ebb thru 0x007ee7 (0x2d bytes) 000271 1773668629.805944 write 0x0307b0 thru 0x0373de (0x6c2f bytes) 000273 1773668629.808050 trunc from 0x0373df to 0x01c40e (0x1afd0 bytes) 000274 1773668629.832483 mapwrite 0x025e29 thru 0x033d48 (0xdf20 bytes) 000275 1773668629.920238 mapread 0x015ddd thru 0x018a34 (0x2c58 bytes) 000278 1773668629.929961 write 0x03d13e thru 0x03ffff (0x2ec2 bytes) 000283 1773668629.934588 read 0x03a112 thru 0x03b9c0 (0x18af bytes) 000286 1773668629.941494 mapread 0x0093f4 thru 0x00e257 (0x4e64 bytes) 000288 1773668629.953176 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1afd1a96 (stale) Mirror 7: 0x1afd1a96 (stale) Mirror 8: 0xe3b6e143 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5b8d984 Mirror 9: 0xe5b8d984 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000292 1773668630.897790 trunc from 0x040000 to 0x036d06 (0x92f9 bytes) 000294 1773668631.011088 read 0x013229 thru 0x021c62 (0xea3a bytes) 000296 1773668631.025780 read 0x01ae15 thru 0x01f169 (0x4355 bytes) 000298 1773668631.026987 mapwrite 0x00d86a thru 0x01255d (0x4cf4 bytes) 000299 1773668631.175126 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36d07): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d07): Mirror 6: 0x2eee1d45 (stale) Mirror 7: 0x2eee1d45 (stale) Mirror 8: 0xfecc4855 (stale) Mirror 9: 0x2b7080c4 (stale) CRC-32 checksum value for chunk [0, 0x36d07): Mirror 1: 0x922d3c28 Mirror 10: 0x922d3c28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d07: skipped 000302 1773668632.137071 read 0x0183c1 thru 0x01df77 (0x5bb7 bytes) 000303 1773668632.147596 read 0x003535 thru 0x009cc5 (0x6791 bytes) 000304 1773668632.161999 trunc from 0x036d07 to 0x000ed7 (0x35e2f bytes) 000307 1773668632.309140 read 0x000399 thru 0x000ed7 (0xb3f bytes) 000308 1773668632.316827 mapread 0x0003c1 thru 0x000ed7 (0xb17 bytes) 000310 1773668632.320475 write 0x02263d thru 0x02f6d3 (0xd097 bytes) 000311 1773668632.322917 mapread 0x00774f thru 0x010d68 (0x961a bytes) 000312 1773668632.330169 write 0x01f245 thru 0x02d6cb (0xe487 bytes) 000313 1773668632.331326 write 0x009852 thru 0x01491d (0xb0cc bytes) 000315 1773668632.332932 write 0x018632 thru 0x01e7f6 (0x61c5 bytes) 000317 1773668632.337505 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 6: 0xbbf4ce45 (stale) Mirror 7: 0xbbf4ce45 (stale) Mirror 8: 0x25e6f3bb (stale) Mirror 9: 0x5cd33f7c (stale) Mirror 10: 0x750b5950 (stale) CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 1: 0xacf888e4 Mirror 11: 0xacf888e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6d4: skipped 000321 1773668633.025586 mapwrite 0x00fa4b thru 0x0158c3 (0x5e79 bytes) 000322 1773668633.214069 mapread 0x021aae thru 0x025950 (0x3ea3 bytes) 000323 1773668633.237533 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 7: 0xbbf4ce45 (stale) Mirror 8: 0x25e6f3bb (stale) Mirror 9: 0x5cd33f7c (stale) Mirror 10: 0x750b5950 (stale) Mirror 11: 0xacf888e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 7: 0xbbf4ce45 (stale) Mirror 8: 0x25e6f3bb (stale) Mirror 9: 0x5cd33f7c (stale) Mirror 10: 0x750b5950 (stale) Mirror 11: 0xacf888e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6d4: skipped 000324 1773668633.597009 trunc from 0x02f6d4 to 0x0356d6 (0x6003 bytes) 000326 1773668633.646345 trunc from 0x0356d7 to 0x004774 (0x30f62 bytes) 000328 1773668633.669023 mapread 0x0026f8 thru 0x00334c (0xc55 bytes) 000329 1773668633.683101 mapwrite 0x00603d thru 0x00ae98 (0x4e5c bytes) 000331 1773668633.800238 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae99): Mirror 7: 0xb4ea27b (stale) Mirror 8: 0xb639d4a8 (stale) Mirror 9: 0xbbac3000 (stale) Mirror 11: 0xd3f22450 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae99): Mirror 7: 0xb4ea27b (stale) Mirror 8: 0xb639d4a8 (stale) Mirror 9: 0xbbac3000 (stale) Mirror 11: 0xd3f22450 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae99: skipped 000332 1773668634.112046 MIRROR_RESYNC 5 mirrors 000334 1773668635.070319 MIRROR_RESYNC 5 mirrors 000336 1773668635.437408 read 0x003ab7 thru 0x00ae98 (0x73e2 bytes) 000337 1773668635.445397 trunc from 0x00ae99 to 0x004449 (0x6a4f bytes) 000340 1773668635.525660 mapwrite 0x01560b thru 0x01d55f (0x7f55 bytes) 000343 1773668635.621540 mapwrite 0x03089d thru 0x03151a (0xc7e bytes) 000344 1773668635.688907 trunc from 0x03151b to 0x01848b (0x1908f bytes) 000345 1773668635.706529 trunc from 0x01848c to 0x025125 (0xcc9a bytes) 000346 1773668635.730011 read 0x01a74a thru 0x025125 (0xa9dc bytes) 000348 1773668635.740540 trunc from 0x025126 to 0x038c71 (0x13b4c bytes) 000349 1773668635.765131 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0xfbef2e18 (stale) Mirror 8: 0xfbef2e18 (stale) Mirror 9: 0xfbef2e18 (stale) Mirror 11: 0xfbef2e18 (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0xebb66bbd Mirror 12: 0xebb66bbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000350 1773668636.469799 mapread 0x021ade thru 0x023e0c (0x232f bytes) 000351 1773668636.498068 read 0x00f462 thru 0x017b0c (0x86ab bytes) 000352 1773668636.522250 write 0x022c37 thru 0x02404a (0x1414 bytes) 000354 1773668636.648374 read 0x021f63 thru 0x026f12 (0x4fb0 bytes) 000356 1773668636.652644 read 0x0298de thru 0x031749 (0x7e6c bytes) 000359 1773668636.682631 mapwrite 0x0330d4 thru 0x037adf (0x4a0c bytes) 000361 1773668636.755870 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0xfbef2e18 (stale) Mirror 8: 0xfbef2e18 (stale) Mirror 9: 0xfbef2e18 (stale) Mirror 11: 0xfbef2e18 (stale) Mirror 12: 0xebb66bbd (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0x99e921c5 Mirror 13: 0x99e921c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000363 1773668637.594543 read 0x023603 thru 0x029b7d (0x657b bytes) 000365 1773668637.606878 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0xfbef2e18 (stale) Mirror 8: 0xfbef2e18 (stale) Mirror 9: 0xfbef2e18 (stale) Mirror 12: 0xebb66bbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0xfbef2e18 (stale) Mirror 8: 0xfbef2e18 (stale) Mirror 9: 0xfbef2e18 (stale) Mirror 12: 0xebb66bbd (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0x99e921c5 Mirror 13: 0x99e921c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000367 1773668638.211935 trunc from 0x038c72 to 0x02ee63 (0x9e0e bytes) 000368 1773668638.284546 read 0x0024b9 thru 0x008d5e (0x68a6 bytes) 000369 1773668638.299380 mapread 0x00655b thru 0x007ff8 (0x1a9e bytes) 000370 1773668638.303467 mapwrite 0x018a35 thru 0x01bf29 (0x34f5 bytes) 000371 1773668638.361549 mapread 0x015d71 thru 0x01873d (0x29cd bytes) 000372 1773668638.362559 read 0x02db66 thru 0x02ee63 (0x12fe bytes) 000373 1773668638.368697 write 0x0349ba thru 0x039fba (0x5601 bytes) 000376 1773668638.372218 mapread 0x009385 thru 0x011bc4 (0x8840 bytes) 000377 1773668638.373798 read 0x00a350 thru 0x010129 (0x5dda bytes) 000378 1773668638.374516 write 0x03a413 thru 0x03b90f (0x14fd bytes) 000379 1773668638.376069 mapwrite 0x0002e6 thru 0x00e466 (0xe181 bytes) 000380 1773668638.420460 mapread 0x020f9d thru 0x029979 (0x89dd bytes) 000388 1773668638.428922 write 0x019fd3 thru 0x01f4ed (0x551b bytes) 000389 1773668638.434890 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b910): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b910): Mirror 7: 0xac4adc41 (stale) Mirror 8: 0xac4adc41 (stale) Mirror 9: 0xac4adc41 (stale) Mirror 12: 0x2aa1f67c (stale) Mirror 13: 0xfc81d86d (stale) CRC-32 checksum value for chunk [0, 0x3b910): Mirror 1: 0xfdccf35f Mirror 14: 0xfdccf35f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b910: skipped 000390 1773668639.139486 mapwrite 0x03874c thru 0x03ca65 (0x431a bytes) 000393 1773668639.355887 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 7: 0xa5237b8 (stale) Mirror 8: 0xa5237b8 (stale) Mirror 9: 0xa5237b8 (stale) Mirror 12: 0x8d27f2d3 (stale) Mirror 13: 0x2efdca6c (stale) Mirror 14: 0x2bb57af3 (stale) CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 1: 0x1e45ff98 Mirror 15: 0x1e45ff98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca66: skipped 000395 1773668640.106386 read 0x0046bc thru 0x01299e (0xe2e3 bytes) 000396 1773668640.115247 write 0x00dd87 thru 0x01a3e5 (0xc65f bytes) 000398 1773668640.186159 trunc from 0x03ca66 to 0x02a23a (0x1282b bytes) truncating to largest ever: 0x3ea68 000399 1773668640.214501 trunc from 0x02a23b to 0x03ea67 (0x1482d bytes) 000402 1773668640.231135 read 0x02eef4 thru 0x039b37 (0xac44 bytes) 000403 1773668640.241554 mapwrite 0x024b1e thru 0x032d96 (0xe279 bytes) 000405 1773668640.298193 trunc from 0x03ea68 to 0x0085aa (0x364bd bytes) 000406 1773668640.312967 read 0x001b35 thru 0x0085aa (0x6a76 bytes) 000407 1773668640.314233 trunc from 0x0085ab to 0x02ce2f (0x24885 bytes) 000412 1773668640.331678 read 0x021c29 thru 0x025a4c (0x3e24 bytes) 000414 1773668640.344144 mapread 0x002a7c thru 0x0104b6 (0xda3b bytes) 000415 1773668640.345484 mapread 0x0126b2 thru 0x01cc1d (0xa56c bytes) 000416 1773668640.348083 write 0x02f0e7 thru 0x037608 (0x8522 bytes) 000417 1773668640.350526 mapwrite 0x017377 thru 0x0221e4 (0xae6e bytes) 000418 1773668640.408595 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37609): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37609): Mirror 7: 0x54d86c88 (stale) Mirror 8: 0x54d86c88 (stale) Mirror 9: 0x54d86c88 (stale) Mirror 12: 0x93abdb0a (stale) Mirror 13: 0x9f96d179 (stale) Mirror 14: 0x7b3e1b09 (stale) Mirror 15: 0x7b3e1b09 (stale) CRC-32 checksum value for chunk [0, 0x37609): Mirror 1: 0xa8cadcde Mirror 16: 0xa8cadcde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37609: skipped 000422 1773668641.142549 write 0x025bf7 thru 0x026e6e (0x1278 bytes) 000423 1773668641.243043 mapread 0x02316b thru 0x02f68a (0xc520 bytes) 000424 1773668641.262371 mapwrite 0x021193 thru 0x0277ca (0x6638 bytes) 000426 1773668641.322075 trunc from 0x037609 to 0x026ba5 (0x10a63 bytes) 000428 1773668641.350816 write 0x027f35 thru 0x028e2c (0xef8 bytes) 000429 1773668641.351678 mapread 0x00d608 thru 0x018bbc (0xb5b5 bytes) 000430 1773668641.362244 read 0x0066b7 thru 0x00aa18 (0x4362 bytes) 000436 1773668641.369807 mapwrite 0x00291e thru 0x0117e8 (0xeecb bytes) 000437 1773668641.443692 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28e2d): Mirror 8: 0xda8b39d3 (stale) Mirror 9: 0xda8b39d3 (stale) Mirror 12: 0x2d629f76 (stale) Mirror 13: 0x54b60b17 (stale) Mirror 14: 0xd7b1b701 (stale) Mirror 15: 0xd7b1b701 (stale) Mirror 16: 0xc719c25c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e2d): Mirror 8: 0xda8b39d3 (stale) Mirror 9: 0xda8b39d3 (stale) Mirror 12: 0x2d629f76 (stale) Mirror 13: 0x54b60b17 (stale) Mirror 14: 0xd7b1b701 (stale) Mirror 15: 0xd7b1b701 (stale) Mirror 16: 0xc719c25c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e2d: skipped 000438 1773668641.884467 mapwrite 0x030445 thru 0x033203 (0x2dbf bytes) 000441 1773668641.953911 trunc from 0x033204 to 0x013ad9 (0x1f72a bytes) 000443 1773668641.965474 read 0x000ef5 thru 0x005be0 (0x4cec bytes) 000445 1773668641.976704 mapwrite 0x030c1a thru 0x03ffff (0xf3e6 bytes) 000446 1773668642.099652 trunc from 0x040000 to 0x02bae9 (0x14516 bytes) 000449 1773668642.120492 read 0x00c452 thru 0x01501c (0x8bcb bytes) 000451 1773668642.135337 read 0x0130e8 thru 0x01d047 (0x9f60 bytes) 000453 1773668642.136784 mapwrite 0x01d283 thru 0x02ad60 (0xdade bytes) 000455 1773668642.188899 mapread 0x00a05e thru 0x0171fa (0xd19d bytes) 000458 1773668642.196305 MIRROR_RESYNC 8 mirrors 000460 1773668643.326462 trunc from 0x02baea to 0x035906 (0x9e1d bytes) 000461 1773668643.414520 mapread 0x02ac86 thru 0x0357b3 (0xab2e bytes) 000462 1773668643.434280 trunc from 0x035907 to 0x018b26 (0x1cde0 bytes) 000463 1773668643.452330 trunc from 0x018b27 to 0x01938e (0x868 bytes) 000466 1773668643.469001 trunc from 0x01938f to 0x020749 (0x73bb bytes) 000468 1773668643.490142 read 0x00f9e4 thru 0x0130ec (0x3709 bytes) 000469 1773668643.499079 mapwrite 0x0206eb thru 0x02e6a3 (0xdfb9 bytes) 000470 1773668643.622467 trunc from 0x02e6a4 to 0x0206a0 (0xe003 bytes) 000474 1773668643.651646 write 0x02fa59 thru 0x03c8cb (0xce73 bytes) 000475 1773668643.659530 trunc from 0x03c8cc to 0x0307e2 (0xc0e9 bytes) 000477 1773668643.676298 read 0x01e7ff thru 0x029184 (0xa986 bytes) 000478 1773668643.703268 MIRROR_RESYNC 8 mirrors 000479 1773668644.638049 mapread 0x00222c thru 0x00ac9e (0x8a73 bytes) 000480 1773668644.654020 mapwrite 0x03b3a6 thru 0x03ffff (0x4c5a bytes) 000481 1773668644.859838 write 0x0322c7 thru 0x03ffff (0xdd39 bytes) 000483 1773668644.872037 read 0x0113cd thru 0x01e723 (0xd357 bytes) 000485 1773668644.887750 mapread 0x03574a thru 0x03665e (0xf15 bytes) 000487 1773668644.890511 mapread 0x0095c7 thru 0x00f067 (0x5aa1 bytes) 000490 1773668644.897579 write 0x002eb6 thru 0x0081ab (0x52f6 bytes) 000491 1773668644.898627 trunc from 0x040000 to 0x02314d (0x1ceb2 bytes) 000492 1773668644.916720 mapread 0x0153df thru 0x01d6de (0x8300 bytes) 000493 1773668644.926195 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2314e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2314e): Mirror 8: 0xf31a2697 (stale) Mirror 9: 0xf31a2697 (stale) Mirror 12: 0xf31a2697 (stale) Mirror 13: 0xf31a2697 (stale) Mirror 14: 0xf31a2697 (stale) Mirror 15: 0xf31a2697 (stale) Mirror 16: 0xf31a2697 (stale) CRC-32 checksum value for chunk [0, 0x2314e): Mirror 1: 0x1fa3f943 Mirror 17: 0x1fa3f943 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2314e: skipped 000496 1773668645.666792 MIRROR_RESYNC 9 mirrors 000502 1773668646.717781 read 0x01f1ff thru 0x02314d (0x3f4f bytes) 000505 1773668646.724861 mapwrite 0x0205aa thru 0x02f78b (0xf1e2 bytes) 000507 1773668646.868140 trunc from 0x02f78c to 0x02c19e (0x35ed bytes) 000509 1773668646.893299 mapread 0x02a275 thru 0x02c19e (0x1f2a bytes) 000510 1773668646.894220 mapwrite 0x018782 thru 0x021c4b (0x94ca bytes) 000511 1773668646.986321 mapread 0x013fe4 thru 0x016cef (0x2d0c bytes) 000512 1773668646.996099 MIRROR_RESYNC 9 mirrors 000514 1773668648.352514 write 0x03ee3c thru 0x03ffff (0x11c4 bytes) 000515 1773668648.417058 MIRROR_RESYNC 9 mirrors 000516 1773668649.254719 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) 000517 1773668649.534001 MIRROR_RESYNC 8 mirrors 000518 1773668649.939454 trunc from 0x040000 to 0x01e729 (0x218d6 bytes) 000521 1773668650.020304 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 8: 0xbe66b40d (stale) Mirror 9: 0xbe66b40d (stale) Mirror 13: 0xbe66b40d (stale) Mirror 14: 0xbe66b40d (stale) Mirror 15: 0xbe66b40d (stale) Mirror 16: 0xbe66b40d (stale) Mirror 17: 0xbe66b40d (stale) CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 1: 0xbe66b40d Mirror 18: 0xbe66b40d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e72a: skipped 000522 1773668650.870440 mapread 0x014aa3 thru 0x01867c (0x3bda bytes) 000524 1773668650.888165 write 0x02abfd thru 0x03991e (0xed22 bytes) 000525 1773668650.979028 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3991f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3991f): Mirror 8: 0xd41dc13e (stale) Mirror 9: 0xd41dc13e (stale) Mirror 13: 0xd41dc13e (stale) Mirror 14: 0xd41dc13e (stale) Mirror 15: 0xd41dc13e (stale) Mirror 16: 0xd41dc13e (stale) Mirror 17: 0xd41dc13e (stale) Mirror 18: 0x3d931c22 (stale) CRC-32 checksum value for chunk [0, 0x3991f): Mirror 1: 0x5f8b43bb Mirror 19: 0x5f8b43bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3991f: skipped 000527 1773668651.838337 write 0x03c8e1 thru 0x03ffff (0x371f bytes) 000528 1773668651.895290 mapread 0x038639 thru 0x03ffff (0x79c7 bytes) 000529 1773668651.913391 mapwrite 0x014ded thru 0x01c715 (0x7929 bytes) 000531 1773668651.966784 write 0x00e7d9 thru 0x019687 (0xaeaf bytes) 000532 1773668651.969089 mapwrite 0x0311ba thru 0x03cdfc (0xbc43 bytes) 000533 1773668652.019757 mapwrite 0x023553 thru 0x02a392 (0x6e40 bytes) 000535 1773668652.110525 mapread 0x01f691 thru 0x020ffe (0x196e bytes) 000537 1773668652.112836 mapwrite 0x0078c5 thru 0x014a34 (0xd170 bytes) 000539 1773668652.161553 mapwrite 0x00d8a8 thru 0x0199af (0xc108 bytes) 000540 1773668652.214946 trunc from 0x040000 to 0x002b64 (0x3d49b bytes) 000542 1773668652.239451 mapwrite 0x0123a6 thru 0x02165e (0xf2b9 bytes) 000543 1773668652.326079 mapread 0x00ed5b thru 0x01ea46 (0xfcec bytes) 000546 1773668652.334164 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2165f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2165f): Mirror 8: 0xcdf8d770 (stale) Mirror 9: 0xcdf8d770 (stale) Mirror 13: 0xcdf8d770 (stale) Mirror 14: 0xcdf8d770 (stale) Mirror 15: 0xcdf8d770 (stale) Mirror 16: 0xcdf8d770 (stale) Mirror 17: 0xcdf8d770 (stale) Mirror 18: 0x3dd6c05e (stale) Mirror 19: 0x3dd6c05e (stale) CRC-32 checksum value for chunk [0, 0x2165f): Mirror 1: 0x94af4d06 Mirror 20: 0x94af4d06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2165f: skipped 000547 1773668652.985304 trunc from 0x02165f to 0x03de2d (0x1c7cf bytes) 000548 1773668653.077500 mapread 0x02dcfb thru 0x03a867 (0xcb6d bytes) 000550 1773668653.111019 trunc from 0x03de2e to 0x006c0e (0x3721f bytes) 000551 1773668653.127512 read 0x00514a thru 0x006c0e (0x1ac5 bytes) 000552 1773668653.131564 mapread 0x001391 thru 0x006c0e (0x587e bytes) 000553 1773668653.137986 read 0x0027cc thru 0x002eb1 (0x6e6 bytes) 000556 1773668653.139810 read 0x00325d thru 0x006c0e (0x39b2 bytes) 000557 1773668653.142397 write 0x03538d thru 0x03ffff (0xac73 bytes) 000558 1773668653.144195 mapread 0x00300f thru 0x00ec4f (0xbc41 bytes) 000560 1773668653.154591 mapwrite 0x0283eb thru 0x02cdbc (0x49d2 bytes) 000561 1773668653.211511 mapwrite 0x0078ad thru 0x013fb7 (0xc70b bytes) 000563 1773668653.309490 mapwrite 0x029d86 thru 0x036368 (0xc5e3 bytes) 000564 1773668653.382848 write 0x0141c4 thru 0x01fa24 (0xb861 bytes) 000565 1773668653.385277 trunc from 0x040000 to 0x017bd0 (0x2842f bytes) 000566 1773668653.405600 trunc from 0x017bd1 to 0x00e036 (0x9b9a bytes) 000567 1773668653.435809 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe037): Mirror 8: 0xfb1352ef (stale) Mirror 9: 0xfb1352ef (stale) Mirror 14: 0xfb1352ef (stale) Mirror 15: 0xfb1352ef (stale) Mirror 16: 0xfb1352ef (stale) Mirror 17: 0xfb1352ef (stale) Mirror 18: 0xfb1352ef (stale) Mirror 19: 0xfb1352ef (stale) Mirror 20: 0x692d6e11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe037): Mirror 8: 0xfb1352ef (stale) Mirror 9: 0xfb1352ef (stale) Mirror 14: 0xfb1352ef (stale) Mirror 15: 0xfb1352ef (stale) Mirror 16: 0xfb1352ef (stale) Mirror 17: 0xfb1352ef (stale) Mirror 18: 0xfb1352ef (stale) Mirror 19: 0xfb1352ef (stale) Mirror 20: 0x692d6e11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe037: skipped 000570 1773668653.816250 mapread 0x003751 thru 0x00e036 (0xa8e6 bytes) 000572 1773668653.844243 mapread 0x005f87 thru 0x00e036 (0x80b0 bytes) 000574 1773668653.846984 write 0x002c01 thru 0x007505 (0x4905 bytes) 000575 1773668653.847990 read 0x00031a thru 0x007e79 (0x7b60 bytes) 000577 1773668653.861121 MIRROR_RESYNC 10 mirrors 000579 1773668655.227899 mapread 0x00b283 thru 0x00e036 (0x2db4 bytes) 000581 1773668655.245425 trunc from 0x00e037 to 0x02d98b (0x1f955 bytes) 000582 1773668655.320414 trunc from 0x02d98c to 0x039e0e (0xc483 bytes) 000583 1773668655.351377 trunc from 0x039e0f to 0x018dda (0x21034 bytes) 000585 1773668655.373810 trunc from 0x018ddb to 0x033238 (0x1a45e bytes) 000586 1773668655.403002 mapwrite 0x0331b8 thru 0x033ee3 (0xd2c bytes) 000588 1773668655.535167 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 8: 0x890bf27c (stale) Mirror 9: 0x890bf27c (stale) Mirror 14: 0x890bf27c (stale) Mirror 15: 0x890bf27c (stale) Mirror 16: 0x890bf27c (stale) Mirror 17: 0x890bf27c (stale) Mirror 18: 0x890bf27c (stale) Mirror 19: 0x890bf27c (stale) Mirror 20: 0x890bf27c (stale) CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 1: 0xdb5cdf72 Mirror 21: 0xdb5cdf72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ee4: skipped 000589 1773668656.381496 write 0x02dd29 thru 0x039392 (0xb66a bytes) 000590 1773668656.468530 read 0x0027bd thru 0x00d2ed (0xab31 bytes) 000591 1773668656.477398 read 0x01866b thru 0x01905c (0x9f2 bytes) 000592 1773668656.488461 trunc from 0x039393 to 0x028ff9 (0x10399 bytes) 000595 1773668656.520710 mapread 0x010fd1 thru 0x01cd1f (0xbd4f bytes) 000597 1773668656.523889 mapwrite 0x017078 thru 0x01c1bb (0x5144 bytes) 000600 1773668656.563296 read 0x009990 thru 0x014135 (0xa7a6 bytes) 000601 1773668656.564167 mapread 0x018692 thru 0x01f00f (0x697e bytes) 000605 1773668656.567280 write 0x0173d0 thru 0x023c5f (0xc890 bytes) 000607 1773668656.569520 write 0x0359d6 thru 0x037eeb (0x2516 bytes) 000608 1773668656.570675 trunc from 0x037eec to 0x018518 (0x1f9d3 bytes) 000611 1773668656.598587 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18519): Mirror 8: 0x705fb0e8 (stale) Mirror 9: 0x705fb0e8 (stale) Mirror 14: 0x705fb0e8 (stale) Mirror 16: 0x705fb0e8 (stale) Mirror 17: 0x705fb0e8 (stale) Mirror 18: 0x705fb0e8 (stale) Mirror 19: 0x705fb0e8 (stale) Mirror 20: 0x705fb0e8 (stale) Mirror 21: 0x705fb0e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18519): Mirror 8: 0x705fb0e8 (stale) Mirror 9: 0x705fb0e8 (stale) Mirror 14: 0x705fb0e8 (stale) Mirror 16: 0x705fb0e8 (stale) Mirror 17: 0x705fb0e8 (stale) Mirror 18: 0x705fb0e8 (stale) Mirror 19: 0x705fb0e8 (stale) Mirror 20: 0x705fb0e8 (stale) Mirror 21: 0x705fb0e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18519: skipped 000612 1773668657.041438 mapwrite 0x0243b1 thru 0x030037 (0xbc87 bytes) 000613 1773668657.132777 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30038): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30038): Mirror 8: 0x3e03e3b7 (stale) Mirror 9: 0x3e03e3b7 (stale) Mirror 14: 0x3e03e3b7 (stale) Mirror 16: 0x3e03e3b7 (stale) Mirror 17: 0x3e03e3b7 (stale) Mirror 18: 0x3e03e3b7 (stale) Mirror 19: 0x3e03e3b7 (stale) Mirror 20: 0x3e03e3b7 (stale) Mirror 21: 0x3e03e3b7 (stale) CRC-32 checksum value for chunk [0, 0x30038): Mirror 1: 0x77448b0c Mirror 22: 0x77448b0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30038: skipped 000620 1773668657.850935 read 0x017c33 thru 0x01ad2c (0x30fa bytes) 000621 1773668657.859463 write 0x03e147 thru 0x03ffff (0x1eb9 bytes) 000623 1773668657.912133 read 0x02c89a thru 0x0304be (0x3c25 bytes) 000626 1773668657.928916 mapread 0x0304a2 thru 0x03ffff (0xfb5e bytes) 000628 1773668657.931188 mapwrite 0x03dbd5 thru 0x03ffff (0x242b bytes) 000630 1773668657.998774 read 0x009dbc thru 0x01186f (0x7ab4 bytes) 000633 1773668658.001139 read 0x01bac8 thru 0x02aab2 (0xefeb bytes) 000635 1773668658.002474 mapwrite 0x037657 thru 0x03ffff (0x89a9 bytes) 000636 1773668658.043277 mapread 0x02854e thru 0x037d50 (0xf803 bytes) 000637 1773668658.044522 mapread 0x006269 thru 0x00f277 (0x900f bytes) 000640 1773668658.050159 trunc from 0x040000 to 0x0244cc (0x1bb33 bytes) 000642 1773668658.086706 mapwrite 0x03dfd5 thru 0x03ffff (0x202b bytes) 000643 1773668658.182554 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf77ef44a (stale) Mirror 9: 0xf77ef44a (stale) Mirror 14: 0xf77ef44a (stale) Mirror 16: 0xf77ef44a (stale) Mirror 17: 0xf77ef44a (stale) Mirror 18: 0xf77ef44a (stale) Mirror 19: 0xf77ef44a (stale) Mirror 20: 0xf77ef44a (stale) Mirror 21: 0xa4a4129e (stale) Mirror 22: 0xe1b3cd03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c8b66a9 Mirror 23: 0x3c8b66a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1773668658.820115 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf77ef44a (stale) Mirror 9: 0xf77ef44a (stale) Mirror 14: 0xf77ef44a (stale) Mirror 16: 0xf77ef44a (stale) Mirror 17: 0xf77ef44a (stale) Mirror 18: 0xf77ef44a (stale) Mirror 19: 0xf77ef44a (stale) Mirror 20: 0xf77ef44a (stale) Mirror 21: 0xa4a4129e (stale) Mirror 22: 0xe1b3cd03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf77ef44a (stale) Mirror 9: 0xf77ef44a (stale) Mirror 14: 0xf77ef44a (stale) Mirror 16: 0xf77ef44a (stale) Mirror 17: 0xf77ef44a (stale) Mirror 18: 0xf77ef44a (stale) Mirror 19: 0xf77ef44a (stale) Mirror 20: 0xf77ef44a (stale) Mirror 21: 0xa4a4129e (stale) Mirror 22: 0xe1b3cd03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1773668659.395448 write 0x0333e3 thru 0x0399be (0x65dc bytes) 000646 1773668659.414997 MIRROR_RESYNC 11 mirrors 000647 1773668660.669248 mapwrite 0x01140b thru 0x018d7a (0x7970 bytes) 000649 1773668660.815751 trunc from 0x040000 to 0x01a746 (0x258b9 bytes) 000650 1773668660.838138 write 0x01b9e0 thru 0x01e028 (0x2649 bytes) 000652 1773668660.841119 mapread 0x0026cc thru 0x00c299 (0x9bce bytes) 000653 1773668660.848153 read 0x011f60 thru 0x012520 (0x5c1 bytes) 000654 1773668660.848814 trunc from 0x01e029 to 0x008c58 (0x153d0 bytes) 000657 1773668660.867937 trunc from 0x008c59 to 0x00c98e (0x3d36 bytes) 000658 1773668660.875543 trunc from 0x00c98f to 0x02a8b5 (0x1df27 bytes) 000660 1773668660.896942 mapwrite 0x00a612 thru 0x0120d8 (0x7ac7 bytes) 000661 1773668660.960474 mapread 0x00167b thru 0x00df87 (0xc90d bytes) 000662 1773668660.962683 write 0x028900 thru 0x02ced5 (0x45d6 bytes) 000663 1773668660.974224 mapwrite 0x0131b6 thru 0x01a574 (0x73bf bytes) 000664 1773668661.011838 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ced6): Mirror 8: 0xc57a8530 (stale) Mirror 9: 0xc57a8530 (stale) Mirror 14: 0xc57a8530 (stale) Mirror 16: 0xc57a8530 (stale) Mirror 17: 0xc57a8530 (stale) Mirror 18: 0xc57a8530 (stale) Mirror 19: 0xc57a8530 (stale) Mirror 20: 0xc57a8530 (stale) Mirror 22: 0xc57a8530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ced6): Mirror 8: 0xc57a8530 (stale) Mirror 9: 0xc57a8530 (stale) Mirror 14: 0xc57a8530 (stale) Mirror 16: 0xc57a8530 (stale) Mirror 17: 0xc57a8530 (stale) Mirror 18: 0xc57a8530 (stale) Mirror 19: 0xc57a8530 (stale) Mirror 20: 0xc57a8530 (stale) Mirror 22: 0xc57a8530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ced6: skipped 000665 1773668661.478368 read 0x019ac9 thru 0x025d0f (0xc247 bytes) 000666 1773668661.486034 trunc from 0x02ced6 to 0x01a3a3 (0x12b32 bytes) 000667 1773668661.505198 MIRROR_RESYNC 10 mirrors 000670 1773668662.536946 mapread 0x004f09 thru 0x0086e5 (0x37dd bytes) 000672 1773668662.563249 trunc from 0x01a3a4 to 0x03284c (0x184a9 bytes) 000673 1773668662.620072 mapread 0x027656 thru 0x03284c (0xb1f7 bytes) 000674 1773668662.630798 trunc from 0x03284d to 0x017f6a (0x1a8e2 bytes) 000677 1773668662.653575 read 0x00c8c2 thru 0x017f6a (0xb6a9 bytes) 000678 1773668662.673903 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17f6b): Mirror 8: 0xee97fd21 (stale) Mirror 9: 0xee97fd21 (stale) Mirror 14: 0xee97fd21 (stale) Mirror 16: 0xee97fd21 (stale) Mirror 18: 0xee97fd21 (stale) Mirror 19: 0xee97fd21 (stale) Mirror 20: 0xee97fd21 (stale) Mirror 22: 0xee97fd21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f6b): Mirror 8: 0xee97fd21 (stale) Mirror 9: 0xee97fd21 (stale) Mirror 14: 0xee97fd21 (stale) Mirror 16: 0xee97fd21 (stale) Mirror 18: 0xee97fd21 (stale) Mirror 19: 0xee97fd21 (stale) Mirror 20: 0xee97fd21 (stale) Mirror 22: 0xee97fd21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f6b: skipped 000680 1773668663.085749 mapwrite 0x02665a thru 0x031787 (0xb12e bytes) 000682 1773668663.217934 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x625bdd4 (stale) Mirror 9: 0x625bdd4 (stale) Mirror 14: 0x625bdd4 (stale) Mirror 16: 0x625bdd4 (stale) Mirror 18: 0x625bdd4 (stale) Mirror 19: 0x625bdd4 (stale) Mirror 20: 0x625bdd4 (stale) Mirror 22: 0x625bdd4 (stale) CRC-32 checksum value for chunk [0, 0x31788): Mirror 1: 0xd125861e Mirror 23: 0xd125861e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31788: skipped 000684 1773668663.797839 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x625bdd4 (stale) Mirror 9: 0x625bdd4 (stale) Mirror 16: 0x625bdd4 (stale) Mirror 18: 0x625bdd4 (stale) Mirror 19: 0x625bdd4 (stale) Mirror 20: 0x625bdd4 (stale) Mirror 22: 0x625bdd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x625bdd4 (stale) Mirror 9: 0x625bdd4 (stale) Mirror 16: 0x625bdd4 (stale) Mirror 18: 0x625bdd4 (stale) Mirror 19: 0x625bdd4 (stale) Mirror 20: 0x625bdd4 (stale) Mirror 22: 0x625bdd4 (stale) CRC-32 checksum value for chunk [0, 0x31788): Mirror 1: 0xd125861e Mirror 23: 0xd125861e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31788: skipped 000686 1773668664.241531 mapwrite 0x0381e2 thru 0x038cbc (0xadb bytes) 000687 1773668664.391963 mapwrite 0x025269 thru 0x030021 (0xadb9 bytes) 000688 1773668664.465964 trunc from 0x038cbd to 0x012037 (0x26c85 bytes) 000690 1773668664.479019 write 0x009e8b thru 0x013022 (0x9198 bytes) 000691 1773668664.485832 mapread 0x002f5f thru 0x00f2a7 (0xc349 bytes) 000692 1773668664.489364 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0xeeac7cef (stale) Mirror 9: 0xeeac7cef (stale) Mirror 16: 0xeeac7cef (stale) Mirror 18: 0xeeac7cef (stale) Mirror 19: 0xeeac7cef (stale) Mirror 20: 0xeeac7cef (stale) Mirror 22: 0xeeac7cef (stale) Mirror 23: 0xeeac7cef (stale) CRC-32 checksum value for chunk [0, 0x13023): Mirror 1: 0x46654886 Mirror 24: 0x46654886 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13023: skipped 000693 1773668665.128144 read 0x006aca thru 0x00d63c (0x6b73 bytes) 000694 1773668665.148158 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0xeeac7cef (stale) Mirror 9: 0xeeac7cef (stale) Mirror 16: 0xeeac7cef (stale) Mirror 18: 0xeeac7cef (stale) Mirror 19: 0xeeac7cef (stale) Mirror 20: 0xeeac7cef (stale) Mirror 22: 0xeeac7cef (stale) Mirror 23: 0xeeac7cef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0xeeac7cef (stale) Mirror 9: 0xeeac7cef (stale) Mirror 16: 0xeeac7cef (stale) Mirror 18: 0xeeac7cef (stale) Mirror 19: 0xeeac7cef (stale) Mirror 20: 0xeeac7cef (stale) Mirror 22: 0xeeac7cef (stale) Mirror 23: 0xeeac7cef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13023: skipped 000695 1773668665.598701 write 0x027026 thru 0x02ee7e (0x7e59 bytes) 000696 1773668665.618028 trunc from 0x02ee7f to 0x00b34d (0x23b31 bytes) 000697 1773668665.659270 write 0x026b94 thru 0x03065a (0x9ac7 bytes) 000700 1773668665.669219 trunc from 0x03065b to 0x025bac (0xaaae bytes) 000701 1773668665.699800 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x25bad): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bad): Mirror 8: 0x6d4314f2 (stale) Mirror 9: 0x6d4314f2 (stale) Mirror 16: 0x6d4314f2 (stale) Mirror 18: 0x6d4314f2 (stale) Mirror 19: 0x6d4314f2 (stale) Mirror 20: 0x6d4314f2 (stale) Mirror 22: 0x6d4314f2 (stale) Mirror 23: 0xd33d1b74 (stale) CRC-32 checksum value for chunk [0, 0x25bad): Mirror 1: 0xd61dc249 Mirror 24: 0xd61dc249 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bad: skipped 000703 1773668666.514881 mapwrite 0x01bce9 thru 0x02aee9 (0xf201 bytes) 000704 1773668666.722447 mapwrite 0x021218 thru 0x02a87c (0x9665 bytes) 000705 1773668666.772031 trunc from 0x02aeea to 0x015f12 (0x14fd7 bytes) 000708 1773668666.800450 write 0x01f487 thru 0x028eee (0x9a68 bytes) 000709 1773668666.803315 write 0x024168 thru 0x02b2f7 (0x7190 bytes) 000710 1773668666.804745 mapread 0x00dcb4 thru 0x015f1f (0x826c bytes) 000711 1773668666.818806 MIRROR_RESYNC 10 mirrors 000713 1773668668.128191 mapwrite 0x0112c6 thru 0x015a22 (0x475d bytes) 000714 1773668668.332641 read 0x029be9 thru 0x02b2f7 (0x170f bytes) 000716 1773668668.354281 write 0x03fd49 thru 0x03ffff (0x2b7 bytes) 000717 1773668668.363252 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8c56dc43 (stale) Mirror 9: 0x8c56dc43 (stale) Mirror 16: 0x8c56dc43 (stale) Mirror 19: 0x8c56dc43 (stale) Mirror 20: 0x8c56dc43 (stale) Mirror 22: 0x8c56dc43 (stale) Mirror 23: 0x8c56dc43 (stale) Mirror 24: 0x8c56dc43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x8c56dc43 (stale) Mirror 9: 0x8c56dc43 (stale) Mirror 16: 0x8c56dc43 (stale) Mirror 19: 0x8c56dc43 (stale) Mirror 20: 0x8c56dc43 (stale) Mirror 22: 0x8c56dc43 (stale) Mirror 23: 0x8c56dc43 (stale) Mirror 24: 0x8c56dc43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1773668668.930571 trunc from 0x040000 to 0x02e0cc (0x11f33 bytes) 000720 1773668668.955185 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 8: 0xf7e2962a (stale) Mirror 9: 0xf7e2962a (stale) Mirror 16: 0xf7e2962a (stale) Mirror 19: 0xf7e2962a (stale) Mirror 20: 0xf7e2962a (stale) Mirror 22: 0xf7e2962a (stale) Mirror 23: 0xf7e2962a (stale) Mirror 24: 0xf7e2962a (stale) CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 1: 0x253bc23d Mirror 25: 0x253bc23d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0cd: skipped 000721 1773668669.824328 mapread 0x0150f5 thru 0x01d19f (0x80ab bytes) 000724 1773668669.852599 trunc from 0x02e0cd to 0x03aa74 (0xc9a8 bytes) 000733 1773668669.924921 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 8: 0xdd974255 (stale) Mirror 9: 0xdd974255 (stale) Mirror 16: 0xdd974255 (stale) Mirror 19: 0xdd974255 (stale) Mirror 20: 0xdd974255 (stale) Mirror 22: 0xdd974255 (stale) Mirror 23: 0xdd974255 (stale) Mirror 24: 0xdd974255 (stale) Mirror 25: 0x459eef57 (stale) CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 1: 0x459eef57 Mirror 26: 0x459eef57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa75: skipped 000735 1773668670.903202 write 0x015c6a thru 0x01c2be (0x6655 bytes) 000736 1773668670.987250 trunc from 0x03aa75 to 0x0363bc (0x46b8 bytes) 000737 1773668671.024348 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x363bd): Mirror 8: 0x73c7b0b4 (stale) Mirror 16: 0x73c7b0b4 (stale) Mirror 19: 0x73c7b0b4 (stale) Mirror 20: 0x73c7b0b4 (stale) Mirror 22: 0x73c7b0b4 (stale) Mirror 23: 0x73c7b0b4 (stale) Mirror 24: 0x73c7b0b4 (stale) Mirror 25: 0x940461ec (stale) Mirror 26: 0x940461ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x363bd): Mirror 8: 0x73c7b0b4 (stale) Mirror 16: 0x73c7b0b4 (stale) Mirror 19: 0x73c7b0b4 (stale) Mirror 20: 0x73c7b0b4 (stale) Mirror 22: 0x73c7b0b4 (stale) Mirror 23: 0x73c7b0b4 (stale) Mirror 24: 0x73c7b0b4 (stale) Mirror 25: 0x940461ec (stale) Mirror 26: 0x940461ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x363bd: skipped 000738 1773668671.546350 read 0x01e181 thru 0x02073e (0x25be bytes) 000739 1773668671.551762 trunc from 0x0363bd to 0x001d28 (0x34694 bytes) 000744 1773668671.569321 read 0x000bc6 thru 0x001d28 (0x1163 bytes) 000745 1773668671.576207 mapread 0x000e12 thru 0x001d28 (0xf17 bytes) 000746 1773668671.580369 MIRROR_RESYNC 10 mirrors 000748 1773668672.724463 mapwrite 0x01e73d thru 0x0215ff (0x2ec3 bytes) 000750 1773668672.875980 mapread 0x007504 thru 0x00bc38 (0x4735 bytes) 000751 1773668672.900684 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x30d5f594 (stale) Mirror 16: 0x30d5f594 (stale) Mirror 19: 0x30d5f594 (stale) Mirror 22: 0x30d5f594 (stale) Mirror 23: 0x30d5f594 (stale) Mirror 24: 0x30d5f594 (stale) Mirror 25: 0x30d5f594 (stale) Mirror 26: 0x30d5f594 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x30d5f594 (stale) Mirror 16: 0x30d5f594 (stale) Mirror 19: 0x30d5f594 (stale) Mirror 22: 0x30d5f594 (stale) Mirror 23: 0x30d5f594 (stale) Mirror 24: 0x30d5f594 (stale) Mirror 25: 0x30d5f594 (stale) Mirror 26: 0x30d5f594 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21600: skipped 000752 1773668673.357888 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x30d5f594 (stale) Mirror 16: 0x30d5f594 (stale) Mirror 19: 0x30d5f594 (stale) Mirror 22: 0x30d5f594 (stale) Mirror 23: 0x30d5f594 (stale) Mirror 24: 0x30d5f594 (stale) Mirror 25: 0x30d5f594 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x30d5f594 (stale) Mirror 16: 0x30d5f594 (stale) Mirror 19: 0x30d5f594 (stale) Mirror 22: 0x30d5f594 (stale) Mirror 23: 0x30d5f594 (stale) Mirror 24: 0x30d5f594 (stale) Mirror 25: 0x30d5f594 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21600: skipped 000756 1773668673.740247 MIRROR_RESYNC 8 mirrors 000759 1773668674.800471 mapwrite 0x0121f7 thru 0x022025 (0xfe2f bytes) 000760 1773668674.964178 read 0x008624 thru 0x012044 (0x9a21 bytes) 000761 1773668674.986711 write 0x00c1c2 thru 0x018b80 (0xc9bf bytes) 000762 1773668674.988460 mapread 0x00db5a thru 0x01d597 (0xfa3e bytes) 000764 1773668674.992743 write 0x02cd35 thru 0x032957 (0x5c23 bytes) 000765 1773668674.994945 mapread 0x013788 thru 0x013e84 (0x6fd bytes) 000766 1773668674.995793 read 0x02d296 thru 0x032957 (0x56c2 bytes) 000768 1773668674.997495 mapread 0x00928d thru 0x0141a9 (0xaf1d bytes) 000769 1773668674.998729 read 0x010127 thru 0x01a0b4 (0x9f8e bytes) 000770 1773668674.999859 mapread 0x019b2d thru 0x022949 (0x8e1d bytes) 000774 1773668675.003488 trunc from 0x032958 to 0x038663 (0x5d0c bytes) 000776 1773668675.027191 write 0x006206 thru 0x008f51 (0x2d4c bytes) 000778 1773668675.036679 trunc from 0x038664 to 0x0207f7 (0x17e6c bytes) 000779 1773668675.057692 read 0x000d02 thru 0x00cfaf (0xc2ae bytes) 000781 1773668675.059437 mapwrite 0x0278a8 thru 0x0285de (0xd37 bytes) 000787 1773668675.132145 read 0x01eaea thru 0x0285de (0x9af5 bytes) 000788 1773668675.133083 write 0x027d09 thru 0x02cd3b (0x5033 bytes) 000789 1773668675.134201 read 0x01dfdb thru 0x028aa4 (0xaaca bytes) 000792 1773668675.136268 write 0x03aa1f thru 0x03ffff (0x55e1 bytes) 000793 1773668675.144215 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x22310ecb (stale) Mirror 16: 0x22310ecb (stale) Mirror 19: 0x22310ecb (stale) Mirror 22: 0x22310ecb (stale) Mirror 23: 0x22310ecb (stale) Mirror 24: 0x22310ecb (stale) Mirror 25: 0x22310ecb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4eb3df Mirror 26: 0xc4eb3df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1773668675.865505 mapread 0x016e49 thru 0x017b70 (0xd28 bytes) 000796 1773668675.874083 trunc from 0x040000 to 0x0378e4 (0x871b bytes) 000799 1773668675.948546 write 0x00790e thru 0x009c56 (0x2349 bytes) 000800 1773668675.953971 mapwrite 0x030805 thru 0x0354e2 (0x4cde bytes) 000801 1773668676.014376 write 0x0389d6 thru 0x03ffff (0x762a bytes) 000802 1773668676.015703 trunc from 0x040000 to 0x0367d8 (0x9827 bytes) 000803 1773668676.033825 write 0x02dcc7 thru 0x02e4dd (0x817 bytes) 000805 1773668676.035184 write 0x039252 thru 0x03ffff (0x6dae bytes) 000806 1773668676.036216 write 0x03021a thru 0x038ac8 (0x88af bytes) 000807 1773668676.042348 mapread 0x03c980 thru 0x03ffff (0x3680 bytes) 000808 1773668676.043747 trunc from 0x040000 to 0x0104dc (0x2fb23 bytes) 000811 1773668676.059314 mapwrite 0x02a904 thru 0x02bd06 (0x1403 bytes) 000812 1773668676.128808 write 0x006de8 thru 0x01556e (0xe787 bytes) 000814 1773668676.138824 trunc from 0x02bd07 to 0x00f14b (0x1cbbb bytes) 000815 1773668676.155356 mapread 0x00522b thru 0x00f14b (0x9f21 bytes) 000816 1773668676.157753 read 0x00a985 thru 0x00e134 (0x37b0 bytes) 000817 1773668676.164801 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xf14c): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf14c): Mirror 8: 0xafc5889e (stale) Mirror 16: 0xafc5889e (stale) Mirror 19: 0xafc5889e (stale) Mirror 22: 0xafc5889e (stale) Mirror 23: 0xafc5889e (stale) Mirror 24: 0xafc5889e (stale) Mirror 25: 0xafc5889e (stale) Mirror 26: 0xe529e0bc (stale) CRC-32 checksum value for chunk [0, 0xf14c): Mirror 1: 0x20cf5de9 Mirror 27: 0x20cf5de9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf14c: skipped 000819 1773668676.746392 mapread 0x009210 thru 0x00ef91 (0x5d82 bytes) 000821 1773668676.806400 MIRROR_RESYNC 10 mirrors 000822 1773668677.908164 mapwrite 0x0142c2 thru 0x021743 (0xd482 bytes) 000823 1773668678.103087 MIRROR_RESYNC 10 mirrors 000824 1773668679.266947 MIRROR_RESYNC 10 mirrors 000825 1773668679.695352 MIRROR_EXTEND to 11 mirrors 000826 1773668680.233378 read 0x000b01 thru 0x005621 (0x4b21 bytes) 000828 1773668680.239275 read 0x01b116 thru 0x01b937 (0x822 bytes) 000831 1773668680.247843 write 0x01a4db thru 0x01fbb7 (0x56dd bytes) 000832 1773668680.341174 trunc from 0x021744 to 0x0289d6 (0x7293 bytes) 000833 1773668680.364350 mapread 0x01c1af thru 0x023a2f (0x7881 bytes) 000834 1773668680.377356 mapread 0x012969 thru 0x01e3dc (0xba74 bytes) 000836 1773668680.411238 write 0x01cf3a thru 0x01ff08 (0x2fcf bytes) 000837 1773668680.413002 write 0x027494 thru 0x034f0e (0xda7b bytes) 000840 1773668680.421176 write 0x0320d4 thru 0x03f61a (0xd547 bytes) 000842 1773668680.427051 read 0x01801f thru 0x01c53d (0x451f bytes) 000843 1773668680.427883 write 0x03ef51 thru 0x03ff6c (0x101c bytes) 000846 1773668680.432746 trunc from 0x03ff6d to 0x0352d0 (0xac9c bytes) 000848 1773668680.446939 trunc from 0x0352d1 to 0x012f3e (0x22392 bytes) 000849 1773668680.471545 mapread 0x00a466 thru 0x012f3e (0x8ad9 bytes) 000850 1773668680.483570 write 0x0212ef thru 0x02ed48 (0xda5a bytes) 000851 1773668680.485404 read 0x016ba7 thru 0x01e65f (0x7ab9 bytes) 000855 1773668680.500199 write 0x0042f4 thru 0x008b75 (0x4882 bytes) 000856 1773668680.502090 write 0x0201b6 thru 0x0266de (0x6529 bytes) 000859 1773668680.509994 MIRROR_RESYNC 11 mirrors 000860 1773668681.783348 mapwrite 0x037d69 thru 0x03ffff (0x8297 bytes) 000861 1773668681.942293 read 0x00decb thru 0x00f07b (0x11b1 bytes) 000862 1773668681.960463 trunc from 0x040000 to 0x001a64 (0x3e59b bytes) 000863 1773668681.993359 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a65): Mirror 8: 0xfcbae09e (stale) Mirror 16: 0xfcbae09e (stale) Mirror 19: 0xfcbae09e (stale) Mirror 22: 0xfcbae09e (stale) Mirror 24: 0xfcbae09e (stale) Mirror 25: 0xfcbae09e (stale) Mirror 26: 0xfcbae09e (stale) Mirror 27: 0xfcbae09e (stale) Mirror 28: 0xfcbae09e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a65): Mirror 8: 0xfcbae09e (stale) Mirror 16: 0xfcbae09e (stale) Mirror 19: 0xfcbae09e (stale) Mirror 22: 0xfcbae09e (stale) Mirror 24: 0xfcbae09e (stale) Mirror 25: 0xfcbae09e (stale) Mirror 26: 0xfcbae09e (stale) Mirror 27: 0xfcbae09e (stale) Mirror 28: 0xfcbae09e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a65: skipped 000864 1773668682.382313 mapwrite 0x016865 thru 0x026677 (0xfe13 bytes) 000865 1773668682.502186 trunc from 0x026678 to 0x0171b8 (0xf4bf bytes) 000866 1773668682.525593 write 0x0105d4 thru 0x01fd3d (0xf76a bytes) 000869 1773668682.548134 trunc from 0x01fd3e to 0x03b04e (0x1b311 bytes) 000871 1773668682.575234 mapread 0x01da1a thru 0x02be22 (0xe409 bytes) 000873 1773668682.621950 trunc from 0x03b04f to 0x02ad2f (0x1031f bytes) 000874 1773668682.654526 mapread 0x008b4c thru 0x013460 (0xa915 bytes) 000875 1773668682.666414 MIRROR_RESYNC 10 mirrors 000876 1773668683.773041 mapwrite 0x021054 thru 0x02fa35 (0xe9e2 bytes) 000879 1773668683.899189 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 8: 0xd6e73c7e (stale) Mirror 16: 0xd6e73c7e (stale) Mirror 19: 0xd6e73c7e (stale) Mirror 22: 0xd6e73c7e (stale) Mirror 24: 0xd6e73c7e (stale) Mirror 25: 0xd6e73c7e (stale) Mirror 26: 0xd6e73c7e (stale) Mirror 27: 0xd6e73c7e (stale) Mirror 28: 0xd6e73c7e (stale) CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 1: 0x37b79596 Mirror 29: 0x37b79596 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa36: skipped 000880 1773668684.699247 write 0x027b33 thru 0x02f117 (0x75e5 bytes) 000881 1773668684.771795 read 0x00393d thru 0x01083f (0xcf03 bytes) 000882 1773668684.778225 mapwrite 0x016645 thru 0x021f95 (0xb951 bytes) 000883 1773668684.871053 mapwrite 0x02f92a thru 0x03dc51 (0xe328 bytes) 000887 1773668685.069560 write 0x02e253 thru 0x03d630 (0xf3de bytes) 000888 1773668685.074883 trunc from 0x03dc52 to 0x00214d (0x3bb04 bytes) 000889 1773668685.105299 read 0x00047b thru 0x00214d (0x1cd3 bytes) 000895 1773668685.125669 trunc from 0x00214e to 0x028c7a (0x26b2d bytes) 000896 1773668685.173693 read 0x01b862 thru 0x024bed (0x938c bytes) 000897 1773668685.196927 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c7b): Mirror 8: 0x6876d39c (stale) Mirror 16: 0x6876d39c (stale) Mirror 19: 0x6876d39c (stale) Mirror 22: 0x6876d39c (stale) Mirror 24: 0x6876d39c (stale) Mirror 25: 0x6876d39c (stale) Mirror 26: 0x6876d39c (stale) Mirror 27: 0x6876d39c (stale) Mirror 28: 0x6876d39c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c7b): Mirror 8: 0x6876d39c (stale) Mirror 16: 0x6876d39c (stale) Mirror 19: 0x6876d39c (stale) Mirror 22: 0x6876d39c (stale) Mirror 24: 0x6876d39c (stale) Mirror 25: 0x6876d39c (stale) Mirror 26: 0x6876d39c (stale) Mirror 27: 0x6876d39c (stale) Mirror 28: 0x6876d39c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c7b: skipped 000898 1773668685.756275 write 0x0298a2 thru 0x036aab (0xd20a bytes) 000899 1773668685.757878 mapread 0x02fe65 thru 0x036aab (0x6c47 bytes) 000901 1773668685.761054 write 0x005e57 thru 0x00d95f (0x7b09 bytes) 000902 1773668685.768649 mapread 0x019c7b thru 0x023501 (0x9887 bytes) 000903 1773668685.789366 mapread 0x0198e6 thru 0x020b27 (0x7242 bytes) 000904 1773668685.794693 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: 0x5a82b994 (stale) Mirror 16: 0x5a82b994 (stale) Mirror 19: 0x5a82b994 (stale) Mirror 22: 0x5a82b994 (stale) Mirror 24: 0x5a82b994 (stale) Mirror 25: 0x5a82b994 (stale) Mirror 26: 0x5a82b994 (stale) Mirror 27: 0x5a82b994 (stale) Mirror 28: 0x5a82b994 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x74fbeb26 Mirror 29: 0x74fbeb26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000905 1773668686.601908 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: 0x5a82b994 (stale) Mirror 16: 0x5a82b994 (stale) Mirror 19: 0x5a82b994 (stale) Mirror 22: 0x5a82b994 (stale) Mirror 24: 0x5a82b994 (stale) Mirror 25: 0x5a82b994 (stale) Mirror 26: 0x5a82b994 (stale) Mirror 27: 0x5a82b994 (stale) Mirror 28: 0x5a82b994 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x74fbeb26 Mirror 29: 0x74fbeb26 Mirror 30: 0x74fbeb26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000906 1773668687.324906 read 0x022a50 thru 0x02f488 (0xca39 bytes) 000907 1773668687.340573 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 16: 0x5a82b994 (stale) Mirror 19: 0x5a82b994 (stale) Mirror 22: 0x5a82b994 (stale) Mirror 24: 0x5a82b994 (stale) Mirror 25: 0x5a82b994 (stale) Mirror 26: 0x5a82b994 (stale) Mirror 27: 0x5a82b994 (stale) Mirror 28: 0x5a82b994 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 16: 0x5a82b994 (stale) Mirror 19: 0x5a82b994 (stale) Mirror 22: 0x5a82b994 (stale) Mirror 24: 0x5a82b994 (stale) Mirror 25: 0x5a82b994 (stale) Mirror 26: 0x5a82b994 (stale) Mirror 27: 0x5a82b994 (stale) Mirror 28: 0x5a82b994 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x74fbeb26 Mirror 29: 0x74fbeb26 Mirror 30: 0x74fbeb26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000909 1773668687.903636 read 0x02526a thru 0x026e79 (0x1c10 bytes) 000912 1773668687.911199 write 0x019ea6 thru 0x0232ba (0x9415 bytes) 000913 1773668688.006984 trunc from 0x036aac to 0x02bf13 (0xab98 bytes) 000914 1773668688.037491 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xe05c9bd8 (stale) Mirror 19: 0xe05c9bd8 (stale) Mirror 22: 0xe05c9bd8 (stale) Mirror 24: 0xe05c9bd8 (stale) Mirror 25: 0xe05c9bd8 (stale) Mirror 26: 0xe05c9bd8 (stale) Mirror 27: 0xe05c9bd8 (stale) Mirror 28: 0xe05c9bd8 (stale) Mirror 29: 0x432420be (stale) Mirror 30: 0x432420be (stale) CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 1: 0xbcd403cc Mirror 31: 0xbcd403cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf14: skipped 000916 1773668689.102430 write 0x01bae1 thru 0x02b1d9 (0xf6f9 bytes) 000917 1773668689.208066 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x308:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xe05c9bd8 (stale) Mirror 19: 0xe05c9bd8 (stale) Mirror 24: 0xe05c9bd8 (stale) Mirror 25: 0xe05c9bd8 (stale) Mirror 26: 0xe05c9bd8 (stale) Mirror 27: 0xe05c9bd8 (stale) Mirror 28: 0xe05c9bd8 (stale) Mirror 29: 0x432420be (stale) Mirror 30: 0x432420be (stale) Mirror 31: 0xbcd403cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xe05c9bd8 (stale) Mirror 19: 0xe05c9bd8 (stale) Mirror 24: 0xe05c9bd8 (stale) Mirror 25: 0xe05c9bd8 (stale) Mirror 26: 0xe05c9bd8 (stale) Mirror 27: 0xe05c9bd8 (stale) Mirror 28: 0xe05c9bd8 (stale) Mirror 29: 0x432420be (stale) Mirror 30: 0x432420be (stale) Mirror 31: 0xbcd403cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf14: skipped 000918 1773668689.750592 write 0x006a73 thru 0x00b5b4 (0x4b42 bytes) 000919 1773668689.756133 trunc from 0x02bf14 to 0x002b46 (0x293cd bytes) 000920 1773668689.765464 read 0x00016b thru 0x002b46 (0x29dc bytes) 000921 1773668689.772058 read 0x0006d5 thru 0x002b46 (0x2472 bytes) 000922 1773668689.773874 write 0x03df82 thru 0x03ffff (0x207e bytes) 000923 1773668689.776201 write 0x03de1b thru 0x03ffff (0x21e5 bytes) 000924 1773668689.777296 mapwrite 0x020fec thru 0x0284cd (0x74e2 bytes) 000928 1773668689.835314 read 0x0381f7 thru 0x03cef3 (0x4cfd bytes) 000929 1773668689.836227 trunc from 0x040000 to 0x002a0f (0x3d5f0 bytes) 000930 1773668689.851565 trunc from 0x002a10 to 0x02324c (0x2083d bytes) 000931 1773668689.869095 read 0x01bfa7 thru 0x01dbcf (0x1c29 bytes) 000932 1773668689.876734 read 0x013654 thru 0x021ed8 (0xe885 bytes) 000937 1773668689.880336 trunc from 0x02324d to 0x02b5f0 (0x83a4 bytes) 000938 1773668689.891843 trunc from 0x02b5f1 to 0x01fe57 (0xb799 bytes) 000940 1773668689.917046 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 16: 0x5eaba316 (stale) Mirror 19: 0x5eaba316 (stale) Mirror 24: 0x5eaba316 (stale) Mirror 25: 0x5eaba316 (stale) Mirror 26: 0x5eaba316 (stale) Mirror 27: 0x5eaba316 (stale) Mirror 28: 0x5eaba316 (stale) Mirror 29: 0x59fdf828 (stale) Mirror 30: 0x59fdf828 (stale) Mirror 31: 0xb6b03e85 (stale) CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 1: 0xd8b461e7 Mirror 32: 0xd8b461e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe58: skipped 000941 1773668690.630512 read 0x010042 thru 0x01411d (0x40dc bytes) 000943 1773668690.666886 write 0x011a44 thru 0x02095e (0xef1b bytes) 000944 1773668690.820275 read 0x019e09 thru 0x02095e (0x6b56 bytes) 000946 1773668690.826231 mapwrite 0x022be5 thru 0x02dee7 (0xb303 bytes) 000948 1773668690.928971 trunc from 0x02dee8 to 0x0287ee (0x56f9 bytes) 000952 1773668690.944496 mapwrite 0x03f055 thru 0x03ffff (0xfab bytes) 000953 1773668691.002579 MIRROR_RESYNC 12 mirrors 000954 1773668692.658296 write 0x013337 thru 0x014e79 (0x1b43 bytes) 000956 1773668692.744974 write 0x005109 thru 0x0116b3 (0xc5ab bytes) 000958 1773668692.765730 trunc from 0x040000 to 0x03a2e1 (0x5d1e bytes) 000959 1773668692.788219 mapread 0x0159e6 thru 0x022bad (0xd1c8 bytes) 000961 1773668692.823948 read 0x01fdf2 thru 0x02514e (0x535d bytes) 000962 1773668692.824773 write 0x02e4d7 thru 0x0347d3 (0x62fd bytes) 000964 1773668692.848071 read 0x01f813 thru 0x027a1e (0x820c bytes) 000965 1773668692.848790 read 0x01eddb thru 0x01fa39 (0xc5f bytes) 000967 1773668692.849900 trunc from 0x03a2e2 to 0x03bd5b (0x1a7a bytes) 000968 1773668692.884421 mapwrite 0x0155fd thru 0x01b5a5 (0x5fa9 bytes) 000969 1773668692.944872 read 0x0349ae thru 0x03bd5b (0x73ae bytes) 000972 1773668692.975806 write 0x0004d0 thru 0x0031f6 (0x2d27 bytes) 000973 1773668692.976878 trunc from 0x03bd5c to 0x03be71 (0x116 bytes) 000974 1773668693.019086 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: 0x8165cf94 (stale) Mirror 19: 0x8165cf94 (stale) Mirror 24: 0x8165cf94 (stale) Mirror 25: 0x8165cf94 (stale) Mirror 26: 0x8165cf94 (stale) Mirror 27: 0x8165cf94 (stale) Mirror 28: 0x8165cf94 (stale) Mirror 29: 0x8165cf94 (stale) Mirror 30: 0x8165cf94 (stale) Mirror 31: 0x8165cf94 (stale) Mirror 32: 0x8165cf94 (stale) CRC-32 checksum value for chunk [0, 0x3be72): Mirror 1: 0x2e4d797a Mirror 33: 0x2e4d797a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be72: skipped 000977 1773668694.023490 write 0x01f90f thru 0x02072c (0xe1e bytes) 000978 1773668694.111030 write 0x015ee4 thru 0x01fddd (0x9efa bytes) 000983 1773668694.140825 MIRROR_RESYNC 13 mirrors 000985 1773668695.709404 MIRROR_EXTEND to 14 mirrors 000987 1773668696.368583 mapread 0x005900 thru 0x00d25f (0x7960 bytes) 000988 1773668696.396194 mapwrite 0x01bdd9 thru 0x028eb7 (0xd0df bytes) 000989 1773668696.568984 mapread 0x02a12c thru 0x0386c9 (0xe59e bytes) 000990 1773668696.582550 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: 0x2ecb11e6 (stale) Mirror 19: 0x2ecb11e6 (stale) Mirror 24: 0x2ecb11e6 (stale) Mirror 25: 0x2ecb11e6 (stale) Mirror 26: 0x2ecb11e6 (stale) Mirror 27: 0x2ecb11e6 (stale) Mirror 28: 0x2ecb11e6 (stale) Mirror 29: 0x2ecb11e6 (stale) Mirror 30: 0x2ecb11e6 (stale) Mirror 31: 0x2ecb11e6 (stale) Mirror 32: 0x2ecb11e6 (stale) Mirror 33: 0x2ecb11e6 (stale) Mirror 34: 0x2ecb11e6 (stale) CRC-32 checksum value for chunk [0, 0x3be72): Mirror 1: 0xfbf02219 Mirror 35: 0xfbf02219 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be72: skipped 000993 1773668697.351577 mapwrite 0x03673d thru 0x037969 (0x122d bytes) 000994 1773668697.469168 write 0x007a20 thru 0x00a9cc (0x2fad bytes) 000995 1773668697.478592 mapwrite 0x019fb2 thru 0x02734c (0xd39b bytes) 000996 1773668697.553554 trunc from 0x03be72 to 0x01e20e (0x1dc63 bytes) 000998 1773668697.609396 write 0x02cfea thru 0x0395cd (0xc5e4 bytes) 000999 1773668697.615295 write 0x00b1bd thru 0x00de22 (0x2c66 bytes) 001000 1773668697.616582 read 0x02ed17 thru 0x02f984 (0xc6e bytes) All operations completed A-OK! PASS 70a (111s) == sanity-flr test 71: check mirror extend parent fid ==== 09:45:01 (1773668701) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:45:04 (1773668704) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x455:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x459:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187885 s, 11.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x455:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x456:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x459:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199429 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x455:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x456:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x45a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x459:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x455:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x456:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x45a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x459:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x457:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 09:45:11 (1773668711) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre2 Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7646432 .. Extending file size to 2459264 .. Extending file size to 8799744 .. Extending file size to 3758240 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7782144 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8542752 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9258624 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8820640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56394 56395 56396 56398 56399 done 192.168.203.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre2 (opts:) 192.168.203.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:46:30 (1773668790) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre2 Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8275808 .. Extending file size to 7251488 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7568064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8100864 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8808032 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8111072 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8407104 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8839424 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8562848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58140 58142 58143 58145 58147 done 192.168.203.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre2 (opts:) 192.168.203.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:47:44 (1773668864) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4352 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4352 2204032 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112571 s, 18.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.268891 s, 7.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x70f7078a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:47:56 (1773668876) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0716337 s, 29.3 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 09:48:00 (1773668880) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x363:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x470:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x470:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x471:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x46c:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:48:04 (1773668884) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160586 s, 13.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x472:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x46d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x46e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:48:09 (1773668889) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137559 s, 15.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x473:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x474:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x46f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x473:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x474:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x46f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x470:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:48:14 (1773668894) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257691 s, 8.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x475:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x471:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x475:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x471:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x472:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:48:43 (1773668923) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg334-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071942 s, 14.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x476:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x474:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x473:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x476:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x474:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x473:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x477:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:49:35 (1773668975) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg334-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x475:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x476:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139586 s, 15.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x475:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x476:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x478:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x475:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x476:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x478:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:50:26 (1773669026) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg334-server mds-ost sync done. Creating new pool test_204f oleg334-server: Pool lustre.test_204f created Adding targets to pool oleg334-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg334-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0701234 s, 29.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x477:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x479:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x477:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x479:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg334-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg334-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg334-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:51:26 (1773669086) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:51:30 (1773669090) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:51:34 (1773669094) oleg334-server: Pool lustre.test_206 created Adding targets to pool oleg334-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg334-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg334-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg334-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg334-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:51:51 (1773669111) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597402 s, 17.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264247 s, 39.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa57201d1 Mirror 3: 0xa57201d1 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:51:58 (1773669118) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:52:00 (1773669120) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:52:02 (1773669122) Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.088495 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x16e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ec0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ec0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ec0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x740000 Read this (4 bytes): BBBB 192.168.203.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg334-client.virtnet /mnt/lustre2 (opts:) PASS 209a (23s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:52:25 (1773669145) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.011412 s, 718 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:52:29 (1773669149) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219706 s, 47.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x388 lmm_fid: [0x200000404:0x388:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x388 lmm_fid: [0x200000404:0x388:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x483:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834824 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 09:52:30.000000000 -0400 Modify: 2026-03-16 09:52:30.000000000 -0400 Change: 2026-03-16 09:52:31.000000000 -0400 Birth: 2026-03-16 09:52:30.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:52:34 (1773669154) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:52:37 (1773669157) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.367175 s, 112 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0549702 s, 74.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 16 09:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x38a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 16 09:52 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:52:42 (1773669162) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:52:46 (1773669166) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 986 sec ============ 09:52:51 (1773669171) === sanity-flr: start cleanup 09:52:52 (1773669172) === === sanity-flr: finish cleanup 09:52:56 (1773669176) ===