-----============= acceptance-small: sanity-flr ============----- Tue Sep 15 09:48:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg209-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:49:18 (1789480158) === oleg209-client.virtnet: executing check_config_client /mnt/lustre oleg209-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg209-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e1f18ffd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1f18ffd800.idle_timeout=debug disable quota as required oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:50:00 (1789480200) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7622] == sanity-flr test 0a: lfs mirror create with -N option == 09:50:03 (1789480203) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:50:25 (1789480225) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:50:30 (1789480230) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:50:34 (1789480234) PASS 0d (36s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:51:10 (1789480270) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:51:15 (1789480275) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:51:19 (1789480279) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e1f18ffd800.stats=clear osc.lustre-OST0001-osc-ffff9e1f18ffd800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:51:29 (1789480289) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:51:41 (1789480301) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:51:51 (1789480311) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:51:55 (1789480315) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:52:04 (1789480324) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 09:52:16 (1789480336) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:52:25 (1789480345) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:52:30 (1789480350) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:52:39 (1789480359) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:52:52 (1789480372) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:53:01 (1789480381) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 2.17094 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244637 s, 4.3 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:53:15 (1789480395) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.261287 s, 28.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:53:25 (1789480405) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:53:36 (1789480416) Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:54:52 (1789480492) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.456941 s, 25.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:55:38 (1789480538) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:57:02 (1789480622) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.86383 s, 8.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 15 09:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.456s user 0m0.000s sys 0m0.215s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.943s user 0m0.002s sys 0m0.168s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.490s user 0m0.001s sys 0m0.212s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1f18ffd800.ost_server_uuid in FULL state after 0 sec PASS 33b (107s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:58:49 (1789480729) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:58:53 (1789480733) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:58:56 (1789480736) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 09:59:00 (1789480740) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:59:08 (1789480748) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.523833 s, 24.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:59:20 (1789480760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630377 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349704 s, 30.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:59:28 (1789480768) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:59:37 (1789480777) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201266 s, 509 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000987639 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000988465 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00327252 s, 313 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:59:49 (1789480789) create test files with size 15759435 .. ae741f00dbc603a5e8b3fed74cc47e24 - f2df49c7b48532d9d7204deea653edef - 3e2c4dbea712de5c33554b38df3ee80e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 10:00:14 (1789480814) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:00:34 (1789480834) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:00:41 (1789480841) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199272 s, 5.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027903 s, 37.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:00:51 (1789480851) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163125 s, 12.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.625066 s, 6.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.379067 s, 5.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 10:01:10 (1789480870) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:01:13 (1789480873) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:01:16 (1789480876) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366643 s, 28.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 10:01:25 (1789480885) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:01:29 (1789480889) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:01:40 (1789480900) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.298154 s, 35.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:01:53 (1789480913) llite.lustre-ffff9e1f2074a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.373183 s, 28.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9e1f0f116000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:02:09 (1789480929) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:02:48 (1789480968) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:03:24 (1789481004) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:03:27 (1789481007) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.315003 s, 10.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.122843 s, 25.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb7d8951d (nosync,stale) Mirror 2: 0xb7d8951d (nosync,stale) ** verify mirror 2 doesn't change original checksum: dcc2ad049b52d10cf5dc352defcc4d15 - mirror 1 checksum: 51d5877f6335d49b811caaba0a6619b2 - mirror 2 checksum: dcc2ad049b52d10cf5dc352defcc4d15 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T10:03:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T10:03:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: dcc2ad049b52d10cf5dc352defcc4d15 - mirror 1 checksum: 51d5877f6335d49b811caaba0a6619b2 - mirror 2 checksum: 51d5877f6335d49b811caaba0a6619b2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T10:03:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T10:03:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:03:40 (1789481020) Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 10:03:47 (1789481027) shut down facet: mds1 facet_host: oleg209-server facet_failover_host: oleg209-server Failover mds1 to oleg209-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 10:04:09 (1789481049) targets are mounted 10:04:09 (1789481049) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363108 s, 28.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:04:30 (1789481070) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:04:34 (1789481074) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:04:37 (1789481077) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00514883 s, 3.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:04:44 (1789481084) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:04:48 (1789481088) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00744155 s, 550 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.127321 s, 32.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:04:56 (1789481096) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:05:19 (1789481119) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:05:49 (1789481149) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:06:12 (1789481172) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:06:36 (1789481196) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 10:06:43 (1789481203) mirror create pid 54355 mirror split pid 54357 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54355 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54357 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 10:07:50 (1789481270) Seed set to 1272 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1789481272.774650 write 0x001050 thru 0x006a7e (0x5a2f bytes) 000006 1789481272.783142 write 0x00b789 thru 0x019f80 (0xe7f8 bytes) 000009 1789481272.788096 mapwrite 0x038d6a thru 0x03ffff (0x7296 bytes) 000010 1789481272.894755 mapwrite 0x021ebe thru 0x02735c (0x549f bytes) 000011 1789481272.993223 mapwrite 0x02e50f thru 0x03c9c9 (0xe4bb bytes) truncating to largest ever: 0x178e9 000013 1789481273.067417 trunc from 0x040000 to 0x0178e8 (0x28717 bytes) 000014 1789481273.090950 read 0x00df4e thru 0x016891 (0x8944 bytes) 000015 1789481273.092053 mapwrite 0x0191bc thru 0x02289c (0x96e1 bytes) 000016 1789481273.179597 MIRROR_EXTEND to 2 mirrors 000017 1789481273.744999 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000018 1789481273.890130 punch from 0x003685 to 0x00a476 (0x6df2 bytes) 000019 1789481273.910358 read 0x000c3d thru 0x0072c2 (0x6686 bytes) 000020 1789481273.923707 MIRROR_EXTEND to 2 mirrors 000021 1789481274.659908 mapwrite 0x012eb6 thru 0x01cd78 (0x9ec3 bytes) 000022 1789481274.833898 MIRROR_RESYNC 2 mirrors 000023 1789481275.532299 punch from 0x010891 to 0x0205f3 (0xfd63 bytes) 000024 1789481275.611585 mapwrite 0x005d70 thru 0x009b9b (0x3e2c bytes) 000025 1789481275.703082 mapread 0x019fe8 thru 0x01d574 (0x358d bytes) 000026 1789481275.711736 read 0x010741 thru 0x019653 (0x8f13 bytes) 000029 1789481275.719455 mapread 0x01e093 thru 0x01fd07 (0x1c75 bytes) 000033 1789481275.724417 punch from 0x018b1b to 0x018c18 (0xfe bytes) 000034 1789481275.737543 read 0x01992c thru 0x02289c (0x8f71 bytes) 000035 1789481275.747611 write 0x029810 thru 0x033ed8 (0xa6c9 bytes) 000037 1789481275.752099 trunc from 0x033ed9 to 0x010967 (0x23571 bytes) 000038 1789481275.782866 read 0x0069c3 thru 0x010967 (0x9fa5 bytes) 000040 1789481275.785841 mapread 0x00c46e thru 0x010967 (0x44fa bytes) 000041 1789481275.790962 MIRROR_RESYNC 2 mirrors 000042 1789481276.604947 MIRROR_RESYNC 2 mirrors 000043 1789481276.969119 write 0x037215 thru 0x03a384 (0x3170 bytes) 000044 1789481277.041903 punch from 0x018ef2 to 0x02222f (0x933e bytes) 000045 1789481277.056401 trunc from 0x03a385 to 0x0158a4 (0x24ae0 bytes) 000047 1789481277.079742 punch from 0x005df7 to 0x00b024 (0x522e bytes) 000049 1789481277.087427 punch from 0x00e5ee to 0x0158a4 (0x72b7 bytes) 000051 1789481277.093515 punch from 0x012267 to 0x0158a4 (0x363e bytes) 000053 1789481277.099899 punch from 0x00c934 to 0x011b4d (0x521a bytes) 000054 1789481277.104961 mapread 0x011c31 thru 0x0158a4 (0x3c74 bytes) 000056 1789481277.130073 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000059 1789481277.284315 write 0x005300 thru 0x00dd3a (0x8a3b bytes) 000062 1789481277.301061 read 0x002895 thru 0x0107d2 (0xdf3e bytes) 000063 1789481277.309374 read 0x00e22c thru 0x0158a4 (0x7679 bytes) 000065 1789481277.318234 mapread 0x002d43 thru 0x012c59 (0xff17 bytes) 000066 1789481277.319891 mapread 0x00b2bb thru 0x0158a4 (0xa5ea bytes) 000067 1789481277.321127 mapread 0x00e1fd thru 0x0158a4 (0x76a8 bytes) 000068 1789481277.322506 trunc from 0x0158a5 to 0x002073 (0x13831 bytes) 000070 1789481277.334244 read 0x0006a0 thru 0x002073 (0x19d4 bytes) 000071 1789481277.334812 mapwrite 0x00787b thru 0x010f29 (0x96af bytes) 000073 1789481277.397461 punch from 0x005ec2 to 0x00d9a2 (0x7ae1 bytes) 000075 1789481277.402046 mapwrite 0x01b068 thru 0x0232ac (0x8245 bytes) 000076 1789481277.465724 write 0x03bf8e thru 0x03ffff (0x4072 bytes) 000077 1789481277.467710 punch from 0x03c399 to 0x03ffff (0x3c67 bytes) 000078 1789481277.479267 read 0x031258 thru 0x03ce04 (0xbbad bytes) 000079 1789481277.490697 punch from 0x00d74e to 0x01c7cf (0xf082 bytes) 000080 1789481277.496257 read 0x00bd51 thru 0x016ca8 (0xaf58 bytes) truncating to largest ever: 0x392b8 000082 1789481277.507547 trunc from 0x040000 to 0x0392b7 (0x6d48 bytes) 000083 1789481277.523588 mapwrite 0x015d98 thru 0x01c88e (0x6af7 bytes) 000084 1789481277.556899 mapread 0x0117a6 thru 0x01f0ab (0xd906 bytes) 000085 1789481277.558519 mapread 0x0321b3 thru 0x032f30 (0xd7e bytes) 000087 1789481277.560768 write 0x016faa thru 0x01b112 (0x4169 bytes) 000088 1789481277.562022 read 0x00ffbd thru 0x0181bf (0x8203 bytes) 000089 1789481277.563145 mapread 0x01ae05 thru 0x024e86 (0xa082 bytes) 000090 1789481277.563991 write 0x03adcb thru 0x03e0fb (0x3331 bytes) 000091 1789481277.568718 MIRROR_EXTEND to 2 mirrors 000092 1789481278.342460 mapread 0x003f52 thru 0x00c3ef (0x849e bytes) 000093 1789481278.369477 read 0x01d65e thru 0x01dd96 (0x739 bytes) 000094 1789481278.377163 read 0x035741 thru 0x03e0fb (0x89bb bytes) truncating to largest ever: 0x3cb3d 000098 1789481278.389167 trunc from 0x03e0fc to 0x03cb3c (0x15bf bytes) 000099 1789481278.454224 read 0x029c09 thru 0x0302e2 (0x66da bytes) 000100 1789481278.462806 punch from 0x009932 to 0x01419a (0xa869 bytes) 000101 1789481278.467126 trunc from 0x03cb3d to 0x02a8fb (0x12241 bytes) 000104 1789481278.496866 mapwrite 0x03f6b3 thru 0x03ffff (0x94d bytes) 000105 1789481278.575146 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000106 1789481278.760653 mapwrite 0x000ab2 thru 0x00356d (0x2abc bytes) 000107 1789481278.822695 write 0x026e22 thru 0x02c745 (0x5924 bytes) 000108 1789481278.833084 write 0x033084 thru 0x03ffff (0xcf7c bytes) 000110 1789481278.846382 mapwrite 0x02ee2b thru 0x03e2ff (0xf4d5 bytes) 000111 1789481278.902807 write 0x014cad thru 0x01fc17 (0xaf6b bytes) 000117 1789481278.917344 trunc from 0x040000 to 0x02dad5 (0x1252a bytes) 000118 1789481278.933815 punch from 0x000464 to 0x005741 (0x52de bytes) 000119 1789481278.950542 punch from 0x0224ac to 0x02b908 (0x945d bytes) 000120 1789481278.955588 trunc from 0x02dad6 to 0x015b81 (0x17f54 bytes) 000121 1789481278.973391 mapwrite 0x02233e thru 0x02933b (0x6ffe bytes) 000122 1789481279.076116 trunc from 0x02933c to 0x00e97c (0x1a9bf bytes) 000123 1789481279.098171 mapread 0x0051a0 thru 0x006075 (0xed6 bytes) 000126 1789481279.110252 trunc from 0x00e97d to 0x014b16 (0x619a bytes) 000127 1789481279.127702 trunc from 0x014b17 to 0x037f1e (0x23408 bytes) 000129 1789481279.145058 read 0x00fef8 thru 0x01ac1e (0xad27 bytes) 000130 1789481279.156419 punch from 0x02a4c6 to 0x02f72e (0x5269 bytes) 000131 1789481279.160729 mapread 0x00a1d7 thru 0x017026 (0xce50 bytes) 000132 1789481279.162385 mapread 0x02d095 thru 0x0353ae (0x831a bytes) 000133 1789481279.170731 mapwrite 0x02b564 thru 0x0367d9 (0xb276 bytes) 000134 1789481279.233823 mapread 0x035f0c thru 0x036ba4 (0xc99 bytes) 000136 1789481279.235951 mapwrite 0x0283dd thru 0x02b802 (0x3426 bytes) 000137 1789481279.274795 trunc from 0x037f1f to 0x0108be (0x27660 bytes) 000139 1789481279.300255 mapread 0x00f79c thru 0x0108be (0x1123 bytes) 000140 1789481279.301241 mapread 0x00db63 thru 0x0108be (0x2d5c bytes) 000142 1789481279.307854 MIRROR_EXTEND to 2 mirrors 000143 1789481280.045294 write 0x007810 thru 0x00deec (0x66dd bytes) 000144 1789481280.115494 mapwrite 0x0181f2 thru 0x01ee6d (0x6c7c bytes) 000145 1789481280.237112 punch from 0x008f7c to 0x010172 (0x71f7 bytes) 000146 1789481280.242713 read 0x01e71f thru 0x01ee6d (0x74f bytes) 000147 1789481280.243388 punch from 0x01994c to 0x01ee6d (0x5522 bytes) 000149 1789481280.249449 mapread 0x00b96c thru 0x01674e (0xade3 bytes) 000150 1789481280.263032 mapwrite 0x0394b9 thru 0x03dbe3 (0x472b bytes) 000151 1789481280.332962 mapwrite 0x02e218 thru 0x03105a (0x2e43 bytes) 000152 1789481280.425716 write 0x02bded thru 0x030294 (0x44a8 bytes) 000153 1789481280.427807 read 0x014f5f thru 0x019e63 (0x4f05 bytes) 000154 1789481280.449516 trunc from 0x03dbe4 to 0x010b66 (0x2d07d bytes) 000156 1789481280.482988 write 0x02395a thru 0x028467 (0x4b0e bytes) 000157 1789481280.484388 mapwrite 0x000d22 thru 0x007c79 (0x6f58 bytes) 000158 1789481280.586429 read 0x0104af thru 0x015765 (0x52b7 bytes) 000159 1789481280.610830 mapwrite 0x026d37 thru 0x02a0f4 (0x33be bytes) 000161 1789481280.687406 read 0x0278e7 thru 0x029050 (0x176a bytes) 000164 1789481280.707082 punch from 0x00978a to 0x0194de (0xfd55 bytes) 000165 1789481280.717331 mapread 0x00ddc1 thru 0x019666 (0xb8a6 bytes) 000168 1789481280.732360 mapread 0x0281a1 thru 0x02a0f4 (0x1f54 bytes) 000169 1789481280.734091 mapread 0x002ab4 thru 0x010b39 (0xe086 bytes) 000170 1789481280.736540 trunc from 0x02a0f5 to 0x00742d (0x22cc7 bytes) 000171 1789481280.760939 read 0x0031ec thru 0x00742d (0x4242 bytes) 000172 1789481280.761623 mapread 0x000a14 thru 0x002a2f (0x201c bytes) 000173 1789481280.762506 read 0x004311 thru 0x00742d (0x311d bytes) 000176 1789481280.766781 mapwrite 0x01727f thru 0x026082 (0xee04 bytes) 000177 1789481280.866649 mapread 0x00a4da thru 0x018505 (0xe02c bytes) 000178 1789481280.869385 read 0x018834 thru 0x0212e9 (0x8ab6 bytes) truncating to largest ever: 0x3ed24 000179 1789481280.870880 trunc from 0x026083 to 0x03ed23 (0x18ca1 bytes) 000180 1789481280.892320 mapwrite 0x01297e thru 0x01e760 (0xbde3 bytes) 000182 1789481280.934921 mapread 0x0013a0 thru 0x00ba99 (0xa6fa bytes) 000183 1789481280.936565 punch from 0x038dc4 to 0x03ed23 (0x5f60 bytes) 000185 1789481280.943133 mapread 0x02785c thru 0x0282ec (0xa91 bytes) 000186 1789481280.954399 punch from 0x02d630 to 0x038e10 (0xb7e1 bytes) 000188 1789481280.959320 punch from 0x0277bb to 0x02d2dd (0x5b23 bytes) 000189 1789481280.965024 trunc from 0x03ed24 to 0x0074df (0x37844 bytes) 000190 1789481280.987559 read 0x006bc4 thru 0x0074df (0x91c bytes) 000191 1789481280.997321 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000192 1789481281.180079 trunc from 0x0074e0 to 0x027c7e (0x2079f bytes) 000193 1789481281.209548 write 0x011d9f thru 0x012920 (0xb82 bytes) 000194 1789481281.247696 mapwrite 0x000706 thru 0x003354 (0x2c4f bytes) 000196 1789481281.321653 read 0x0263ab thru 0x0274c3 (0x1119 bytes) 000197 1789481281.326930 mapwrite 0x00262e thru 0x00b16d (0x8b40 bytes) 000198 1789481281.379417 write 0x021bcf thru 0x0261a3 (0x45d5 bytes) 000199 1789481281.385732 mapread 0x01b830 thru 0x027c7e (0xc44f bytes) 000200 1789481281.395483 mapread 0x027887 thru 0x027c7e (0x3f8 bytes) 000201 1789481281.397253 mapread 0x001f57 thru 0x00b960 (0x9a0a bytes) 000202 1789481281.398564 punch from 0x004abf to 0x010131 (0xb673 bytes) 000204 1789481281.412478 punch from 0x00e38e to 0x01cad0 (0xe743 bytes) 000206 1789481281.423286 MIRROR_EXTEND to 2 mirrors 000207 1789481282.055788 mapwrite 0x000c98 thru 0x0094da (0x8843 bytes) 000208 1789481282.270002 read 0x011b40 thru 0x0121bd (0x67e bytes) 000209 1789481282.277627 read 0x01a39d thru 0x026fc9 (0xcc2d bytes) 000210 1789481282.291855 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27c7f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c7f): Mirror 2: 0x6c2100ce (stale) CRC-32 checksum value for chunk [0, 0x27c7f): Mirror 1: 0xd672295f Mirror 3: 0xd672295f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c7f: skipped 000211 1789481283.151509 punch from 0x014e18 to 0x0204fc (0xb6e5 bytes) 000212 1789481283.246449 read 0x0239b2 thru 0x027c7e (0x42cd bytes) 000213 1789481283.259045 mapread 0x00ce06 thru 0x01bf74 (0xf16f bytes) 000214 1789481283.308317 write 0x0137d1 thru 0x018944 (0x5174 bytes) 000218 1789481283.326085 MIRROR_RESYNC 3 mirrors 000221 1789481284.174396 mapwrite 0x0338e8 thru 0x03ffff (0xc718 bytes) 000222 1789481284.400529 trunc from 0x040000 to 0x017bd7 (0x28428 bytes) 000223 1789481284.425698 mapwrite 0x0347ae thru 0x03ffff (0xb852 bytes) 000224 1789481284.527727 mapwrite 0x02450c thru 0x02a1f1 (0x5ce6 bytes) 000225 1789481284.621392 punch from 0x01e586 to 0x022288 (0x3d03 bytes) 000226 1789481284.633135 write 0x030fe3 thru 0x0324ac (0x14ca bytes) 000227 1789481284.635463 trunc from 0x040000 to 0x02f5b2 (0x10a4d bytes) 000228 1789481284.669018 trunc from 0x02f5b3 to 0x02d15b (0x2457 bytes) 000229 1789481284.704252 mapread 0x01a549 thru 0x01ef1f (0x49d7 bytes) 000230 1789481284.729368 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d15c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d15c): Mirror 2: 0x583c0a04 (stale) Mirror 3: 0x583c0a04 (stale) CRC-32 checksum value for chunk [0, 0x2d15c): Mirror 1: 0x910a3c98 Mirror 4: 0x910a3c98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d15c: skipped 000232 1789481285.882365 read 0x00f693 thru 0x01dd16 (0xe684 bytes) 000233 1789481285.892524 trunc from 0x02d15c to 0x0289e4 (0x4777 bytes) 000234 1789481285.976269 read 0x003041 thru 0x006192 (0x3152 bytes) 000237 1789481285.989109 read 0x0205bb thru 0x0289e4 (0x842a bytes) 000238 1789481285.990106 punch from 0x007ede to 0x0149bd (0xcae0 bytes) truncating to largest ever: 0x3f11c 000239 1789481285.996423 trunc from 0x0289e5 to 0x03f11b (0x16737 bytes) 000240 1789481286.025317 trunc from 0x03f11c to 0x025b5e (0x195bd bytes) 000242 1789481286.058869 MIRROR_RESYNC 4 mirrors 000245 1789481286.894247 write 0x02a6d1 thru 0x02ae84 (0x7b4 bytes) 000247 1789481286.948297 punch from 0x0288b0 to 0x02ae84 (0x25d5 bytes) 000248 1789481286.963841 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ae85): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae85): Mirror 2: 0x478fabd8 (stale) Mirror 3: 0x478fabd8 (stale) Mirror 4: 0x478fabd8 (stale) CRC-32 checksum value for chunk [0, 0x2ae85): Mirror 1: 0x478fabd8 Mirror 5: 0x478fabd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae85: skipped 000249 1789481287.757136 mapread 0x006e8a thru 0x00ce93 (0x600a bytes) 000250 1789481287.785578 mapwrite 0x0111a5 thru 0x0166c8 (0x5524 bytes) 000251 1789481287.949627 read 0x029e85 thru 0x02a173 (0x2ef bytes) 000252 1789481287.956794 trunc from 0x02ae85 to 0x0167a8 (0x146dc bytes) 000253 1789481287.985112 trunc from 0x0167a9 to 0x02fa42 (0x1929a bytes) 000254 1789481288.003840 read 0x008d63 thru 0x00a860 (0x1afe bytes) 000255 1789481288.014503 mapwrite 0x00ee32 thru 0x01ddc3 (0xef92 bytes) 000256 1789481288.067521 write 0x021327 thru 0x02cd9a (0xba74 bytes) 000257 1789481288.069358 trunc from 0x02fa43 to 0x01302b (0x1ca17 bytes) 000258 1789481288.084806 punch from 0x0040ab to 0x00490e (0x864 bytes) 000259 1789481288.089956 mapread 0x012af4 thru 0x01302b (0x538 bytes) 000261 1789481288.096278 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1302c): Mirror 2: 0xd3df6e7b (stale) Mirror 4: 0xd3df6e7b (stale) Mirror 5: 0xd3df6e7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1302c): Mirror 2: 0xd3df6e7b (stale) Mirror 4: 0xd3df6e7b (stale) Mirror 5: 0xd3df6e7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1302c: skipped 000262 1789481288.452147 write 0x012824 thru 0x013476 (0xc53 bytes) 000263 1789481288.469175 punch from 0x00141d to 0x01049b (0xf07f bytes) 000264 1789481288.479570 trunc from 0x013477 to 0x03be65 (0x289ef bytes) 000265 1789481288.503316 write 0x03a7aa thru 0x03ffff (0x5856 bytes) 000268 1789481288.513731 mapread 0x01b73c thru 0x01cf37 (0x17fc bytes) 000269 1789481288.525765 read 0x020f8f thru 0x02a8e7 (0x9959 bytes) 000271 1789481288.536281 mapread 0x00df25 thru 0x01a106 (0xc1e2 bytes) 000273 1789481288.559983 read 0x026fdf thru 0x02b508 (0x452a bytes) 000274 1789481288.572837 trunc from 0x040000 to 0x00636e (0x39c91 bytes) 000275 1789481288.592865 read 0x006164 thru 0x00636e (0x20b bytes) 000276 1789481288.593453 punch from 0x003347 to 0x00636e (0x3028 bytes) 000277 1789481288.600629 trunc from 0x00636f to 0x017bd9 (0x1186b bytes) 000278 1789481288.620824 mapread 0x00180c thru 0x002da7 (0x159c bytes) 000282 1789481288.631707 MIRROR_RESYNC 4 mirrors 000284 1789481289.583257 mapread 0x005aa3 thru 0x00d607 (0x7b65 bytes) 000287 1789481289.606975 write 0x00b73a thru 0x012383 (0x6c4a bytes) 000288 1789481289.683817 read 0x010e7a thru 0x017bd9 (0x6d60 bytes) 000289 1789481289.695605 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17bda): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17bda): Mirror 2: 0x389eb8c7 (stale) Mirror 4: 0x389eb8c7 (stale) Mirror 5: 0x389eb8c7 (stale) CRC-32 checksum value for chunk [0, 0x17bda): Mirror 1: 0x6b63ab79 Mirror 6: 0x6b63ab79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17bda: skipped 000290 1789481290.771313 MIRROR_RESYNC 5 mirrors 000291 1789481291.628339 write 0x007fac thru 0x0163a3 (0xe3f8 bytes) 000292 1789481291.694783 trunc from 0x017bda to 0x035126 (0x1d54d bytes) 000293 1789481291.731448 mapread 0x02c1cf thru 0x02f49e (0x32d0 bytes) 000294 1789481291.766872 mapread 0x031205 thru 0x035126 (0x3f22 bytes) 000295 1789481291.772314 punch from 0x028a7a to 0x02a78f (0x1d16 bytes) 000296 1789481291.832962 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35127): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35127): Mirror 2: 0x90d025e5 (stale) Mirror 4: 0x90d025e5 (stale) Mirror 5: 0x90d025e5 (stale) Mirror 6: 0x90d025e5 (stale) CRC-32 checksum value for chunk [0, 0x35127): Mirror 1: 0xc998f1f8 Mirror 7: 0xc998f1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35127: skipped 000297 1789481293.090595 punch from 0x032250 to 0x035126 (0x2ed7 bytes) 000298 1789481293.201817 MIRROR_RESYNC 6 mirrors 000299 1789481294.514363 mapread 0x01969c thru 0x02954b (0xfeb0 bytes) 000301 1789481294.545611 read 0x000dbb thru 0x010186 (0xf3cc bytes) 000304 1789481294.554527 read 0x0210fe thru 0x02b4ed (0xa3f0 bytes) 000306 1789481294.555226 punch from 0x01ef59 to 0x0281ce (0x9276 bytes) 000307 1789481294.633902 punch from 0x006721 to 0x012764 (0xc044 bytes) 000308 1789481294.642405 read 0x01f718 thru 0x022297 (0x2b80 bytes) 000310 1789481294.655322 write 0x027eda thru 0x02914d (0x1274 bytes) 000311 1789481294.656408 read 0x0347f6 thru 0x035126 (0x931 bytes) 000312 1789481294.657502 read 0x02e722 thru 0x0328e4 (0x41c3 bytes) 000314 1789481294.662459 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35127): Mirror 2: 0xc998f1f8 (stale) Mirror 4: 0xc998f1f8 (stale) Mirror 5: 0xc998f1f8 (stale) Mirror 7: 0xc998f1f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35127): Mirror 2: 0xc998f1f8 (stale) Mirror 4: 0xc998f1f8 (stale) Mirror 5: 0xc998f1f8 (stale) Mirror 7: 0xc998f1f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35127: skipped 000317 1789481295.016605 mapread 0x02b754 thru 0x035126 (0x99d3 bytes) 000320 1789481295.037185 trunc from 0x035127 to 0x01f357 (0x15dcf bytes) 000321 1789481295.057915 punch from 0x009e96 to 0x00c5a1 (0x270c bytes) 000324 1789481295.063478 write 0x00b462 thru 0x01397a (0x8519 bytes) 000325 1789481295.073809 trunc from 0x01f358 to 0x001ce9 (0x1d66e bytes) 000326 1789481295.093803 write 0x00b853 thru 0x014c31 (0x93df bytes) 000331 1789481295.098628 mapread 0x00895c thru 0x0124a5 (0x9b4a bytes) 000332 1789481295.107492 punch from 0x006ab9 to 0x011074 (0xa5bc bytes) 000333 1789481295.122510 mapwrite 0x014ddf thru 0x015e28 (0x104a bytes) 000335 1789481295.182992 trunc from 0x015e29 to 0x02b5f5 (0x157cd bytes) 000337 1789481295.205418 write 0x00be4c thru 0x014f71 (0x9126 bytes) 000339 1789481295.212184 read 0x016af0 thru 0x0232d5 (0xc7e6 bytes) 000340 1789481295.220596 write 0x0206bd thru 0x022069 (0x19ad bytes) 000341 1789481295.230116 MIRROR_RESYNC 5 mirrors 000343 1789481296.090266 trunc from 0x02b5f6 to 0x026abd (0x4b38 bytes) 000344 1789481296.192424 read 0x01be63 thru 0x0259f6 (0x9b94 bytes) 000347 1789481296.201694 mapread 0x00cb5e thru 0x00f50b (0x29ae bytes) 000348 1789481296.218103 punch from 0x014472 to 0x0237ad (0xf33c bytes) 000349 1789481296.223544 read 0x0255b0 thru 0x026abd (0x150e bytes) 000352 1789481296.234595 write 0x005d5f thru 0x008cdc (0x2f7e bytes) 000353 1789481296.235600 mapwrite 0x011fec thru 0x01895c (0x6971 bytes) 000354 1789481296.282350 trunc from 0x026abe to 0x00ced0 (0x19bed bytes) 000355 1789481296.300903 punch from 0x00c44c to 0x00ced0 (0xa85 bytes) 000357 1789481296.312693 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xced1): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xced1): Mirror 2: 0x58436d6c (stale) Mirror 4: 0x58436d6c (stale) Mirror 5: 0x58436d6c (stale) Mirror 7: 0x58436d6c (stale) CRC-32 checksum value for chunk [0, 0xced1): Mirror 1: 0xb2b6d2a8 Mirror 8: 0xb2b6d2a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xced1: skipped 000358 1789481296.959178 trunc from 0x00ced1 to 0x03570a (0x2883a bytes) 000359 1789481297.070558 read 0x01f353 thru 0x02ca4a (0xd6f8 bytes) 000360 1789481297.081697 write 0x01b10c thru 0x02a02f (0xef24 bytes) 000361 1789481297.091921 punch from 0x02b581 to 0x02c367 (0xde7 bytes) 000362 1789481297.113502 MIRROR_RESYNC 6 mirrors 000363 1789481298.073282 write 0x00c637 thru 0x014f1e (0x88e8 bytes) 000364 1789481298.153575 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3570b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3570b): Mirror 2: 0xe5347ecc (stale) Mirror 4: 0xe5347ecc (stale) Mirror 5: 0xe5347ecc (stale) Mirror 7: 0xe5347ecc (stale) Mirror 8: 0xe5347ecc (stale) CRC-32 checksum value for chunk [0, 0x3570b): Mirror 1: 0x8ce44af8 Mirror 9: 0x8ce44af8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3570b: skipped 000366 1789481299.508430 read 0x022007 thru 0x028a8b (0x6a85 bytes) 000367 1789481299.529347 read 0x0203f9 thru 0x02375d (0x3365 bytes) 000369 1789481299.565488 MIRROR_RESYNC 7 mirrors 000371 1789481300.915343 mapwrite 0x034672 thru 0x03b223 (0x6bb2 bytes) 000377 1789481301.091404 punch from 0x00dbaf to 0x015791 (0x7be3 bytes) 000378 1789481301.100881 mapwrite 0x02c078 thru 0x037156 (0xb0df bytes) 000379 1789481301.152069 read 0x028043 thru 0x037cca (0xfc88 bytes) 000383 1789481301.164227 trunc from 0x03b224 to 0x038123 (0x3100 bytes) 000385 1789481301.213549 punch from 0x010271 to 0x01475e (0x44ee bytes) 000387 1789481301.230192 mapwrite 0x003a3a thru 0x00b63b (0x7c02 bytes) 000389 1789481301.328136 mapwrite 0x0309de thru 0x03ffff (0xf622 bytes) 000390 1789481301.411132 punch from 0x002a4a to 0x011979 (0xef30 bytes) 000391 1789481301.424044 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd98b35d9 (stale) Mirror 4: 0xd98b35d9 (stale) Mirror 5: 0xd98b35d9 (stale) Mirror 7: 0xd98b35d9 (stale) Mirror 9: 0xd98b35d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd98b35d9 (stale) Mirror 4: 0xd98b35d9 (stale) Mirror 5: 0xd98b35d9 (stale) Mirror 7: 0xd98b35d9 (stale) Mirror 9: 0xd98b35d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000393 1789481301.931308 read 0x0332e0 thru 0x03eb95 (0xb8b6 bytes) 000394 1789481301.941582 trunc from 0x040000 to 0x02da22 (0x125dd bytes) 000397 1789481301.962577 write 0x0075fa thru 0x00a8cd (0x32d4 bytes) 000399 1789481301.976032 mapwrite 0x03342c thru 0x03d0bf (0x9c94 bytes) 000401 1789481302.082135 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 2: 0x37356813 (stale) Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) Mirror 9: 0x37356813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 2: 0x37356813 (stale) Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) Mirror 9: 0x37356813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0c0: skipped 000403 1789481302.579056 mapwrite 0x00edea thru 0x0168e2 (0x7af9 bytes) 000404 1789481302.697207 mapread 0x01144b thru 0x01ef3a (0xdaf0 bytes) 000406 1789481302.709369 mapread 0x033804 thru 0x03653a (0x2d37 bytes) 000408 1789481302.720201 read 0x02db29 thru 0x03d0bf (0xf597 bytes) 000409 1789481302.727326 mapread 0x0343fd thru 0x03c149 (0x7d4d bytes) 000410 1789481302.729058 read 0x038feb thru 0x03d0bf (0x40d5 bytes) 000411 1789481302.729776 punch from 0x00491b to 0x00af39 (0x661f bytes) 000413 1789481302.737201 mapwrite 0x0122bf thru 0x013fd9 (0x1d1b bytes) 000416 1789481302.776244 mapread 0x000c67 thru 0x0062b5 (0x564f bytes) 000418 1789481302.791073 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) Mirror 9: 0x37356813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) Mirror 9: 0x37356813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0c0: skipped 000419 1789481303.185009 write 0x02f0e9 thru 0x0330b8 (0x3fd0 bytes) 000421 1789481303.201353 read 0x0157ee thru 0x01fe43 (0xa656 bytes) 000422 1789481303.222616 mapwrite 0x03255d thru 0x03727c (0x4d20 bytes) 000424 1789481303.324527 punch from 0x027424 to 0x03352e (0xc10b bytes) 000425 1789481303.331368 read 0x039734 thru 0x03d0bf (0x398c bytes) 000426 1789481303.332388 punch from 0x0110ef to 0x01e656 (0xd568 bytes) 000427 1789481303.348472 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 4: 0x37356813 (stale) Mirror 5: 0x37356813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0c0: skipped 000428 1789481303.884206 read 0x0373c4 thru 0x03d0bf (0x5cfc bytes) 000429 1789481303.892970 trunc from 0x03d0c0 to 0x032c01 (0xa4be bytes) 000433 1789481303.908338 trunc from 0x032c02 to 0x01524c (0x1d9b5 bytes) 000434 1789481303.922340 punch from 0x003b44 to 0x00cb9c (0x9059 bytes) 000436 1789481303.925959 read 0x00e341 thru 0x01524c (0x6f0c bytes) 000437 1789481303.932887 punch from 0x00d5fd to 0x01524c (0x7c50 bytes) 000438 1789481303.936001 mapread 0x003bd7 thru 0x004cb0 (0x10da bytes) 000439 1789481303.939604 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1524d): Mirror 5: 0x432e77e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1524d): Mirror 5: 0x432e77e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1524d: skipped 000440 1789481304.196062 read 0x00363c thru 0x0108b7 (0xd27c bytes) 000441 1789481304.206410 MIRROR_RESYNC 2 mirrors 000442 1789481305.029869 read 0x008f0d thru 0x0121ed (0x92e1 bytes) 000443 1789481305.044869 write 0x0246dd thru 0x0261c5 (0x1ae9 bytes) 000446 1789481305.125847 write 0x03e4f6 thru 0x03ffff (0x1b0a bytes) 000449 1789481305.129819 read 0x0087ba thru 0x00f187 (0x69ce bytes) 000451 1789481305.141301 trunc from 0x040000 to 0x02925c (0x16da3 bytes) 000452 1789481305.156980 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000455 1789481305.305819 trunc from 0x02925d to 0x0165dc (0x12c80 bytes) 000459 1789481305.329293 read 0x004332 thru 0x00fc14 (0xb8e3 bytes) 000460 1789481305.338444 trunc from 0x0165dd to 0x02a138 (0x13b5c bytes) 000462 1789481305.361027 MIRROR_EXTEND to 2 mirrors 000463 1789481306.179971 punch from 0x001e22 to 0x009bc5 (0x7da4 bytes) 000464 1789481306.361417 mapread 0x00f852 thru 0x0183c1 (0x8b70 bytes) 000465 1789481306.383808 trunc from 0x02a139 to 0x03df84 (0x13e4c bytes) 000466 1789481306.436800 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000467 1789481306.582256 write 0x011f28 thru 0x01c5a7 (0xa680 bytes) 000468 1789481306.598290 MIRROR_EXTEND to 2 mirrors 000470 1789481307.484311 mapwrite 0x01f617 thru 0x020dde (0x17c8 bytes) 000472 1789481307.613202 read 0x00c110 thru 0x01017b (0x406c bytes) 000473 1789481307.621823 read 0x022f39 thru 0x02da5e (0xab26 bytes) 000475 1789481307.647651 punch from 0x031905 to 0x0356e6 (0x3de2 bytes) 000476 1789481307.652962 mapwrite 0x030d46 thru 0x0397b2 (0x8a6d bytes) 000477 1789481307.718484 read 0x00e4aa thru 0x01bbe7 (0xd73e bytes) 000478 1789481307.719290 mapread 0x00eeb9 thru 0x015f58 (0x70a0 bytes) 000479 1789481307.720248 write 0x03be42 thru 0x03e796 (0x2955 bytes) 000480 1789481307.723249 trunc from 0x03e797 to 0x003107 (0x3b68f bytes) 000482 1789481307.748155 read 0x000128 thru 0x003107 (0x2fe0 bytes) 000483 1789481307.753630 write 0x02c503 thru 0x02e420 (0x1f1e bytes) 000485 1789481307.756148 write 0x00e759 thru 0x00ee22 (0x6ca bytes) 000486 1789481307.764869 punch from 0x025466 to 0x02e420 (0x8fbb bytes) 000487 1789481307.776844 read 0x010d00 thru 0x019607 (0x8908 bytes) 000488 1789481307.777842 mapread 0x0189cc thru 0x02208f (0x96c4 bytes) 000489 1789481307.779558 punch from 0x026eec to 0x02d79c (0x68b1 bytes) 000491 1789481307.785717 write 0x0017c6 thru 0x0093e8 (0x7c23 bytes) 000492 1789481307.788282 read 0x0198aa thru 0x01a90e (0x1065 bytes) 000494 1789481307.790123 read 0x00509a thru 0x00a78c (0x56f3 bytes) 000496 1789481307.792165 trunc from 0x02e421 to 0x020cf3 (0xd72d bytes) 000497 1789481307.815182 mapwrite 0x037325 thru 0x039882 (0x255e bytes) 000498 1789481307.886328 mapwrite 0x01fc5d thru 0x02ad22 (0xb0c6 bytes) 000499 1789481307.936279 write 0x00d648 thru 0x01049e (0x2e57 bytes) 000501 1789481307.937995 mapwrite 0x00be51 thru 0x0137de (0x798e bytes) 000502 1789481307.982138 trunc from 0x039883 to 0x0228c4 (0x16fbe bytes) 000505 1789481308.002328 punch from 0x00f5ee to 0x01f0d5 (0xfae8 bytes) 000506 1789481308.007792 mapwrite 0x00b189 thru 0x012810 (0x7688 bytes) 000507 1789481308.073830 mapwrite 0x01c309 thru 0x02b211 (0xef09 bytes) 000510 1789481308.212794 write 0x01490f thru 0x015cca (0x13bc bytes) 000512 1789481308.229865 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b212): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b212): Mirror 2: 0xa47636f0 (stale) CRC-32 checksum value for chunk [0, 0x2b212): Mirror 1: 0xca08a4f5 Mirror 3: 0xca08a4f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b212: skipped 000514 1789481309.032587 read 0x00b279 thru 0x0139bd (0x8745 bytes) 000515 1789481309.047864 write 0x02312e thru 0x02a2df (0x71b2 bytes) 000516 1789481309.128010 mapwrite 0x02599c thru 0x02746d (0x1ad2 bytes) 000517 1789481309.172953 write 0x007c8b thru 0x01047f (0x87f5 bytes) 000519 1789481309.183826 read 0x022294 thru 0x02b211 (0x8f7e bytes) 000520 1789481309.202978 write 0x03df7c thru 0x03ffff (0x2084 bytes) 000522 1789481309.204625 mapwrite 0x027ec1 thru 0x0344e5 (0xc625 bytes) 000523 1789481309.264516 trunc from 0x040000 to 0x03b318 (0x4ce7 bytes) 000524 1789481309.316061 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b319): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b319): Mirror 2: 0x6ef4ea05 (stale) Mirror 3: 0x3ecf319 (stale) CRC-32 checksum value for chunk [0, 0x3b319): Mirror 1: 0xeecb5e2 Mirror 4: 0xeecb5e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b319: skipped 000526 1789481310.110111 write 0x02490f thru 0x02c5af (0x7ca1 bytes) 000527 1789481310.188618 mapread 0x0223bf thru 0x031e72 (0xfab4 bytes) 000528 1789481310.207384 read 0x0085bf thru 0x01450f (0xbf51 bytes) 000530 1789481310.220277 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b319): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b319): Mirror 2: 0x6ef4ea05 (stale) Mirror 3: 0x3ecf319 (stale) Mirror 4: 0xeecb5e2 (stale) CRC-32 checksum value for chunk [0, 0x3b319): Mirror 1: 0x131b78c8 Mirror 5: 0x131b78c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b319: skipped 000531 1789481311.052817 read 0x02e520 thru 0x0341fb (0x5cdc bytes) 000532 1789481311.060652 punch from 0x033c33 to 0x03b318 (0x76e6 bytes) 000534 1789481311.122988 punch from 0x017253 to 0x01d0e8 (0x5e96 bytes) 000535 1789481311.128498 write 0x0040ad thru 0x006166 (0x20ba bytes) 000537 1789481311.138142 read 0x039af1 thru 0x03b318 (0x1828 bytes) 000538 1789481311.157047 read 0x02e62c thru 0x03b318 (0xcced bytes) 000540 1789481311.158612 mapwrite 0x03cfe7 thru 0x03ffff (0x3019 bytes) 000541 1789481311.263325 read 0x01dbf0 thru 0x0232ac (0x56bd bytes) 000544 1789481311.274459 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf0097d52 (stale) Mirror 3: 0xd7590a05 (stale) Mirror 5: 0xda281742 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf0097d52 (stale) Mirror 3: 0xd7590a05 (stale) Mirror 5: 0xda281742 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000546 1789481311.797121 mapread 0x01e933 thru 0x0281a0 (0x986e bytes) 000549 1789481311.818264 write 0x02a271 thru 0x02d5fa (0x338a bytes) 000550 1789481311.827236 mapread 0x036fbd thru 0x03737d (0x3c1 bytes) 000552 1789481311.838611 write 0x01d2dd thru 0x02bcc7 (0xe9eb bytes) 000554 1789481311.842026 mapread 0x0259d8 thru 0x032613 (0xcc3c bytes) 000557 1789481311.850039 mapread 0x02069e thru 0x02942e (0x8d91 bytes) 000558 1789481311.851469 trunc from 0x040000 to 0x025e54 (0x1a1ab bytes) 000560 1789481311.868782 write 0x02ff10 thru 0x03716a (0x725b bytes) 000561 1789481311.875187 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: 0x16af90fb (stale) Mirror 3: 0x43496703 (stale) Mirror 5: 0x6901cd22 (stale) CRC-32 checksum value for chunk [0, 0x3716b): Mirror 1: 0x2e31b1ea Mirror 6: 0x2e31b1ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3716b: skipped 000564 1789481312.987445 write 0x01476e thru 0x01d4f5 (0x8d88 bytes) 000565 1789481313.073952 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: 0x16af90fb (stale) Mirror 3: 0x43496703 (stale) Mirror 5: 0x6901cd22 (stale) Mirror 6: 0x2e31b1ea (stale) CRC-32 checksum value for chunk [0, 0x3716b): Mirror 1: 0xeceb0410 Mirror 7: 0xeceb0410 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3716b: skipped 000567 1789481314.292883 mapread 0x00c69c thru 0x01283c (0x61a1 bytes) 000570 1789481314.325145 read 0x00f118 thru 0x0166f9 (0x75e2 bytes) 000571 1789481314.326270 mapread 0x028b48 thru 0x035fe6 (0xd49f bytes) 000572 1789481314.355132 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: 0x16af90fb (stale) Mirror 3: 0x43496703 (stale) Mirror 5: 0x6901cd22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3716b): Mirror 2: 0x16af90fb (stale) Mirror 3: 0x43496703 (stale) Mirror 5: 0x6901cd22 (stale) CRC-32 checksum value for chunk [0, 0x3716b): Mirror 1: 0xeceb0410 Mirror 7: 0xeceb0410 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3716b: skipped 000573 1789481314.973625 mapwrite 0x02eb4b thru 0x03b47e (0xc934 bytes) 000574 1789481315.260567 write 0x0249f1 thru 0x02f7a7 (0xadb7 bytes) 000576 1789481315.269375 mapwrite 0x003865 thru 0x01257f (0xed1b bytes) 000577 1789481315.370207 punch from 0x015360 to 0x024830 (0xf4d1 bytes) 000579 1789481315.386276 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: 0x8ec7996c (stale) Mirror 3: 0xd93bf6ea (stale) Mirror 7: 0x6d8d8bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: 0x8ec7996c (stale) Mirror 3: 0xd93bf6ea (stale) Mirror 7: 0x6d8d8bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b47f: skipped 000580 1789481315.784981 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: 0x8ec7996c (stale) Mirror 3: 0xd93bf6ea (stale) Mirror 7: 0x6d8d8bc (stale) CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 1: 0x4bd77c0a Mirror 8: 0x4bd77c0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b47f: skipped 000581 1789481316.668295 read 0x00a9e3 thru 0x014fc5 (0xa5e3 bytes) 000582 1789481316.682471 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 2: 0x8ec7996c (stale) Mirror 3: 0xd93bf6ea (stale) Mirror 7: 0x6d8d8bc (stale) CRC-32 checksum value for chunk [0, 0x3b47f): Mirror 1: 0x4bd77c0a Mirror 8: 0x4bd77c0a Mirror 9: 0x4bd77c0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b47f: skipped 000583 1789481317.508534 punch from 0x008dbe to 0x00df53 (0x5196 bytes) 000584 1789481317.651356 trunc from 0x03b47f to 0x035b79 (0x5905 bytes) 000585 1789481317.675058 trunc from 0x035b7a to 0x03da1c (0x7ea3 bytes) 000587 1789481317.698429 trunc from 0x03da1d to 0x0130fe (0x2a91e bytes) 000589 1789481317.721302 mapread 0x00c7a4 thru 0x0130fe (0x695b bytes) 000590 1789481317.740426 write 0x011a4d thru 0x019f6d (0x8521 bytes) 000591 1789481317.746268 MIRROR_RESYNC 6 mirrors 000592 1789481318.838428 mapread 0x0025ca thru 0x00f8cf (0xd306 bytes) 000593 1789481318.867455 read 0x004a2c thru 0x011c8d (0xd262 bytes) 000594 1789481318.872447 mapread 0x008c8a thru 0x014f5d (0xc2d4 bytes) 000595 1789481318.887803 MIRROR_EXTEND to 7 mirrors 000596 1789481319.632177 mapwrite 0x03b9e1 thru 0x03ffff (0x461f bytes) 000597 1789481319.774122 mapread 0x01d2c1 thru 0x02254c (0x528c bytes) 000598 1789481319.785410 punch from 0x01ae78 to 0x01e999 (0x3b22 bytes) 000599 1789481319.796928 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x10f7c9a5 (stale) Mirror 7: 0x10f7c9a5 (stale) Mirror 8: 0x10f7c9a5 (stale) Mirror 9: 0x10f7c9a5 (stale) Mirror 10: 0x10f7c9a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x10f7c9a5 (stale) Mirror 7: 0x10f7c9a5 (stale) Mirror 8: 0x10f7c9a5 (stale) Mirror 9: 0x10f7c9a5 (stale) Mirror 10: 0x10f7c9a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1789481320.191088 mapwrite 0x01ce57 thru 0x01e396 (0x1540 bytes) 000603 1789481320.260068 read 0x027d1e thru 0x035448 (0xd72b bytes) 000604 1789481320.271649 mapwrite 0x003356 thru 0x00cb3a (0x97e5 bytes) 000605 1789481320.344554 trunc from 0x040000 to 0x022bd2 (0x1d42d bytes) 000607 1789481320.363144 write 0x02f410 thru 0x0383f2 (0x8fe3 bytes) 000608 1789481320.364719 trunc from 0x0383f3 to 0x0294ed (0xef05 bytes) 000609 1789481320.392505 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x294ee): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x294ee): Mirror 3: 0x8e6216a4 (stale) Mirror 7: 0x8e6216a4 (stale) Mirror 8: 0x8e6216a4 (stale) Mirror 9: 0x8e6216a4 (stale) Mirror 10: 0x8e6216a4 (stale) CRC-32 checksum value for chunk [0, 0x294ee): Mirror 1: 0x150b57f0 Mirror 11: 0x150b57f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x294ee: skipped 000610 1789481321.388447 read 0x00a1a5 thru 0x01142c (0x7288 bytes) 000611 1789481321.403185 read 0x017408 thru 0x0249b4 (0xd5ad bytes) 000612 1789481321.423195 trunc from 0x0294ee to 0x01a6fe (0xedef bytes) 000613 1789481321.552655 write 0x019f05 thru 0x027945 (0xda41 bytes) 000614 1789481321.566377 punch from 0x01223f to 0x01f800 (0xd5c2 bytes) 000615 1789481321.579906 mapread 0x00698b thru 0x013160 (0xc7d6 bytes) 000616 1789481321.660457 mapread 0x01fc90 thru 0x023312 (0x3683 bytes) 000619 1789481321.675560 mapwrite 0x017670 thru 0x01fccc (0x865d bytes) 000620 1789481321.788414 punch from 0x0149db to 0x023a75 (0xf09b bytes) 000621 1789481321.794653 trunc from 0x027946 to 0x02a8ee (0x2fa9 bytes) 000622 1789481321.826123 trunc from 0x02a8ef to 0x03cec6 (0x125d8 bytes) 000623 1789481321.853317 punch from 0x00aea9 to 0x011c7c (0x6dd4 bytes) 000624 1789481321.861970 read 0x00fb4d thru 0x01ad5d (0xb211 bytes) 000625 1789481321.874984 mapread 0x022c4e thru 0x0320b2 (0xf465 bytes) 000628 1789481321.879018 mapwrite 0x0038cb thru 0x003cae (0x3e4 bytes) 000629 1789481321.915641 mapread 0x01eef3 thru 0x028e16 (0x9f24 bytes) 000631 1789481321.918312 mapread 0x006c41 thru 0x016303 (0xf6c3 bytes) 000632 1789481321.919533 mapwrite 0x02dd84 thru 0x03bfbf (0xe23c bytes) 000633 1789481321.978078 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cec7): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cec7): Mirror 3: 0xdbc15a20 (stale) Mirror 7: 0xdbc15a20 (stale) Mirror 8: 0xdbc15a20 (stale) Mirror 9: 0xdbc15a20 (stale) Mirror 10: 0xdbc15a20 (stale) Mirror 11: 0xcb287fe (stale) CRC-32 checksum value for chunk [0, 0x3cec7): Mirror 1: 0x564c4e86 Mirror 12: 0x564c4e86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cec7: skipped 000634 1789481322.988183 punch from 0x026bf0 to 0x033f3a (0xd34b bytes) 000636 1789481323.063314 read 0x0246c4 thru 0x02be93 (0x77d0 bytes) 000638 1789481323.068931 mapwrite 0x0182e8 thru 0x01efc8 (0x6ce1 bytes) 000639 1789481323.118061 read 0x01ea7d thru 0x025e65 (0x73e9 bytes) 000641 1789481323.130718 read 0x023d6a thru 0x02581c (0x1ab3 bytes) 000642 1789481323.131619 write 0x0223ce thru 0x028d12 (0x6945 bytes) 000644 1789481323.134328 mapread 0x0398f8 thru 0x03cec6 (0x35cf bytes) 000646 1789481323.137190 trunc from 0x03cec7 to 0x020820 (0x1c6a6 bytes) 000647 1789481323.163993 punch from 0x01ab13 to 0x020820 (0x5d0e bytes) 000648 1789481323.170307 mapwrite 0x019816 thru 0x023802 (0x9fed bytes) 000649 1789481323.243177 trunc from 0x023803 to 0x008ca5 (0x1ab5d bytes) 000650 1789481323.271449 mapwrite 0x0251d9 thru 0x03239c (0xd1c4 bytes) 000651 1789481323.370823 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3239d): Mirror 3: 0x995c5c97 (stale) Mirror 7: 0x995c5c97 (stale) Mirror 8: 0x995c5c97 (stale) Mirror 9: 0x995c5c97 (stale) Mirror 10: 0x995c5c97 (stale) Mirror 11: 0xbeb28d15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3239d): Mirror 3: 0x995c5c97 (stale) Mirror 7: 0x995c5c97 (stale) Mirror 8: 0x995c5c97 (stale) Mirror 9: 0x995c5c97 (stale) Mirror 10: 0x995c5c97 (stale) Mirror 11: 0xbeb28d15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3239d: skipped 000652 1789481323.894364 MIRROR_RESYNC 7 mirrors 000653 1789481325.807368 trunc from 0x03239d to 0x01869b (0x19d01 bytes) 000654 1789481325.879047 mapread 0x008309 thru 0x00e61a (0x6312 bytes) 000655 1789481325.903241 trunc from 0x01869c to 0x015c7d (0x2a1e bytes) 000656 1789481325.922944 mapwrite 0x01664c thru 0x017c50 (0x1605 bytes) 000657 1789481325.989410 punch from 0x00916d to 0x0170fe (0xdf92 bytes) 000660 1789481325.996435 mapwrite 0x01b971 thru 0x0242e4 (0x8974 bytes) 000661 1789481326.079559 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: 0xb0d37b29 (stale) Mirror 7: 0xb0d37b29 (stale) Mirror 8: 0xb0d37b29 (stale) Mirror 9: 0xb0d37b29 (stale) Mirror 10: 0xb0d37b29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: 0xb0d37b29 (stale) Mirror 7: 0xb0d37b29 (stale) Mirror 8: 0xb0d37b29 (stale) Mirror 9: 0xb0d37b29 (stale) Mirror 10: 0xb0d37b29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242e5: skipped 000662 1789481326.513966 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: 0xb0d37b29 (stale) Mirror 7: 0xb0d37b29 (stale) Mirror 8: 0xb0d37b29 (stale) Mirror 9: 0xb0d37b29 (stale) Mirror 10: 0xb0d37b29 (stale) CRC-32 checksum value for chunk [0, 0x242e5): Mirror 1: 0x8a5e29d4 Mirror 11: 0x8a5e29d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242e5: skipped 000663 1789481327.499027 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242e5): Mirror 3: 0xb0d37b29 (stale) Mirror 7: 0xb0d37b29 (stale) Mirror 8: 0xb0d37b29 (stale) Mirror 9: 0xb0d37b29 (stale) Mirror 10: 0xb0d37b29 (stale) CRC-32 checksum value for chunk [0, 0x242e5): Mirror 1: 0x8a5e29d4 Mirror 11: 0x8a5e29d4 Mirror 12: 0x8a5e29d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242e5: skipped 000665 1789481328.517432 mapwrite 0x01a87b thru 0x022c0d (0x8393 bytes) 000666 1789481328.694434 trunc from 0x0242e5 to 0x03530b (0x11027 bytes) 000667 1789481328.733332 trunc from 0x03530c to 0x0358d8 (0x5cd bytes) 000668 1789481328.755586 trunc from 0x0358d9 to 0x03466a (0x126e bytes) 000669 1789481328.773353 mapread 0x02ea46 thru 0x03466a (0x5c25 bytes) 000673 1789481328.790208 write 0x017129 thru 0x01a1b1 (0x3089 bytes) 000674 1789481328.795910 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3466b): Mirror 3: 0x6f431ae2 (stale) Mirror 7: 0x6f431ae2 (stale) Mirror 8: 0x6f431ae2 (stale) Mirror 9: 0x6f431ae2 (stale) Mirror 10: 0x6f431ae2 (stale) Mirror 11: 0xa199a89a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3466b): Mirror 3: 0x6f431ae2 (stale) Mirror 7: 0x6f431ae2 (stale) Mirror 8: 0x6f431ae2 (stale) Mirror 9: 0x6f431ae2 (stale) Mirror 10: 0x6f431ae2 (stale) Mirror 11: 0xa199a89a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3466b: skipped 000675 1789481329.481606 write 0x034c10 thru 0x0375ed (0x29de bytes) 000677 1789481329.489075 write 0x01ce9e thru 0x021498 (0x45fb bytes) 000678 1789481329.499827 read 0x01b557 thru 0x01bf19 (0x9c3 bytes) 000679 1789481329.506114 write 0x02e129 thru 0x03adf5 (0xcccd bytes) 000680 1789481329.511578 trunc from 0x03adf6 to 0x03afed (0x1f8 bytes) 000681 1789481329.531028 trunc from 0x03afee to 0x03cd8c (0x1d9f bytes) 000683 1789481329.553124 punch from 0x03874d to 0x03cd8c (0x4640 bytes) 000684 1789481329.578555 trunc from 0x03cd8d to 0x01784c (0x25540 bytes) 000685 1789481329.595922 mapread 0x002831 thru 0x00d501 (0xacd1 bytes) 000686 1789481329.617385 punch from 0x003451 to 0x00df8b (0xab3b bytes) 000688 1789481329.623299 mapread 0x00ee06 thru 0x015828 (0x6a23 bytes) 000692 1789481329.641791 MIRROR_RESYNC 7 mirrors 000693 1789481330.872942 punch from 0x0144d2 to 0x01784c (0x337b bytes) 000694 1789481330.953627 mapwrite 0x02d50a thru 0x034c4d (0x7744 bytes) 000695 1789481331.104349 read 0x0266d7 thru 0x034941 (0xe26b bytes) 000697 1789481331.111050 punch from 0x003fe7 to 0x00f2e6 (0xb300 bytes) 000698 1789481331.117998 trunc from 0x034c4e to 0x0051fa (0x2fa53 bytes) 000700 1789481331.136084 read 0x003104 thru 0x0051fa (0x20f7 bytes) 000701 1789481331.147378 write 0x02a521 thru 0x02ed1c (0x47fc bytes) 000703 1789481331.149023 write 0x003b37 thru 0x0059eb (0x1eb5 bytes) 000704 1789481331.149969 write 0x0312d8 thru 0x0339b5 (0x26de bytes) 000705 1789481331.152342 trunc from 0x0339b6 to 0x02ec7b (0x4d3a bytes) 000706 1789481331.175537 punch from 0x01da24 to 0x021bc8 (0x41a5 bytes) 000710 1789481331.190957 punch from 0x01b375 to 0x028c81 (0xd90d bytes) 000712 1789481331.197609 mapwrite 0x01e3ab thru 0x021195 (0x2deb bytes) 000713 1789481331.260808 mapread 0x008405 thru 0x017684 (0xf280 bytes) 000714 1789481331.261840 read 0x01c4fb thru 0x01d478 (0xf7e bytes) 000719 1789481331.264495 write 0x00c441 thru 0x019904 (0xd4c4 bytes) 000720 1789481331.265755 mapread 0x005937 thru 0x006469 (0xb33 bytes) 000721 1789481331.266999 write 0x02e160 thru 0x0321a2 (0x4043 bytes) 000722 1789481331.268257 mapwrite 0x008b15 thru 0x013e7f (0xb36b bytes) 000723 1789481331.333137 mapwrite 0x012854 thru 0x02114b (0xe8f8 bytes) 000724 1789481331.461734 write 0x029f56 thru 0x02eaf0 (0x4b9b bytes) 000725 1789481331.463354 write 0x030ca8 thru 0x039e8b (0x91e4 bytes) 000726 1789481331.466583 read 0x0395bf thru 0x039e8b (0x8cd bytes) 000727 1789481331.468808 read 0x014d67 thru 0x01ca71 (0x7d0b bytes) 000729 1789481331.476580 mapwrite 0x007e63 thru 0x016ce2 (0xee80 bytes) 000731 1789481331.572180 punch from 0x02474f to 0x031234 (0xcae6 bytes) 000732 1789481331.579924 punch from 0x025a82 to 0x02f525 (0x9aa4 bytes) 000733 1789481331.585559 mapread 0x018147 thru 0x020153 (0x800d bytes) 000734 1789481331.592426 MIRROR_RESYNC 7 mirrors 000735 1789481333.101978 punch from 0x0050b4 to 0x008cbf (0x3c0c bytes) 000736 1789481333.171062 trunc from 0x039e8c to 0x02a622 (0xf869 bytes) 000737 1789481333.193083 mapread 0x01f785 thru 0x02a273 (0xaaef bytes) 000740 1789481333.220833 mapwrite 0x014283 thru 0x01e8d2 (0xa650 bytes) 000743 1789481333.304541 mapwrite 0x00f21a thru 0x019386 (0xa16d bytes) 000745 1789481333.351975 read 0x004519 thru 0x008da6 (0x488e bytes) 000747 1789481333.362697 trunc from 0x02a623 to 0x039967 (0xf345 bytes) 000749 1789481333.380247 punch from 0x032f2f to 0x0366b8 (0x378a bytes) 000750 1789481333.385594 punch from 0x016722 to 0x023a5b (0xd33a bytes) 000753 1789481333.392631 trunc from 0x039968 to 0x00d63c (0x2c32b bytes) 000754 1789481333.413894 write 0x00708f thru 0x00ae9d (0x3e0f bytes) 000755 1789481333.420017 write 0x014ea9 thru 0x01a3d8 (0x5530 bytes) 000756 1789481333.421292 punch from 0x012d47 to 0x018d43 (0x5ffd bytes) 000757 1789481333.438586 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a3d9): Mirror 3: 0x28839335 (stale) Mirror 7: 0x28839335 (stale) Mirror 8: 0x28839335 (stale) Mirror 10: 0x28839335 (stale) Mirror 11: 0x28839335 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3d9): Mirror 3: 0x28839335 (stale) Mirror 7: 0x28839335 (stale) Mirror 8: 0x28839335 (stale) Mirror 10: 0x28839335 (stale) Mirror 11: 0x28839335 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3d9: skipped 000760 1789481333.828396 read 0x01226e thru 0x01237b (0x10e bytes) 000761 1789481333.836020 write 0x0377c4 thru 0x03ffff (0x883c bytes) 000762 1789481333.838379 write 0x02758d thru 0x028280 (0xcf4 bytes) 000763 1789481333.849514 punch from 0x035f50 to 0x039ceb (0x3d9c bytes) 000765 1789481333.866456 trunc from 0x040000 to 0x01c7e3 (0x2381c bytes) 000767 1789481333.892165 punch from 0x018afe to 0x01c7e3 (0x3ce6 bytes) 000769 1789481333.899199 mapwrite 0x03e1ca thru 0x03ffff (0x1e36 bytes) 000772 1789481333.977397 punch from 0x00f07c to 0x01b340 (0xc2c5 bytes) 000774 1789481333.985400 trunc from 0x040000 to 0x0022d6 (0x3dd29 bytes) 000776 1789481334.008244 trunc from 0x0022d7 to 0x0258a5 (0x235cf bytes) 000778 1789481334.031156 write 0x00f57f thru 0x01e577 (0xeff9 bytes) 000780 1789481334.043338 write 0x00cf65 thru 0x0169eb (0x9a87 bytes) 000781 1789481334.050108 trunc from 0x0258a6 to 0x0179d4 (0xded1 bytes) 000782 1789481334.070061 punch from 0x008035 to 0x00b14b (0x3117 bytes) 000783 1789481334.084430 punch from 0x010da0 to 0x0179d4 (0x6c35 bytes) 000784 1789481334.089205 read 0x007662 thru 0x00f0cc (0x7a6b bytes) 000785 1789481334.100919 mapwrite 0x03ba44 thru 0x03ffff (0x45bc bytes) 000786 1789481334.189507 mapread 0x03083d thru 0x035056 (0x481a bytes) 000788 1789481334.192033 write 0x00c08a thru 0x00d390 (0x1307 bytes) 000789 1789481334.193927 mapread 0x026da8 thru 0x03613d (0xf396 bytes) 000790 1789481334.195397 trunc from 0x040000 to 0x037967 (0x8698 bytes) 000792 1789481334.221347 MIRROR_RESYNC 6 mirrors 000793 1789481335.786948 MIRROR_RESYNC 6 mirrors 000794 1789481336.247081 MIRROR_RESYNC 6 mirrors 000795 1789481336.801743 mapread 0x003a86 thru 0x0094d5 (0x5a50 bytes) 000796 1789481336.821233 read 0x01d7b0 thru 0x02c11e (0xe96f bytes) 000797 1789481336.840923 trunc from 0x037968 to 0x021c56 (0x15d11 bytes) 000800 1789481336.936331 mapwrite 0x008963 thru 0x017816 (0xeeb4 bytes) 000802 1789481337.011550 punch from 0x0139d6 to 0x021c56 (0xe281 bytes) 000803 1789481337.018719 mapwrite 0x0286d9 thru 0x02c6de (0x4006 bytes) 000804 1789481337.090597 mapwrite 0x01c668 thru 0x0220ae (0x5a47 bytes) 000806 1789481337.155805 mapread 0x0279f2 thru 0x02c6de (0x4ced bytes) 000807 1789481337.156973 write 0x01a42d thru 0x02028d (0x5e61 bytes) 000808 1789481337.159040 write 0x0247d5 thru 0x0298bd (0x50e9 bytes) 000811 1789481337.162273 mapread 0x023aa9 thru 0x02c6de (0x8c36 bytes) 000812 1789481337.163891 read 0x009685 thru 0x00cbea (0x3566 bytes) 000813 1789481337.164705 write 0x035112 thru 0x037ba4 (0x2a93 bytes) 000814 1789481337.166732 write 0x01f53b thru 0x01f8ec (0x3b2 bytes) 000815 1789481337.167371 read 0x01bae1 thru 0x026d6d (0xb28d bytes) 000816 1789481337.168795 mapread 0x01ce18 thru 0x02312b (0x6314 bytes) 000820 1789481337.183305 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ba5): Mirror 3: 0x67b1aba1 (stale) Mirror 7: 0x67b1aba1 (stale) Mirror 8: 0x67b1aba1 (stale) Mirror 10: 0x67b1aba1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ba5): Mirror 3: 0x67b1aba1 (stale) Mirror 7: 0x67b1aba1 (stale) Mirror 8: 0x67b1aba1 (stale) Mirror 10: 0x67b1aba1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ba5: skipped 000821 1789481337.727309 punch from 0x025e77 to 0x02e997 (0x8b21 bytes) 000822 1789481337.748141 mapwrite 0x021295 thru 0x026e92 (0x5bfe bytes) 000823 1789481337.836601 mapwrite 0x034386 thru 0x03b558 (0x71d3 bytes) 000824 1789481337.923171 read 0x020ccb thru 0x029e1d (0x9153 bytes) 000825 1789481337.929240 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b559): Mirror 3: 0xf29f4b6a (stale) Mirror 7: 0xf29f4b6a (stale) Mirror 8: 0xf29f4b6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b559): Mirror 3: 0xf29f4b6a (stale) Mirror 7: 0xf29f4b6a (stale) Mirror 8: 0xf29f4b6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b559: skipped 000827 1789481338.326813 read 0x01844a thru 0x01d8c3 (0x547a bytes) 000828 1789481338.337327 trunc from 0x03b559 to 0x032456 (0x9102 bytes) 000829 1789481338.353915 write 0x02387b thru 0x02dbc8 (0xa34e bytes) 000830 1789481338.364107 read 0x001f5f thru 0x008240 (0x62e2 bytes) 000831 1789481338.375338 punch from 0x0014ed to 0x001544 (0x58 bytes) 000832 1789481338.397924 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32457): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32457): Mirror 3: 0xbb85e46c (stale) Mirror 7: 0xbb85e46c (stale) Mirror 8: 0xbb85e46c (stale) CRC-32 checksum value for chunk [0, 0x32457): Mirror 1: 0xe8524f0a Mirror 9: 0xe8524f0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32457: skipped 000833 1789481339.325272 read 0x0036dd thru 0x00fc8f (0xc5b3 bytes) 000837 1789481339.338336 write 0x016b2f thru 0x020453 (0x9925 bytes) 000838 1789481339.490875 punch from 0x01ae57 to 0x01ce7d (0x2027 bytes) 000839 1789481339.508479 trunc from 0x032457 to 0x03c848 (0xa3f2 bytes) 000841 1789481339.569968 mapwrite 0x002154 thru 0x0038ff (0x17ac bytes) 000842 1789481339.694522 trunc from 0x03c849 to 0x031715 (0xb133 bytes) 000843 1789481339.717560 write 0x03a6ef thru 0x03ffff (0x5911 bytes) 000844 1789481339.728727 trunc from 0x040000 to 0x036b20 (0x94df bytes) 000845 1789481339.758117 punch from 0x02c275 to 0x02d4e7 (0x1273 bytes) 000848 1789481339.763864 write 0x0149ce thru 0x0194cb (0x4afe bytes) 000849 1789481339.769320 read 0x0354d5 thru 0x036b20 (0x164c bytes) 000851 1789481339.785346 punch from 0x000189 to 0x0061ed (0x6065 bytes) 000852 1789481339.797976 punch from 0x009c31 to 0x01919f (0xf56f bytes) 000853 1789481339.802954 punch from 0x013cbd to 0x016bdf (0x2f23 bytes) 000854 1789481339.806740 mapread 0x006be4 thru 0x015e4a (0xf267 bytes) 000855 1789481339.874911 trunc from 0x036b21 to 0x03968d (0x2b6d bytes) 000856 1789481339.894960 punch from 0x036c85 to 0x03968d (0x2a09 bytes) 000858 1789481339.901182 punch from 0x0273c9 to 0x0341a0 (0xcdd8 bytes) 000859 1789481339.910499 write 0x010959 thru 0x019dea (0x9492 bytes) 000860 1789481339.934603 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3968e): Mirror 3: 0x7c6195b3 (stale) Mirror 8: 0x7c6195b3 (stale) Mirror 9: 0x7f1d02bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3968e): Mirror 3: 0x7c6195b3 (stale) Mirror 8: 0x7c6195b3 (stale) Mirror 9: 0x7f1d02bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3968e: skipped 000861 1789481340.345931 trunc from 0x03968e to 0x002f44 (0x36749 bytes) 000862 1789481340.367188 punch from 0x002534 to 0x002f44 (0xa11 bytes) 000863 1789481340.379139 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f45): Mirror 3: 0x788636c5 (stale) Mirror 9: 0x788636c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f45): Mirror 3: 0x788636c5 (stale) Mirror 9: 0x788636c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f45: skipped 000864 1789481340.699113 trunc from 0x002f45 to 0x0216f0 (0x1e7ac bytes) 000865 1789481340.732971 mapread 0x00b25f thru 0x011f0a (0x6cac bytes) 000866 1789481340.756311 read 0x0056c7 thru 0x010537 (0xae71 bytes) 000867 1789481340.761951 read 0x01551c thru 0x0216f0 (0xc1d5 bytes) 000868 1789481340.773032 mapwrite 0x030ae9 thru 0x03da3e (0xcf56 bytes) 000870 1789481340.886130 punch from 0x00dcd0 to 0x00e7c7 (0xaf8 bytes) 000871 1789481340.896232 mapread 0x023bb8 thru 0x02f6f8 (0xbb41 bytes) 000872 1789481340.902298 MIRROR_RESYNC 3 mirrors 000873 1789481342.194150 mapread 0x0392bc thru 0x03da3e (0x4783 bytes) 000874 1789481342.219396 punch from 0x002699 to 0x00360e (0xf76 bytes) 000875 1789481342.292159 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da3f): Mirror 9: 0x4c3853a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da3f): Mirror 9: 0x4c3853a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da3f: skipped 000877 1789481342.616358 write 0x001996 thru 0x00b8a6 (0x9f11 bytes) 000878 1789481342.631484 trunc from 0x03da3f to 0x0019fb (0x3c043 bytes) 000879 1789481342.659272 write 0x024bfb thru 0x0307fe (0xbc04 bytes) 000880 1789481342.664011 mapwrite 0x02372f thru 0x0288d8 (0x51aa bytes) 000881 1789481342.747074 trunc from 0x0307ff to 0x03b69e (0xaea0 bytes) 000882 1789481342.781233 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 9: 0x3e1fab0c (stale) CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 1: 0x380615cc Mirror 10: 0x380615cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b69f: skipped 000884 1789481343.999963 mapwrite 0x0096cc thru 0x017c2f (0xe564 bytes) 000885 1789481344.183085 trunc from 0x03b69f to 0x00f4ec (0x2c1b2 bytes) 000886 1789481344.207879 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf4ed): Mirror 10: 0x7b39f35a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf4ed): Mirror 10: 0x7b39f35a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf4ed: skipped 000887 1789481344.614253 trunc from 0x00f4ed to 0x00d70d (0x1ddf bytes) 000888 1789481344.662177 MIRROR_RESYNC 2 mirrors 000889 1789481345.514859 read 0x009b3b thru 0x00d70d (0x3bd3 bytes) 000891 1789481345.537157 MIRROR_EXTEND to 3 mirrors 000892 1789481346.562255 read 0x009fd0 thru 0x00d13f (0x3170 bytes) 000893 1789481346.575284 mapread 0x00a7c5 thru 0x00b3e0 (0xc1c bytes) 000894 1789481346.576690 trunc from 0x00d70e to 0x018032 (0xa925 bytes) 000895 1789481346.681818 mapwrite 0x028502 thru 0x0314b1 (0x8fb0 bytes) 000896 1789481346.911368 mapread 0x026975 thru 0x0314b1 (0xab3d bytes) 000897 1789481346.922492 trunc from 0x0314b2 to 0x02c7a7 (0x4d0a bytes) 000898 1789481346.951708 mapwrite 0x027c62 thru 0x02f583 (0x7922 bytes) 000901 1789481347.145254 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f584): Mirror 11: 0xabbd3826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f584): Mirror 11: 0xabbd3826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f584: skipped 000903 1789481347.445546 read 0x025b24 thru 0x02f583 (0x9a60 bytes) 000904 1789481347.454263 punch from 0x0055db to 0x00cb6f (0x7595 bytes) 000905 1789481347.460218 mapread 0x012740 thru 0x017d98 (0x5659 bytes) 000906 1789481347.479402 mapread 0x029570 thru 0x02f583 (0x6014 bytes) 000907 1789481347.480447 trunc from 0x02f584 to 0x02df78 (0x160b bytes) 000909 1789481347.502359 mapread 0x0036f9 thru 0x010835 (0xd13d bytes) 000910 1789481347.512104 write 0x029b94 thru 0x038bb7 (0xf024 bytes) 000911 1789481347.514089 write 0x0382d0 thru 0x0387f7 (0x528 bytes) 000914 1789481347.515979 read 0x01ce77 thru 0x01f40f (0x2599 bytes) 000915 1789481347.525515 write 0x02b323 thru 0x0319a3 (0x6681 bytes) 000916 1789481347.527244 mapwrite 0x03e308 thru 0x03ffff (0x1cf8 bytes) 000917 1789481347.593056 mapwrite 0x018b93 thru 0x0278ae (0xed1c bytes) 000918 1789481347.636930 punch from 0x03ce74 to 0x03ffff (0x318c bytes) 000919 1789481347.642225 trunc from 0x040000 to 0x00c992 (0x3366d bytes) 000920 1789481347.660186 read 0x0047dd thru 0x00a3c2 (0x5be6 bytes) 000921 1789481347.660731 read 0x0074f7 thru 0x008e79 (0x1983 bytes) 000923 1789481347.661736 mapwrite 0x023893 thru 0x031b6f (0xe2dd bytes) 000924 1789481347.740329 mapwrite 0x022d65 thru 0x02e25f (0xb4fb bytes) 000926 1789481347.802057 MIRROR_RESYNC 2 mirrors 000927 1789481348.745941 write 0x02a8d4 thru 0x033d19 (0x9446 bytes) 000929 1789481348.821290 trunc from 0x033d1a to 0x00f22b (0x24aee bytes) 000930 1789481348.848030 mapwrite 0x03f1a7 thru 0x03ffff (0xe59 bytes) 000931 1789481348.955187 punch from 0x0026c6 to 0x008b97 (0x64d2 bytes) 000932 1789481348.967128 trunc from 0x040000 to 0x005595 (0x3aa6a bytes) 000933 1789481349.009269 MIRROR_RESYNC 2 mirrors 000934 1789481349.664794 mapwrite 0x0150b4 thru 0x01d628 (0x8575 bytes) 000935 1789481349.881086 read 0x00f8c3 thru 0x01c3a0 (0xcade bytes) 000936 1789481349.890108 trunc from 0x01d629 to 0x0202dd (0x2cb5 bytes) 000937 1789481349.921152 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) 000940 1789481350.109722 mapwrite 0x016a24 thru 0x018225 (0x1802 bytes) 000945 1789481350.225411 MIRROR_EXTEND to 2 mirrors 000946 1789481351.092925 read 0x019d8e thru 0x0202dd (0x6550 bytes) 000947 1789481351.099852 read 0x00c5e0 thru 0x01b1e4 (0xec05 bytes) 000948 1789481351.109278 mapwrite 0x02e894 thru 0x0377fd (0x8f6a bytes) 000949 1789481351.251914 punch from 0x010430 to 0x013219 (0x2dea bytes) 000951 1789481351.261228 mapread 0x02b7a6 thru 0x0377fd (0xc058 bytes) 000952 1789481351.263556 punch from 0x030238 to 0x0377fd (0x75c6 bytes) 000953 1789481351.292744 mapread 0x02942d thru 0x034f31 (0xbb05 bytes) 000954 1789481351.350565 mapwrite 0x011ba1 thru 0x017ae9 (0x5f49 bytes) 000955 1789481351.488360 read 0x02fe35 thru 0x0377fd (0x79c9 bytes) 000957 1789481351.503067 read 0x024d5d thru 0x026053 (0x12f7 bytes) 000958 1789481351.503951 mapread 0x00e4d6 thru 0x0128cb (0x43f6 bytes) 000959 1789481351.506086 write 0x016856 thru 0x0189b0 (0x215b bytes) 000960 1789481351.508263 punch from 0x017df8 to 0x0253b9 (0xd5c2 bytes) 000961 1789481351.520345 trunc from 0x0377fe to 0x01b759 (0x1c0a4 bytes) 000962 1789481351.551129 mapread 0x012665 thru 0x0158f9 (0x3295 bytes) 000963 1789481351.554196 write 0x006abc thru 0x00bb3a (0x507f bytes) 000965 1789481351.558056 punch from 0x00dac1 to 0x017a15 (0x9f55 bytes) 000966 1789481351.573828 mapread 0x002826 thru 0x0103dc (0xdbb7 bytes) 000967 1789481351.584642 mapread 0x010f7c thru 0x01b759 (0xa7de bytes) 000968 1789481351.585613 mapwrite 0x0097bc thru 0x0120f5 (0x893a bytes) 000971 1789481351.649273 punch from 0x019726 to 0x01b759 (0x2034 bytes) 000974 1789481351.668138 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b75a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b75a): Mirror 2: 0x42aa2624 (stale) CRC-32 checksum value for chunk [0, 0x1b75a): Mirror 1: 0xa4c09300 Mirror 3: 0xa4c09300 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b75a: skipped 000975 1789481352.698757 read 0x01a6c8 thru 0x01b759 (0x1092 bytes) 000976 1789481352.705182 punch from 0x0064c7 to 0x0089ee (0x2528 bytes) 000977 1789481352.781474 mapwrite 0x0018cf thru 0x008030 (0x6762 bytes) 000978 1789481352.849737 punch from 0x0093dc to 0x0142a9 (0xaece bytes) 000979 1789481352.854197 punch from 0x00d333 to 0x011072 (0x3d40 bytes) 000980 1789481352.859441 write 0x01cd41 thru 0x021e89 (0x5149 bytes) 000981 1789481352.862238 write 0x025249 thru 0x02645d (0x1215 bytes) 000982 1789481352.864140 write 0x0259ed thru 0x02f5c5 (0x9bd9 bytes) truncating to largest ever: 0x3fccb 000983 1789481352.868443 trunc from 0x02f5c6 to 0x03fcca (0x10705 bytes) 000984 1789481352.892053 mapwrite 0x028570 thru 0x0330e6 (0xab77 bytes) 000987 1789481352.952270 mapwrite 0x02e5a9 thru 0x032842 (0x429a bytes) 000988 1789481352.987938 read 0x01f720 thru 0x02c32d (0xcc0e bytes) 000989 1789481352.999348 read 0x016a41 thru 0x01d837 (0x6df7 bytes) 000990 1789481353.000054 mapwrite 0x029fd8 thru 0x039cc3 (0xfcec bytes) 000991 1789481353.040563 mapwrite 0x03ede5 thru 0x03ffff (0x121b bytes) 000992 1789481353.101207 mapread 0x00f4b3 thru 0x01d490 (0xdfde bytes) 000998 1789481353.107905 mapread 0x031496 thru 0x0353f1 (0x3f5c bytes) 000999 1789481353.109119 trunc from 0x040000 to 0x0283aa (0x17c55 bytes) 001000 1789481353.130154 read 0x01c820 thru 0x0283aa (0xbb8b bytes) All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 10:09:18 (1789481358) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 10:09:21 (1789481361) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183251 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234572 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:09:30 (1789481370) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5797760 .. Extending file size to 7280704 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6726304 .. Extending file size to 6913088 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8258432 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7520320 .. Extending file size to 8857120 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8345472 .. Extending file size to 8556448 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8830176 .. Waiting 57476 57477 57478 57481 57484 failed 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:10:53 (1789481453) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8357696 .. Extending file size to 3128192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4460128 .. Extending file size to 4816160 .. Extending file size to 8596224 .. Extending file size to 8830912 .. Extending file size to 8370720 .. Extending file size to 8438976 .. Extending file size to 8697440 .. Extending file size to 8813920 .. Extending file size to 9233120 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58984 58985 58986 58990 58991 done 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:12:10 (1789481530) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123957 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.345376 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x26832fb9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:12:28 (1789481548) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0729248 s, 28.8 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 10:12:33 (1789481553) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x203:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:12:40 (1789481560) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198963 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:12:48 (1789481568) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157993 s, 13.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:12:55 (1789481575) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197728 s, 10.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:13:27 (1789481607) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08173 s, 12.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:14:20 (1789481660) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172141 s, 12.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:15:15 (1789481715) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. Creating new pool test_204f oleg209-server: Pool lustre.test_204f created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0966596 s, 21.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg209-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:16:38 (1789481798) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:16:50 (1789481810) PASS 205b (13s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:17:03 (1789481823) oleg209-server: Pool lustre.test_206 created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg209-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:17:30 (1789481850) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110197 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468238 s, 22.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6a6a886e Mirror 3: 0x6a6a886e PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:17:42 (1789481862) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:17:47 (1789481867) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:17:51 (1789481871) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.193849 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2320000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f40000 Read this (4 bytes): BBBB 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:18:29 (1789481909) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.034339 s, 239 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:18:36 (1789481916) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736519 s, 14.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x228 lmm_fid: [0x200000404:0x228:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x228 lmm_fid: [0x200000404:0x228:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834472 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 10:18:38.000000000 -0400 Modify: 2026-09-15 10:18:38.000000000 -0400 Change: 2026-09-15 10:18:40.000000000 -0400 Birth: 2026-09-15 10:18:39.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:18:45 (1789481925) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:18:48 (1789481928) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.433786 s, 94.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.102144 s, 40.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 15 10:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 15 10:18 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:18:57 (1789481937) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:19:06 (1789481946) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1820 sec =========== 10:19:17 (1789481957) === sanity-flr: start cleanup 10:19:19 (1789481959) === === sanity-flr: finish cleanup 10:19:26 (1789481966) ===