-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 08:21:16 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg424-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:21:37 (1787142097) === oleg424-client.virtnet: executing check_config_client /mnt/lustre oleg424-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg424-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b2c03827000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b2c03827000.idle_timeout=debug disable quota as required oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:22:15 (1787142135) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7633] == sanity-flr test 0a: lfs mirror create with -N option == 08:22:18 (1787142138) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:22:38 (1787142158) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:22:42 (1787142162) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:22:46 (1787142166) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:23:11 (1787142191) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:23:14 (1787142194) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:23:18 (1787142198) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b2c03827000.stats=clear osc.lustre-OST0001-osc-ffff8b2c03827000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:23:27 (1787142207) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:23:38 (1787142218) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:23:47 (1787142227) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:23:49 (1787142229) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:23:57 (1787142237) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 08:24:07 (1787142247) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:24:16 (1787142256) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:24:20 (1787142260) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:24:29 (1787142269) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:24:39 (1787142279) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:24:46 (1787142286) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.887917 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179456 s, 5.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:24:56 (1787142296) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.167715 s, 25.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:25:05 (1787142305) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:25:16 (1787142316) Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:26:29 (1787142389) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.339166 s, 21.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:27:13 (1787142433) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:28:43 (1787142523) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.1235 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 08:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.107s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.165s user 0m0.000s sys 0m0.192s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.493s user 0m0.002s sys 0m0.209s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b2c03827000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:30:20 (1787142620) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:30:23 (1787142623) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:30:27 (1787142627) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:30:31 (1787142631) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:30:39 (1787142639) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.748717 s, 16.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:30:53 (1787142653) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296987 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258844 s, 40.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:31:00 (1787142660) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:31:07 (1787142667) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101808 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182218 s, 562 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000963194 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00171668 s, 597 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:31:17 (1787142677) create test files with size 15747089 .. 08b104b297112a2c7d83dfd4c0c2648e - 8141c70679d9fe4298ad80bb11bf08aa - e016dd82b062cfe831714aded57eb90f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 08:31:43 (1787142703) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:32:00 (1787142720) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:32:08 (1787142728) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175122 s, 6.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258443 s, 40.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:32:17 (1787142737) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116477 s, 18.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.309362 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199869 s, 10.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 08:32:32 (1787142752) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:32:35 (1787142755) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:32:39 (1787142759) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342614 s, 30.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:32:48 (1787142768) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:32:51 (1787142771) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xce:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xce:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:33:02 (1787142782) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.183558 s, 57.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:33:14 (1787142794) llite.lustre-ffff8b2c06b0e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.278608 s, 37.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8b2c18b56800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:33:28 (1787142808) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:34:03 (1787142843) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:34:35 (1787142875) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:34:38 (1787142878) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256207 s, 12.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.226095 s, 13.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x604b591c (nosync,stale) Mirror 2: 0x604b591c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 87706f1541939a2d5ab43163b12d5f61 - mirror 1 checksum: 0a1d77126ad5d72d13df63c0126496f9 - mirror 2 checksum: 87706f1541939a2d5ab43163b12d5f61 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T08:34:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T08:34:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 87706f1541939a2d5ab43163b12d5f61 - mirror 1 checksum: 0a1d77126ad5d72d13df63c0126496f9 - mirror 2 checksum: 0a1d77126ad5d72d13df63c0126496f9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T08:34:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T08:34:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:34:51 (1787142891) Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 08:34:58 (1787142898) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 08:35:32 (1787142932) targets are mounted 08:35:32 (1787142932) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331212 s, 31.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:35:54 (1787142954) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:35:58 (1787142958) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:36:01 (1787142961) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00586317 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:36:09 (1787142969) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:36:13 (1787142973) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011773 s, 348 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0695291 s, 58.9 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:36:21 (1787142981) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:36:44 (1787143004) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:37:14 (1787143034) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:37:37 (1787143057) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:38:02 (1787143082) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 08:38:09 (1787143089) mirror create pid 54310 mirror split pid 54311 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54310 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54311 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:39:17 (1787143157) Seed set to 3159 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1309a 000002 1787143159.715507 trunc from 00000000 to 0x013099 (0x1309a bytes) 000003 1787143159.739881 trunc from 0x01309a to 0x00669e (0xc9fb bytes) 000004 1787143159.758317 read 0x004a7e thru 0x00669e (0x1c21 bytes) 000006 1787143159.765933 read 0x0005b8 thru 0x001c0a (0x1653 bytes) 000007 1787143159.772427 mapread 0x00257f thru 0x00669e (0x4120 bytes) 000008 1787143159.774841 write 0x02a22c thru 0x0306a3 (0x6478 bytes) 000009 1787143159.779656 MIRROR_EXTEND to 2 mirrors 000011 1787143160.889935 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000013 1787143161.211922 mapread 0x026643 thru 0x028a43 (0x2401 bytes) 000019 1787143161.251537 read 0x02f647 thru 0x0306a3 (0x105d bytes) 000021 1787143161.254874 write 0x0083e7 thru 0x00ecbb (0x68d5 bytes) 000022 1787143161.318460 mapread 0x013c69 thru 0x020200 (0xc598 bytes) 000023 1787143161.377841 mapwrite 0x02ee25 thru 0x039685 (0xa861 bytes) 000024 1787143161.504981 mapread 0x037043 thru 0x039685 (0x2643 bytes) 000025 1787143161.506008 mapread 0x032dfb thru 0x039685 (0x688b bytes) 000026 1787143161.519026 MIRROR_EXTEND to 2 mirrors 000027 1787143162.766893 write 0x009d66 thru 0x00ab82 (0xe1d bytes) 000028 1787143162.852382 read 0x000a18 thru 0x003a5b (0x3044 bytes) 000029 1787143162.870747 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000031 1787143163.076481 read 0x01ff7d thru 0x025e36 (0x5eba bytes) 000033 1787143163.091337 mapwrite 0x01d46c thru 0x020aac (0x3641 bytes) 000036 1787143163.168358 mapread 0x00296c thru 0x00af7b (0x8610 bytes) 000037 1787143163.185486 read 0x02dcaf thru 0x036b2c (0x8e7e bytes) 000041 1787143163.207610 read 0x01fa54 thru 0x02b840 (0xbded bytes) 000044 1787143163.215260 mapread 0x01b90f thru 0x021cb0 (0x63a2 bytes) 000045 1787143163.220180 mapread 0x003101 thru 0x01150c (0xe40c bytes) truncating to largest ever: 0x387dc 000046 1787143163.222872 trunc from 0x039686 to 0x0387db (0xeaa bytes) 000047 1787143163.253157 trunc from 0x0387dc to 0x0266b7 (0x12124 bytes) 000050 1787143163.275526 read 0x00caa8 thru 0x01669d (0x9bf6 bytes) 000051 1787143163.283292 MIRROR_EXTEND to 2 mirrors 000052 1787143164.246275 trunc from 0x0266b8 to 0x01a2a3 (0xc414 bytes) 000053 1787143164.501935 mapwrite 0x0318b6 thru 0x033d27 (0x2472 bytes) 000057 1787143164.776616 read 0x01612b thru 0x020d04 (0xabda bytes) 000058 1787143164.808086 MIRROR_RESYNC 2 mirrors 000059 1787143166.197979 read 0x029cb0 thru 0x031de8 (0x8139 bytes) 000061 1787143166.219089 trunc from 0x033d28 to 0x0130d7 (0x20c50 bytes) 000062 1787143166.343477 mapwrite 0x0030bb thru 0x0068a7 (0x37ed bytes) 000064 1787143166.483010 write 0x02d917 thru 0x03635a (0x8a44 bytes) 000065 1787143166.487216 write 0x01cbb8 thru 0x027ae1 (0xaf2a bytes) 000067 1787143166.523128 mapwrite 0x03f95f thru 0x03ffff (0x6a1 bytes) 000070 1787143166.663169 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1c290a7a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f42773f Mirror 3: 0x7f42773f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000071 1787143168.310923 write 0x03b406 thru 0x03ffff (0x4bfa bytes) 000073 1787143168.474468 mapwrite 0x02b929 thru 0x03879d (0xce75 bytes) 000074 1787143168.732897 mapwrite 0x023e68 thru 0x032c32 (0xedcb bytes) truncating to largest ever: 0x3f198 000076 1787143168.920344 trunc from 0x040000 to 0x03f197 (0xe68 bytes) 000077 1787143168.963921 write 0x007f29 thru 0x00c58e (0x4666 bytes) 000078 1787143169.000266 MIRROR_RESYNC 3 mirrors 000080 1787143170.601219 mapwrite 0x032db7 thru 0x03a023 (0x726d bytes) 000082 1787143170.861268 trunc from 0x03f198 to 0x00e2ee (0x30ea9 bytes) 000083 1787143170.892400 read 0x007b1c thru 0x00c022 (0x4507 bytes) 000084 1787143170.905217 mapread 0x0008ce thru 0x00bb47 (0xb27a bytes) 000087 1787143170.919369 mapwrite 0x00e1f8 thru 0x01167f (0x3488 bytes) 000089 1787143171.034856 mapwrite 0x0255bc thru 0x027237 (0x1c7c bytes) 000090 1787143171.149746 trunc from 0x027238 to 0x038f7e (0x11d47 bytes) 000091 1787143171.210365 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f7f): Mirror 3: 0xafc96925 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f7f): Mirror 3: 0xafc96925 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f7f: skipped 000092 1787143171.584261 mapwrite 0x00d4b2 thru 0x01b0b4 (0xdc03 bytes) 000093 1787143171.690320 trunc from 0x038f7f to 0x03d1ac (0x422e bytes) 000098 1787143171.714309 read 0x02eb6f thru 0x037b4b (0x8fdd bytes) 000100 1787143171.727596 read 0x005a41 thru 0x008575 (0x2b35 bytes) 000101 1787143171.739627 trunc from 0x03d1ad to 0x021a15 (0x1b797 bytes) 000102 1787143171.772787 mapread 0x017b5a thru 0x017d16 (0x1bd bytes) 000103 1787143171.774225 write 0x02acf1 thru 0x035475 (0xa785 bytes) 000106 1787143171.780192 read 0x00de4a thru 0x01b04d (0xd204 bytes) 000110 1787143171.782984 trunc from 0x035476 to 0x01d252 (0x18223 bytes) 000112 1787143171.804189 mapread 0x007ed4 thru 0x014f98 (0xd0c5 bytes) 000113 1787143171.806228 mapread 0x01a989 thru 0x01d252 (0x28ca bytes) 000115 1787143171.808340 write 0x0297df thru 0x035a69 (0xc28b bytes) 000116 1787143171.810533 write 0x01464e thru 0x01d804 (0x91b7 bytes) 000118 1787143171.813224 write 0x01da54 thru 0x023c91 (0x623e bytes) 000120 1787143171.827778 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 3: 0x7a973050 (stale) CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 1: 0x8a6272ee Mirror 4: 0x8a6272ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a6a: skipped 000121 1787143173.000976 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 3: 0x7a973050 (stale) CRC-32 checksum value for chunk [0, 0x35a6a): Mirror 1: 0x8a6272ee Mirror 4: 0x8a6272ee Mirror 5: 0x8a6272ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a6a: skipped 000123 1787143174.483761 write 0x024e94 thru 0x034952 (0xfabf bytes) 000125 1787143174.705279 read 0x029c02 thru 0x02b8eb (0x1cea bytes) 000126 1787143174.712656 MIRROR_RESYNC 4 mirrors 000127 1787143176.260622 read 0x00828b thru 0x013424 (0xb19a bytes) 000128 1787143176.271677 mapwrite 0x031889 thru 0x038c1c (0x7394 bytes) 000130 1787143176.456136 mapwrite 0x029ecd thru 0x02d428 (0x355c bytes) 000131 1787143176.527659 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: 0x86b5bc8e (stale) Mirror 4: 0x86b5bc8e (stale) Mirror 5: 0x86b5bc8e (stale) CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 1: 0x216b32b0 Mirror 6: 0x216b32b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1d: skipped 000133 1787143177.568384 read 0x00d358 thru 0x01a525 (0xd1ce bytes) 000135 1787143177.593179 write 0x01c90f thru 0x01d0f7 (0x7e9 bytes) 000136 1787143177.811340 write 0x021b23 thru 0x029e96 (0x8374 bytes) 000137 1787143177.853771 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: 0x86b5bc8e (stale) Mirror 4: 0x86b5bc8e (stale) Mirror 5: 0x86b5bc8e (stale) Mirror 6: 0x216b32b0 (stale) CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 1: 0xc2137a34 Mirror 7: 0xc2137a34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1d: skipped 000138 1787143179.702080 mapwrite 0x01b033 thru 0x027c5c (0xcc2a bytes) 000139 1787143180.088488 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 3: 0x86b5bc8e (stale) Mirror 4: 0x86b5bc8e (stale) Mirror 5: 0x86b5bc8e (stale) Mirror 6: 0x216b32b0 (stale) Mirror 7: 0xc2137a34 (stale) CRC-32 checksum value for chunk [0, 0x38c1d): Mirror 1: 0xee233909 Mirror 8: 0xee233909 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1d: skipped 000141 1787143182.025189 trunc from 0x038c1d to 0x00157a (0x376a2 bytes) 000143 1787143182.207093 mapread 0x000556 thru 0x00157a (0x1025 bytes) 000144 1787143182.234003 mapwrite 0x002e73 thru 0x0054db (0x2669 bytes) 000145 1787143182.435512 write 0x02d25a thru 0x03a263 (0xd00a bytes) 000146 1787143182.450138 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a264): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a264): Mirror 3: 0xc2f06841 (stale) Mirror 4: 0xc2f06841 (stale) Mirror 5: 0xc2f06841 (stale) Mirror 6: 0x667bd5d7 (stale) Mirror 7: 0x614d9a7 (stale) Mirror 8: 0x8bf65b77 (stale) CRC-32 checksum value for chunk [0, 0x3a264): Mirror 1: 0x7e14a7e3 Mirror 9: 0x7e14a7e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a264: skipped 000148 1787143184.393343 trunc from 0x03a264 to 0x0234d1 (0x16d92 bytes) 000150 1787143184.480902 mapwrite 0x00c839 thru 0x014ba4 (0x836c bytes) 000151 1787143184.623111 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x234d2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x234d2): Mirror 3: 0x1ce6c7c1 (stale) Mirror 4: 0x1ce6c7c1 (stale) Mirror 5: 0x1ce6c7c1 (stale) Mirror 6: 0x1ce6c7c1 (stale) Mirror 7: 0xa934f879 (stale) Mirror 8: 0x219c2488 (stale) Mirror 9: 0x13119050 (stale) CRC-32 checksum value for chunk [0, 0x234d2): Mirror 1: 0x8201a2ea Mirror 10: 0x8201a2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x234d2: skipped 000152 1787143186.335185 mapread 0x010a12 thru 0x0151b9 (0x47a8 bytes) 000156 1787143186.429294 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x234d2): Mirror 3: 0x1ce6c7c1 (stale) Mirror 4: 0x1ce6c7c1 (stale) Mirror 5: 0x1ce6c7c1 (stale) Mirror 7: 0xa934f879 (stale) Mirror 8: 0x219c2488 (stale) Mirror 9: 0x13119050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x234d2): Mirror 3: 0x1ce6c7c1 (stale) Mirror 4: 0x1ce6c7c1 (stale) Mirror 5: 0x1ce6c7c1 (stale) Mirror 7: 0xa934f879 (stale) Mirror 8: 0x219c2488 (stale) Mirror 9: 0x13119050 (stale) CRC-32 checksum value for chunk [0, 0x234d2): Mirror 1: 0x8201a2ea Mirror 10: 0x8201a2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x234d2: skipped 000159 1787143187.786070 write 0x00f65a thru 0x013d7b (0x4722 bytes) 000162 1787143187.949666 trunc from 0x0234d2 to 0x021819 (0x1cb8 bytes) 000164 1787143187.993106 mapwrite 0x008cde thru 0x0184e5 (0xf808 bytes) 000167 1787143188.142876 read 0x01be80 thru 0x021819 (0x599a bytes) 000168 1787143188.155152 read 0x01d09a thru 0x021819 (0x4780 bytes) 000169 1787143188.156349 write 0x03f447 thru 0x03ffff (0xbb9 bytes) 000170 1787143188.157440 write 0x0163eb thru 0x01e61b (0x8231 bytes) 000171 1787143188.159380 write 0x01ca9e thru 0x0212da (0x483d bytes) 000172 1787143188.166433 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xef236e5d (stale) Mirror 5: 0xef236e5d (stale) Mirror 7: 0x2d20e1ff (stale) Mirror 8: 0x421edd73 (stale) Mirror 9: 0xb93ad4a1 (stale) Mirror 10: 0x57d7adaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xef236e5d (stale) Mirror 5: 0xef236e5d (stale) Mirror 7: 0x2d20e1ff (stale) Mirror 8: 0x421edd73 (stale) Mirror 9: 0xb93ad4a1 (stale) Mirror 10: 0x57d7adaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1787143188.745170 trunc from 0x040000 to 0x01f17e (0x20e81 bytes) 000175 1787143188.778241 read 0x019473 thru 0x01d854 (0x43e2 bytes) 000179 1787143188.798900 write 0x007465 thru 0x00f79f (0x833b bytes) 000180 1787143188.813757 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1f17f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f17f): Mirror 4: 0x36323126 (stale) Mirror 5: 0x36323126 (stale) Mirror 7: 0xd59c95b4 (stale) Mirror 8: 0xc92a98f0 (stale) Mirror 9: 0xdfa65ed7 (stale) Mirror 10: 0x9712eb37 (stale) CRC-32 checksum value for chunk [0, 0x1f17f): Mirror 1: 0xbeb07c21 Mirror 11: 0xbeb07c21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f17f: skipped 000183 1787143190.316167 mapwrite 0x00a23e thru 0x018bf9 (0xe9bc bytes) 000184 1787143190.618664 MIRROR_RESYNC 8 mirrors 000187 1787143192.413048 read 0x01bf1c thru 0x01cb08 (0xbed bytes) 000188 1787143192.419531 read 0x002ea7 thru 0x012787 (0xf8e1 bytes) 000189 1787143192.429939 read 0x01712a thru 0x01f17e (0x8055 bytes) 000190 1787143192.435274 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000191 1787143192.853095 mapwrite 0x01b42b thru 0x028ba3 (0xd779 bytes) 000192 1787143193.034127 mapread 0x00b49c thru 0x01ad1f (0xf884 bytes) 000193 1787143193.048027 mapwrite 0x035258 thru 0x036ba2 (0x194b bytes) 000195 1787143193.131000 mapread 0x0142fd thru 0x01843b (0x413f bytes) 000196 1787143193.138474 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36ba3): Mirror 5: 0x7bb82dbd (stale) Mirror 7: 0x7bb82dbd (stale) Mirror 8: 0x7bb82dbd (stale) Mirror 9: 0x7bb82dbd (stale) Mirror 11: 0x7bb82dbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ba3): Mirror 5: 0x7bb82dbd (stale) Mirror 7: 0x7bb82dbd (stale) Mirror 8: 0x7bb82dbd (stale) Mirror 9: 0x7bb82dbd (stale) Mirror 11: 0x7bb82dbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ba3: skipped 000198 1787143193.669658 read 0x00ec46 thru 0x00fb5e (0xf19 bytes) 000199 1787143193.688173 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36ba3): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ba3): Mirror 5: 0x7bb82dbd (stale) Mirror 7: 0x7bb82dbd (stale) Mirror 8: 0x7bb82dbd (stale) Mirror 9: 0x7bb82dbd (stale) Mirror 11: 0x7bb82dbd (stale) CRC-32 checksum value for chunk [0, 0x36ba3): Mirror 1: 0x680acf50 Mirror 12: 0x680acf50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ba3: skipped 000200 1787143194.850836 trunc from 0x036ba3 to 0x02a5f6 (0xc5ac bytes) 000201 1787143194.940161 mapwrite 0x01e430 thru 0x022fa8 (0x4b79 bytes) 000202 1787143195.007814 mapread 0x01698d thru 0x01b7be (0x4e32 bytes) 000203 1787143195.019081 trunc from 0x02a5f7 to 0x0141b5 (0x16441 bytes) 000206 1787143195.051136 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x141b6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141b6): Mirror 5: 0x8e746216 (stale) Mirror 7: 0x8e746216 (stale) Mirror 8: 0x8e746216 (stale) Mirror 9: 0x8e746216 (stale) Mirror 11: 0x8e746216 (stale) Mirror 12: 0x8e746216 (stale) CRC-32 checksum value for chunk [0, 0x141b6): Mirror 1: 0x8e746216 Mirror 13: 0x8e746216 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141b6: skipped 000207 1787143196.131760 read 0x00652e thru 0x0141b5 (0xdc88 bytes) 000208 1787143196.166336 mapread 0x0004f4 thru 0x000d34 (0x841 bytes) 000209 1787143196.189908 mapwrite 0x033d07 thru 0x03a731 (0x6a2b bytes) 000210 1787143196.482467 read 0x00c2f9 thru 0x015129 (0x8e31 bytes) 000211 1787143196.493257 write 0x006b05 thru 0x00cc01 (0x60fd bytes) 000213 1787143196.495845 read 0x0081b8 thru 0x010e0a (0x8c53 bytes) 000215 1787143196.497933 mapwrite 0x02f4d1 thru 0x02f6d7 (0x207 bytes) 000217 1787143196.543727 trunc from 0x03a732 to 0x03d0ed (0x29bc bytes) 000218 1787143196.566419 write 0x03d74a thru 0x03ffff (0x28b6 bytes) 000221 1787143196.575746 trunc from 0x040000 to 0x02ca28 (0x135d7 bytes) 000222 1787143196.614384 trunc from 0x02ca29 to 0x02d1ce (0x7a6 bytes) 000223 1787143196.644893 mapread 0x02693c thru 0x02990b (0x2fd0 bytes) 000224 1787143196.646189 mapread 0x0156bd thru 0x023030 (0xd974 bytes) 000227 1787143196.648256 trunc from 0x02d1cf to 0x01de77 (0xf357 bytes) 000228 1787143196.673916 read 0x013f3f thru 0x014875 (0x937 bytes) 000229 1787143196.674921 mapread 0x019e75 thru 0x01de77 (0x4003 bytes) 000235 1787143196.678756 write 0x008a64 thru 0x00a16b (0x1708 bytes) 000236 1787143196.679655 read 0x00f1f7 thru 0x01bb49 (0xc953 bytes) 000240 1787143196.682111 mapread 0x01d507 thru 0x01de77 (0x971 bytes) 000241 1787143196.683314 mapwrite 0x00c798 thru 0x00f47b (0x2ce4 bytes) 000242 1787143196.750074 trunc from 0x01de78 to 0x03a94d (0x1cad6 bytes) 000243 1787143196.788028 write 0x000443 thru 0x00ba6b (0xb629 bytes) 000245 1787143196.791178 write 0x01ee23 thru 0x028d65 (0x9f43 bytes) 000246 1787143196.823338 mapwrite 0x007e75 thru 0x0111d6 (0x9362 bytes) 000247 1787143196.908710 MIRROR_RESYNC 8 mirrors 000248 1787143198.856853 write 0x0310d9 thru 0x035469 (0x4391 bytes) 000249 1787143198.927635 mapwrite 0x00fee9 thru 0x01e0c3 (0xe1db bytes) 000250 1787143199.019252 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a94e): Mirror 5: 0x8183c7e0 (stale) Mirror 7: 0x8183c7e0 (stale) Mirror 8: 0x8183c7e0 (stale) Mirror 9: 0x8183c7e0 (stale) Mirror 11: 0x8183c7e0 (stale) Mirror 13: 0x8183c7e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a94e): Mirror 5: 0x8183c7e0 (stale) Mirror 7: 0x8183c7e0 (stale) Mirror 8: 0x8183c7e0 (stale) Mirror 9: 0x8183c7e0 (stale) Mirror 11: 0x8183c7e0 (stale) Mirror 13: 0x8183c7e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a94e: skipped 000251 1787143199.609562 trunc from 0x03a94e to 0x025e16 (0x14b37 bytes) 000252 1787143199.644055 mapread 0x013be1 thru 0x019db5 (0x61d5 bytes) 000253 1787143199.687249 mapwrite 0x026ced thru 0x02c552 (0x5866 bytes) 000255 1787143199.806331 mapread 0x011675 thru 0x01b547 (0x9ed3 bytes) 000256 1787143199.808614 trunc from 0x02c553 to 0x017f02 (0x14650 bytes) 000257 1787143199.843969 mapwrite 0x01d8c1 thru 0x027196 (0x98d6 bytes) 000258 1787143200.012313 mapwrite 0x0344b7 thru 0x03e556 (0xa0a0 bytes) 000259 1787143200.169809 mapwrite 0x02e2f9 thru 0x038c54 (0xa95c bytes) 000261 1787143200.341126 mapwrite 0x039e5c thru 0x03ffff (0x61a4 bytes) 000263 1787143200.500518 mapread 0x0369a3 thru 0x03ffff (0x965d bytes) 000266 1787143200.521806 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9a66fc1c (stale) Mirror 7: 0x9a66fc1c (stale) Mirror 9: 0x9a66fc1c (stale) Mirror 11: 0x9a66fc1c (stale) Mirror 13: 0x9a66fc1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9a66fc1c (stale) Mirror 7: 0x9a66fc1c (stale) Mirror 9: 0x9a66fc1c (stale) Mirror 11: 0x9a66fc1c (stale) Mirror 13: 0x9a66fc1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000267 1787143201.160414 write 0x011460 thru 0x020351 (0xeef2 bytes) 000269 1787143201.173214 trunc from 0x040000 to 0x036126 (0x9ed9 bytes) 000271 1787143201.200074 trunc from 0x036127 to 0x00a2d5 (0x2be51 bytes) 000272 1787143201.220936 trunc from 0x00a2d6 to 0x00134c (0x8f89 bytes) 000274 1787143201.246097 read 0x0007ad thru 0x00134c (0xba0 bytes) 000275 1787143201.260760 mapread 0x0007a2 thru 0x00134c (0xbab bytes) 000276 1787143201.263106 mapwrite 0x02da2a thru 0x0328e4 (0x4ebb bytes) 000278 1787143201.363863 write 0x02f012 thru 0x03a623 (0xb612 bytes) 000279 1787143201.365253 write 0x03eef3 thru 0x03ffff (0x110d bytes) 000281 1787143201.368064 write 0x02b18f thru 0x039e2a (0xec9c bytes) 000282 1787143201.370133 write 0x02a19c thru 0x0335cd (0x9432 bytes) 000283 1787143201.372001 mapread 0x03cb9b thru 0x03ffff (0x3465 bytes) 000284 1787143201.377657 trunc from 0x040000 to 0x02f163 (0x10e9c bytes) 000285 1787143201.403053 MIRROR_RESYNC 6 mirrors 000286 1787143203.384252 mapwrite 0x0328d6 thru 0x0360f7 (0x3822 bytes) 000287 1787143203.587813 mapwrite 0x018485 thru 0x018545 (0xc1 bytes) 000288 1787143203.655782 mapwrite 0x0201c0 thru 0x02b3c6 (0xb207 bytes) 000290 1787143203.723171 trunc from 0x0360f8 to 0x0235b5 (0x12b42 bytes) 000294 1787143203.790085 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x235b6): Mirror 5: 0x8277370e (stale) Mirror 7: 0x8277370e (stale) Mirror 9: 0x8277370e (stale) Mirror 11: 0x8277370e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235b6): Mirror 5: 0x8277370e (stale) Mirror 7: 0x8277370e (stale) Mirror 9: 0x8277370e (stale) Mirror 11: 0x8277370e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235b6: skipped 000295 1787143204.144176 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x235b6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235b6): Mirror 5: 0x8277370e (stale) Mirror 7: 0x8277370e (stale) Mirror 9: 0x8277370e (stale) Mirror 11: 0x8277370e (stale) CRC-32 checksum value for chunk [0, 0x235b6): Mirror 1: 0x73c384cd Mirror 12: 0x73c384cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235b6: skipped 000296 1787143205.389414 mapread 0x018826 thru 0x0235b5 (0xad90 bytes) 000299 1787143205.458428 read 0x005f8f thru 0x00d8e6 (0x7958 bytes) 000300 1787143205.473438 write 0x004eac thru 0x00602f (0x1184 bytes) 000303 1787143205.589551 mapread 0x008a58 thru 0x00ddd9 (0x5382 bytes) 000304 1787143205.610182 write 0x007c89 thru 0x012c69 (0xafe1 bytes) 000305 1787143205.617958 write 0x0180cc thru 0x01d94c (0x5881 bytes) 000308 1787143205.631368 read 0x01b549 thru 0x0235b5 (0x806d bytes) 000309 1787143205.646297 write 0x03a6dc thru 0x03ffff (0x5924 bytes) 000310 1787143205.653956 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3aa66209 (stale) Mirror 9: 0x3aa66209 (stale) Mirror 11: 0x3aa66209 (stale) Mirror 12: 0xd377f8d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3aa66209 (stale) Mirror 9: 0x3aa66209 (stale) Mirror 11: 0x3aa66209 (stale) Mirror 12: 0xd377f8d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000311 1787143206.109051 trunc from 0x040000 to 0x00f22c (0x30dd3 bytes) 000312 1787143206.125447 mapwrite 0x0180e8 thru 0x01edb9 (0x6cd2 bytes) 000313 1787143206.225988 trunc from 0x01edba to 0x02e6bd (0xf904 bytes) 000315 1787143206.249691 mapread 0x012361 thru 0x0174b7 (0x5157 bytes) 000316 1787143206.265947 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e6be): Mirror 7: 0x5946f50a (stale) Mirror 9: 0x5946f50a (stale) Mirror 12: 0x653b0ec3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6be): Mirror 7: 0x5946f50a (stale) Mirror 9: 0x5946f50a (stale) Mirror 12: 0x653b0ec3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6be: skipped 000317 1787143206.687485 read 0x023611 thru 0x02e6bd (0xb0ad bytes) 000319 1787143206.707083 trunc from 0x02e6be to 0x03a86b (0xc1ae bytes) 000320 1787143206.750435 MIRROR_RESYNC 4 mirrors 000321 1787143208.367916 write 0x02c941 thru 0x031b6b (0x522b bytes) 000322 1787143208.452395 write 0x031aca thru 0x036941 (0x4e78 bytes) 000323 1787143208.460384 write 0x02f518 thru 0x032417 (0x2f00 bytes) 000324 1787143208.468617 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a86c): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a86c): Mirror 7: 0x2321051e (stale) Mirror 9: 0x2321051e (stale) Mirror 12: 0x2321051e (stale) CRC-32 checksum value for chunk [0, 0x3a86c): Mirror 1: 0x4022c47c Mirror 13: 0x4022c47c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a86c: skipped 000325 1787143209.963898 write 0x01676c thru 0x022dec (0xc681 bytes) 000326 1787143210.077894 write 0x029390 thru 0x02c8cc (0x353d bytes) 000327 1787143210.092464 read 0x017748 thru 0x026b85 (0xf43e bytes) 000333 1787143210.106819 read 0x0295f2 thru 0x030bbd (0x75cc bytes) 000335 1787143210.118345 read 0x015f88 thru 0x01b1aa (0x5223 bytes) 000336 1787143210.119513 read 0x01c575 thru 0x026c63 (0xa6ef bytes) 000337 1787143210.121012 trunc from 0x03a86c to 0x01b3d6 (0x1f495 bytes) 000338 1787143210.142855 trunc from 0x01b3d7 to 0x02b0c1 (0xfceb bytes) 000344 1787143210.183953 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b0c2): Mirror 9: 0x3a358759 (stale) Mirror 12: 0x3a358759 (stale) Mirror 13: 0x3a358759 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b0c2): Mirror 9: 0x3a358759 (stale) Mirror 12: 0x3a358759 (stale) Mirror 13: 0x3a358759 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b0c2: skipped 000346 1787143210.660870 mapread 0x00f1c1 thru 0x017567 (0x83a7 bytes) 000347 1787143210.700158 MIRROR_RESYNC 4 mirrors 000349 1787143211.887034 read 0x002255 thru 0x00e261 (0xc00d bytes) 000350 1787143211.918627 read 0x000aa3 thru 0x00b048 (0xa5a6 bytes) 000353 1787143211.934892 mapread 0x00d755 thru 0x0152a7 (0x7b53 bytes) 000356 1787143211.936739 write 0x02eabb thru 0x0361cf (0x7715 bytes) 000357 1787143212.030380 read 0x00503b thru 0x00ef16 (0x9edc bytes) 000359 1787143212.051849 write 0x00c62c thru 0x01a857 (0xe22c bytes) 000360 1787143212.062476 MIRROR_RESYNC 4 mirrors 000361 1787143213.505016 mapread 0x029faa thru 0x02cc64 (0x2cbb bytes) 000365 1787143213.544722 write 0x02c33f thru 0x038f61 (0xcc23 bytes) 000366 1787143213.701097 trunc from 0x038f62 to 0x03ab8e (0x1c2d bytes) 000370 1787143213.777986 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ab8f): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab8f): Mirror 9: 0x53e3d29c (stale) Mirror 12: 0x53e3d29c (stale) Mirror 13: 0x53e3d29c (stale) CRC-32 checksum value for chunk [0, 0x3ab8f): Mirror 1: 0x443aa475 Mirror 14: 0x443aa475 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab8f: skipped 000372 1787143215.134919 mapread 0x00891f thru 0x013bc4 (0xb2a6 bytes) 000373 1787143215.187492 trunc from 0x03ab8f to 0x028e24 (0x11d6a bytes) 000375 1787143215.345166 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28e25): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e25): Mirror 9: 0xb954bb8c (stale) Mirror 12: 0xb954bb8c (stale) Mirror 13: 0xb954bb8c (stale) Mirror 14: 0xb954bb8c (stale) CRC-32 checksum value for chunk [0, 0x28e25): Mirror 1: 0xb954bb8c Mirror 15: 0xb954bb8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e25: skipped 000376 1787143216.676931 mapread 0x019b31 thru 0x01ece5 (0x51b5 bytes) 000377 1787143216.723183 write 0x01f49e thru 0x01fece (0xa31 bytes) 000379 1787143216.832431 mapwrite 0x00b0b2 thru 0x00d4ba (0x2409 bytes) 000381 1787143216.899022 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28e25): Mirror 12: 0xb954bb8c (stale) Mirror 13: 0xb954bb8c (stale) Mirror 14: 0xb954bb8c (stale) Mirror 15: 0xb954bb8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e25): Mirror 12: 0xb954bb8c (stale) Mirror 13: 0xb954bb8c (stale) Mirror 14: 0xb954bb8c (stale) Mirror 15: 0xb954bb8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e25: skipped 000382 1787143217.508618 MIRROR_RESYNC 5 mirrors 000383 1787143219.046551 write 0x03d0b8 thru 0x03ffff (0x2f48 bytes) 000386 1787143219.116789 mapwrite 0x0399be thru 0x03f07d (0x56c0 bytes) 000388 1787143219.211335 write 0x03ac9e thru 0x03b865 (0xbc8 bytes) 000389 1787143219.212424 mapwrite 0x001ad1 thru 0x0094fc (0x7a2c bytes) 000390 1787143219.298160 mapread 0x02fd3b thru 0x034340 (0x4606 bytes) 000393 1787143219.311229 mapread 0x03812b thru 0x03ffff (0x7ed5 bytes) 000395 1787143219.313485 read 0x00f354 thru 0x00f89c (0x549 bytes) 000396 1787143219.318937 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x6f051d9c (stale) Mirror 14: 0x6f051d9c (stale) Mirror 15: 0x6f051d9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x6f051d9c (stale) Mirror 14: 0x6f051d9c (stale) Mirror 15: 0x6f051d9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000398 1787143219.833318 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x6f051d9c (stale) Mirror 15: 0x6f051d9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x6f051d9c (stale) Mirror 15: 0x6f051d9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000399 1787143220.240689 write 0x026ad9 thru 0x02c7ce (0x5cf6 bytes) 000401 1787143220.262702 write 0x01d08a thru 0x028961 (0xb8d8 bytes) 000402 1787143220.269346 write 0x03f8c9 thru 0x03ffff (0x737 bytes) 000404 1787143220.285288 mapwrite 0x021ab1 thru 0x026b8e (0x50de bytes) 000405 1787143220.447348 trunc from 0x040000 to 0x01bc40 (0x243bf bytes) 000408 1787143220.481142 write 0x000c68 thru 0x0108d9 (0xfc72 bytes) 000410 1787143220.497440 mapwrite 0x01fe36 thru 0x02f982 (0xfb4d bytes) 000412 1787143220.646319 read 0x01af25 thru 0x025c92 (0xad6e bytes) 000416 1787143220.649530 read 0x0181fc thru 0x01cc01 (0x4a06 bytes) 000417 1787143220.667935 read 0x00b256 thru 0x017a75 (0xc820 bytes) 000422 1787143220.674053 mapread 0x02df25 thru 0x02f982 (0x1a5e bytes) 000426 1787143220.686742 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f983): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f983): Mirror 14: 0xe133b1a4 (stale) Mirror 15: 0xe133b1a4 (stale) CRC-32 checksum value for chunk [0, 0x2f983): Mirror 1: 0xa91a3f3e Mirror 16: 0xa91a3f3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f983: skipped 000427 1787143221.914689 MIRROR_RESYNC 4 mirrors 000428 1787143223.064465 mapread 0x01b194 thru 0x022e36 (0x7ca3 bytes) 000429 1787143223.089108 trunc from 0x02f983 to 0x03031a (0x998 bytes) 000431 1787143223.166773 mapwrite 0x02a2b0 thru 0x036af4 (0xc845 bytes) 000432 1787143223.288480 trunc from 0x036af5 to 0x029c76 (0xce7e bytes) 000433 1787143223.314014 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c77): Mirror 15: 0x24fc4b14 (stale) Mirror 16: 0x24fc4b14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c77): Mirror 15: 0x24fc4b14 (stale) Mirror 16: 0x24fc4b14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c77: skipped 000434 1787143223.808755 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29c77): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c77): Mirror 15: 0x24fc4b14 (stale) Mirror 16: 0x24fc4b14 (stale) CRC-32 checksum value for chunk [0, 0x29c77): Mirror 1: 0x24fc4b14 Mirror 17: 0x24fc4b14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c77: skipped 000435 1787143224.966036 trunc from 0x029c77 to 0x003dbd (0x25eb9 bytes) 000436 1787143225.118928 trunc from 0x003dbe to 0x034bda (0x30e1d bytes) 000437 1787143225.195350 write 0x034ca3 thru 0x03bfc9 (0x7327 bytes) 000441 1787143225.233033 MIRROR_RESYNC 4 mirrors 000442 1787143226.581226 mapwrite 0x03c9c0 thru 0x03ffff (0x3640 bytes) 000444 1787143226.705380 trunc from 0x040000 to 0x02ed0b (0x112f4 bytes) 000446 1787143226.731888 MIRROR_RESYNC 4 mirrors 000448 1787143228.017361 read 0x010021 thru 0x01d53b (0xd51b bytes) 000449 1787143228.041783 mapwrite 0x035651 thru 0x037aab (0x245b bytes) 000450 1787143228.245056 mapwrite 0x0044da thru 0x00c273 (0x7d9a bytes) 000452 1787143228.347211 read 0x033a6d thru 0x037aab (0x403f bytes) 000453 1787143228.348214 read 0x02a709 thru 0x02f4f0 (0x4de8 bytes) 000455 1787143228.385429 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37aac): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37aac): Mirror 15: 0xde2d5f2a (stale) Mirror 16: 0xde2d5f2a (stale) Mirror 17: 0xde2d5f2a (stale) CRC-32 checksum value for chunk [0, 0x37aac): Mirror 1: 0xefbc0c5c Mirror 18: 0xefbc0c5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37aac: skipped 000459 1787143229.713203 trunc from 0x037aac to 0x02c44b (0xb660 bytes) 000460 1787143229.866014 mapwrite 0x02fe58 thru 0x03cfe9 (0xd192 bytes) 000461 1787143230.008906 read 0x00d000 thru 0x00fa76 (0x2a77 bytes) 000462 1787143230.022087 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfea): Mirror 15: 0x6570d79d (stale) Mirror 16: 0x6570d79d (stale) Mirror 17: 0x6570d79d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfea): Mirror 15: 0x6570d79d (stale) Mirror 16: 0x6570d79d (stale) Mirror 17: 0x6570d79d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfea: skipped 000463 1787143230.426917 mapwrite 0x0282a7 thru 0x032d8c (0xaae6 bytes) 000464 1787143230.511140 MIRROR_RESYNC 4 mirrors 000465 1787143231.972519 read 0x00b7d1 thru 0x01702f (0xb85f bytes) 000466 1787143231.993713 mapread 0x02175f thru 0x03168e (0xff30 bytes) 000467 1787143232.027448 trunc from 0x03cfea to 0x025985 (0x17664 bytes) 000468 1787143232.118277 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25986): Mirror 15: 0xeb5193e6 (stale) Mirror 16: 0xeb5193e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25986): Mirror 15: 0xeb5193e6 (stale) Mirror 16: 0xeb5193e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25986: skipped 000469 1787143232.456273 trunc from 0x025986 to 0x0122bc (0x136c9 bytes) 000470 1787143232.479276 write 0x01acf4 thru 0x01f7c8 (0x4ad5 bytes) 000471 1787143232.482104 mapread 0x010134 thru 0x014ab9 (0x4986 bytes) 000472 1787143232.507048 read 0x01a045 thru 0x01f7c8 (0x5784 bytes) 000473 1787143232.507767 write 0x03de9d thru 0x03ffff (0x2163 bytes) 000474 1787143232.511038 write 0x02b69c thru 0x02f8da (0x423f bytes) 000475 1787143232.538833 mapread 0x033909 thru 0x03fb84 (0xc27c bytes) 000478 1787143232.570768 mapread 0x00bffa thru 0x00e915 (0x291c bytes) 000479 1787143232.574793 read 0x0390ea thru 0x03ffff (0x6f16 bytes) 000483 1787143232.577064 write 0x02cf7e thru 0x036fe9 (0xa06c bytes) 000486 1787143232.579458 read 0x025071 thru 0x02beb5 (0x6e45 bytes) 000487 1787143232.590542 trunc from 0x040000 to 0x02e554 (0x11aab bytes) 000488 1787143232.611891 mapread 0x025114 thru 0x02e554 (0x9441 bytes) 000489 1787143232.615246 mapread 0x00db77 thru 0x011a2e (0x3eb8 bytes) 000491 1787143232.617312 mapread 0x01f3ff thru 0x029775 (0xa377 bytes) 000492 1787143232.618408 mapread 0x002913 thru 0x01026a (0xd958 bytes) 000493 1787143232.619433 write 0x015d2e thru 0x018b1f (0x2df2 bytes) 000496 1787143232.621985 trunc from 0x02e555 to 0x006e7c (0x276d8 bytes) 000498 1787143232.640871 trunc from 0x006e7d to 0x015fa1 (0xf125 bytes) 000501 1787143232.660797 trunc from 0x015fa2 to 0x037aa2 (0x21b01 bytes) 000503 1787143232.683389 read 0x023a52 thru 0x0246be (0xc6d bytes) 000506 1787143232.700797 read 0x02d6db thru 0x033fd4 (0x68fa bytes) 000507 1787143232.701530 write 0x01ce3e thru 0x0210d0 (0x4293 bytes) 000508 1787143232.706639 write 0x03d489 thru 0x03ffff (0x2b77 bytes) 000509 1787143232.710660 read 0x02826b thru 0x0322a7 (0xa03d bytes) 000510 1787143232.717218 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x6775b0d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x6775b0d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1787143233.061361 write 0x03b863 thru 0x03ffff (0x479d bytes) 000513 1787143233.070942 trunc from 0x040000 to 0x010843 (0x2f7bc bytes) 000514 1787143233.124319 mapread 0x010387 thru 0x010843 (0x4bd bytes) 000515 1787143233.139446 mapwrite 0x00f2d5 thru 0x014745 (0x5471 bytes) 000518 1787143233.285103 mapread 0x00dca6 thru 0x014745 (0x6aa0 bytes) 000519 1787143233.289799 mapread 0x006495 thru 0x00af3e (0x4aaa bytes) 000520 1787143233.320054 read 0x013825 thru 0x014745 (0xf21 bytes) 000521 1787143233.321013 read 0x00f752 thru 0x014745 (0x4ff4 bytes) 000522 1787143233.333968 MIRROR_RESYNC 2 mirrors 000523 1787143234.337145 mapwrite 0x036cd9 thru 0x038ecc (0x21f4 bytes) 000526 1787143234.524974 trunc from 0x038ecd to 0x01968f (0x1f83d bytes) 000527 1787143234.555314 write 0x004685 thru 0x00e963 (0xa2df bytes) 000530 1787143234.570258 mapwrite 0x0377a2 thru 0x03f9c8 (0x8227 bytes) 000531 1787143234.659967 write 0x039c1d thru 0x03da48 (0x3e2c bytes) 000532 1787143234.666633 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000533 1787143234.824247 mapwrite 0x013d39 thru 0x023c96 (0xff5e bytes) 000534 1787143234.940443 mapwrite 0x03b2b2 thru 0x03ffff (0x4d4e bytes) 000535 1787143235.063981 write 0x024187 thru 0x026a51 (0x28cb bytes) 000537 1787143235.065908 trunc from 0x040000 to 0x036676 (0x9989 bytes) 000538 1787143235.097307 mapwrite 0x00f2a0 thru 0x013218 (0x3f79 bytes) 000541 1787143235.150232 read 0x033a04 thru 0x036676 (0x2c73 bytes) 000542 1787143235.158837 mapwrite 0x00eaca thru 0x00fa0c (0xf43 bytes) 000543 1787143235.228160 trunc from 0x036677 to 0x02a1a9 (0xc4cd bytes) 000546 1787143235.265528 mapwrite 0x03fc38 thru 0x03ffff (0x3c8 bytes) 000549 1787143235.379180 mapwrite 0x0011e8 thru 0x002ad2 (0x18eb bytes) 000551 1787143235.467730 write 0x015e60 thru 0x02070c (0xa8ad bytes) 000554 1787143235.475010 write 0x031cb3 thru 0x03790b (0x5c59 bytes) 000555 1787143235.490088 mapwrite 0x01aee4 thru 0x027db9 (0xced6 bytes) 000560 1787143235.616415 read 0x01d63b thru 0x0208ab (0x3271 bytes) 000562 1787143235.623759 trunc from 0x040000 to 0x0129f7 (0x2d608 bytes) 000564 1787143235.651876 read 0x0026c9 thru 0x0087d1 (0x6109 bytes) 000566 1787143235.654103 trunc from 0x0129f8 to 0x03dfe8 (0x2b5f1 bytes) 000567 1787143235.673421 mapwrite 0x027252 thru 0x02b1fa (0x3fa9 bytes) 000570 1787143235.739253 write 0x008e87 thru 0x017597 (0xe711 bytes) 000571 1787143235.740374 read 0x014ecf thru 0x01fdea (0xaf1c bytes) 000574 1787143235.743777 write 0x030796 thru 0x03630a (0x5b75 bytes) 000576 1787143235.748184 mapwrite 0x01805a thru 0x022aad (0xaa54 bytes) 000577 1787143235.826530 trunc from 0x03dfe9 to 0x039e99 (0x414f bytes) 000578 1787143235.856014 read 0x036de3 thru 0x039e99 (0x30b7 bytes) 000579 1787143235.867809 MIRROR_EXTEND to 2 mirrors 000580 1787143236.998887 MIRROR_RESYNC 2 mirrors 000582 1787143237.460901 read 0x01c186 thru 0x0218eb (0x5766 bytes) 000583 1787143237.499419 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000584 1787143237.759009 read 0x02dcc7 thru 0x02de11 (0x14b bytes) 000587 1787143237.771978 mapread 0x012556 thru 0x018125 (0x5bd0 bytes) 000588 1787143237.801370 write 0x0354e0 thru 0x03ab12 (0x5633 bytes) 000589 1787143237.827101 mapwrite 0x03466d thru 0x03b0c2 (0x6a56 bytes) 000590 1787143237.949521 trunc from 0x03b0c3 to 0x00ee8c (0x2c236 bytes) 000595 1787143237.990066 read 0x006105 thru 0x00995f (0x385b bytes) 000596 1787143238.003336 mapread 0x006ea3 thru 0x00ee8c (0x7fea bytes) 000597 1787143238.006911 mapwrite 0x0274f4 thru 0x02f91d (0x842a bytes) 000599 1787143238.160490 trunc from 0x02f91e to 0x002a4e (0x2cecf bytes) 000602 1787143238.187958 mapread 0x000106 thru 0x002a4e (0x2949 bytes) 000603 1787143238.188892 read 0x001371 thru 0x002a4e (0x16de bytes) 000604 1787143238.189315 mapwrite 0x018159 thru 0x02177b (0x9623 bytes) 000605 1787143238.261776 mapwrite 0x0348e2 thru 0x03ffff (0xb71e bytes) 000607 1787143238.351487 mapread 0x024650 thru 0x033e90 (0xf841 bytes) 000609 1787143238.355790 read 0x00b55e thru 0x019c31 (0xe6d4 bytes) 000611 1787143238.357043 trunc from 0x040000 to 0x001f9b (0x3e064 bytes) 000615 1787143238.379959 trunc from 0x001f9c to 0x01f366 (0x1d3cb bytes) 000619 1787143238.399877 write 0x0078d7 thru 0x008719 (0xe43 bytes) 000621 1787143238.416042 mapread 0x004f3e thru 0x00f21f (0xa2e2 bytes) 000624 1787143238.419355 mapread 0x00984b thru 0x00d9ed (0x41a3 bytes) 000627 1787143238.424543 mapwrite 0x02512b thru 0x028ed6 (0x3dac bytes) 000628 1787143238.513312 mapread 0x000907 thru 0x0058e4 (0x4fde bytes) 000631 1787143238.515461 write 0x01e52f thru 0x028b44 (0xa616 bytes) 000632 1787143238.517900 read 0x00dd3c thru 0x010d38 (0x2ffd bytes) 000633 1787143238.518456 mapread 0x02612a thru 0x028ed6 (0x2dad bytes) 000634 1787143238.519769 mapread 0x008b54 thru 0x00ccd6 (0x4183 bytes) 000635 1787143238.521259 trunc from 0x028ed7 to 0x0374e2 (0xe60c bytes) 000636 1787143238.542181 read 0x0073e0 thru 0x01160b (0xa22c bytes) 000639 1787143238.544224 write 0x020cde thru 0x026ccd (0x5ff0 bytes) 000640 1787143238.545251 write 0x02e5db thru 0x02edee (0x814 bytes) 000643 1787143238.559957 mapwrite 0x02a991 thru 0x0377f4 (0xce64 bytes) 000645 1787143238.658423 write 0x0191f3 thru 0x0276f7 (0xe505 bytes) 000647 1787143238.664374 read 0x021711 thru 0x028057 (0x6947 bytes) 000649 1787143238.668854 write 0x0083d1 thru 0x0108c5 (0x84f5 bytes) 000650 1787143238.674098 mapwrite 0x02f055 thru 0x03e190 (0xf13c bytes) 000652 1787143238.843184 read 0x013204 thru 0x016a00 (0x37fd bytes) 000654 1787143238.857457 MIRROR_EXTEND to 2 mirrors 000655 1787143239.769213 mapwrite 0x02ad6c thru 0x036096 (0xb32b bytes) 000656 1787143240.047781 read 0x006093 thru 0x00aa36 (0x49a4 bytes) 000657 1787143240.061109 mapwrite 0x00260c thru 0x00d512 (0xaf07 bytes) 000658 1787143240.149025 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000659 1787143240.312004 mapwrite 0x01a84c thru 0x01c225 (0x19da bytes) 000660 1787143240.375303 MIRROR_EXTEND to 2 mirrors 000662 1787143241.157531 mapread 0x020ebc thru 0x029e9a (0x8fdf bytes) 000667 1787143241.224603 write 0x01d5d4 thru 0x01f76f (0x219c bytes) 000669 1787143241.366256 mapwrite 0x01d3ca thru 0x01e23f (0xe76 bytes) 000670 1787143241.459255 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e191): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e191): Mirror 2: 0xbd8dc3d9 (stale) CRC-32 checksum value for chunk [0, 0x3e191): Mirror 1: 0xf70697fe Mirror 3: 0xf70697fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e191: skipped 000673 1787143242.300875 write 0x016d0b thru 0x02550c (0xe802 bytes) 000674 1787143242.436340 mapwrite 0x03efa6 thru 0x03ffff (0x105a bytes) 000675 1787143242.570224 MIRROR_RESYNC 3 mirrors 000680 1787143243.616233 trunc from 0x040000 to 0x037a30 (0x85cf bytes) 000681 1787143243.791521 MIRROR_RESYNC 3 mirrors 000682 1787143245.144860 read 0x010bc8 thru 0x01b78a (0xabc3 bytes) 000683 1787143245.196015 MIRROR_RESYNC 3 mirrors 000684 1787143245.851094 trunc from 0x037a31 to 0x027822 (0x1020e bytes) 000687 1787143245.957992 MIRROR_RESYNC 3 mirrors 000688 1787143247.003659 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000689 1787143247.520438 MIRROR_EXTEND to 3 mirrors 000690 1787143248.389144 MIRROR_RESYNC 3 mirrors 000691 1787143248.795663 mapread 0x01ddf7 thru 0x01e618 (0x822 bytes) 000696 1787143248.811457 write 0x02e06d thru 0x03a705 (0xc699 bytes) 000697 1787143248.913802 write 0x01867c thru 0x01f86f (0x71f4 bytes) 000700 1787143248.958444 write 0x029d8d thru 0x030cba (0x6f2e bytes) 000701 1787143248.968980 mapread 0x01e4fa thru 0x0225d0 (0x40d7 bytes) 000702 1787143248.986302 write 0x02a1c1 thru 0x02ab25 (0x965 bytes) 000704 1787143248.990091 mapwrite 0x0024ab thru 0x00ec79 (0xc7cf bytes) 000706 1787143249.080994 read 0x011719 thru 0x01d6a0 (0xbf88 bytes) 000707 1787143249.100253 mapwrite 0x023a60 thru 0x02ff0c (0xc4ad bytes) 000708 1787143249.162003 write 0x0271ab thru 0x032bdc (0xba32 bytes) 000711 1787143249.166077 mapread 0x01b176 thru 0x0295a0 (0xe42b bytes) 000712 1787143249.167411 mapread 0x028089 thru 0x0372d7 (0xf24f bytes) 000714 1787143249.170031 write 0x02f167 thru 0x03428b (0x5125 bytes) 000717 1787143249.177547 trunc from 0x03a706 to 0x0087a9 (0x31f5c bytes) 000718 1787143249.204091 write 0x02f357 thru 0x03e896 (0xf540 bytes) 000720 1787143249.217460 MIRROR_RESYNC 3 mirrors 000722 1787143250.743454 mapwrite 0x00b3a4 thru 0x00d122 (0x1d7f bytes) 000723 1787143251.027038 MIRROR_RESYNC 3 mirrors 000725 1787143252.268545 MIRROR_EXTEND to 4 mirrors 000727 1787143253.129811 mapread 0x02953b thru 0x02ed10 (0x57d6 bytes) 000728 1787143253.162082 mapread 0x02c1e8 thru 0x035ee9 (0x9d02 bytes) 000729 1787143253.174850 trunc from 0x03e897 to 0x01589d (0x28ff9 bytes) 000730 1787143253.345018 trunc from 0x01589e to 0x00083a (0x15063 bytes) 000731 1787143253.388186 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x83b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83b): Mirror 3: 0x4ec22736 (stale) Mirror 4: 0x4ec22736 (stale) Mirror 5: 0x4ec22736 (stale) CRC-32 checksum value for chunk [0, 0x83b): Mirror 1: 0x4ec22736 Mirror 6: 0x4ec22736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83b: skipped 000733 1787143254.486700 write 0x00914d thru 0x012573 (0x9427 bytes) 000740 1787143254.599121 mapwrite 0x030533 thru 0x03aecc (0xa99a bytes) 000741 1787143254.723199 trunc from 0x03aecd to 0x004734 (0x36798 bytes) 000742 1787143254.754297 mapread 0x00099b thru 0x000c14 (0x27a bytes) 000744 1787143254.766450 trunc from 0x004735 to 0x0111ad (0xca79 bytes) 000745 1787143254.794830 write 0x004f4b thru 0x0109bd (0xba73 bytes) 000746 1787143254.814869 trunc from 0x0111ae to 0x01d17e (0xbfd1 bytes) 000751 1787143254.845662 read 0x004fa1 thru 0x0071f1 (0x2251 bytes) 000752 1787143254.850501 MIRROR_RESYNC 5 mirrors 000753 1787143256.453966 write 0x008b8b thru 0x0134fc (0xa972 bytes) 000756 1787143256.579028 MIRROR_RESYNC 5 mirrors 000757 1787143257.997076 trunc from 0x01d17f to 0x013a7a (0x9704 bytes) 000758 1787143258.110311 mapread 0x002ad3 thru 0x003e19 (0x1347 bytes) 000759 1787143258.122906 mapwrite 0x022e1e thru 0x023b13 (0xcf6 bytes) 000760 1787143258.226861 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23b14): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b14): Mirror 3: 0x855bf63e (stale) Mirror 4: 0x855bf63e (stale) Mirror 5: 0x855bf63e (stale) Mirror 6: 0x855bf63e (stale) CRC-32 checksum value for chunk [0, 0x23b14): Mirror 1: 0xe492ff4b Mirror 7: 0xe492ff4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b14: skipped 000761 1787143259.341473 MIRROR_RESYNC 6 mirrors 000762 1787143260.753638 read 0x010af0 thru 0x01ccf4 (0xc205 bytes) 000763 1787143260.767707 trunc from 0x023b14 to 0x0355ad (0x11a9a bytes) 000764 1787143260.869855 write 0x03b988 thru 0x03ffff (0x4678 bytes) 000766 1787143260.873285 mapwrite 0x0030e4 thru 0x007a40 (0x495d bytes) 000770 1787143260.965266 mapwrite 0x02081b thru 0x023f98 (0x377e bytes) 000771 1787143261.059325 mapwrite 0x02ae39 thru 0x033e07 (0x8fcf bytes) 000772 1787143261.155470 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc08e88 (stale) Mirror 4: 0xc08e88 (stale) Mirror 5: 0xc08e88 (stale) Mirror 6: 0xc08e88 (stale) Mirror 7: 0xc08e88 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x301b1895 Mirror 8: 0x301b1895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1787143262.226710 read 0x03349f thru 0x03a507 (0x7069 bytes) 000777 1787143262.250645 mapread 0x036f69 thru 0x03d9c2 (0x6a5a bytes) 000780 1787143262.305676 trunc from 0x040000 to 0x02d039 (0x12fc6 bytes) 000783 1787143262.439196 MIRROR_RESYNC 7 mirrors 000784 1787143264.493444 MIRROR_RESYNC 7 mirrors 000785 1787143265.198869 trunc from 0x02d03a to 0x031c64 (0x4c2b bytes) 000786 1787143265.418648 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31c65): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c65): Mirror 3: 0x2cf71519 (stale) Mirror 4: 0x2cf71519 (stale) Mirror 5: 0x2cf71519 (stale) Mirror 6: 0x2cf71519 (stale) Mirror 7: 0x2cf71519 (stale) Mirror 8: 0x2cf71519 (stale) CRC-32 checksum value for chunk [0, 0x31c65): Mirror 1: 0x2cf71519 Mirror 9: 0x2cf71519 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c65: skipped 000788 1787143266.915312 write 0x03a191 thru 0x03ffff (0x5e6f bytes) 000789 1787143267.031291 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x114196b4 (stale) Mirror 4: 0x114196b4 (stale) Mirror 6: 0x114196b4 (stale) Mirror 7: 0x114196b4 (stale) Mirror 8: 0x114196b4 (stale) Mirror 9: 0x114196b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x114196b4 (stale) Mirror 4: 0x114196b4 (stale) Mirror 6: 0x114196b4 (stale) Mirror 7: 0x114196b4 (stale) Mirror 8: 0x114196b4 (stale) Mirror 9: 0x114196b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1787143267.828504 read 0x02ae5f thru 0x02efb6 (0x4158 bytes) 000791 1787143267.851329 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x114196b4 (stale) Mirror 4: 0x114196b4 (stale) Mirror 6: 0x114196b4 (stale) Mirror 7: 0x114196b4 (stale) Mirror 9: 0x114196b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x114196b4 (stale) Mirror 4: 0x114196b4 (stale) Mirror 6: 0x114196b4 (stale) Mirror 7: 0x114196b4 (stale) Mirror 9: 0x114196b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1787143268.441315 mapread 0x01f6bc thru 0x01fb1d (0x462 bytes) 000796 1787143268.453216 mapread 0x009e88 thru 0x00c6b6 (0x282f bytes) 000797 1787143268.473439 trunc from 0x040000 to 0x0175bc (0x28a43 bytes) 000799 1787143268.506892 mapwrite 0x0000fe thru 0x008318 (0x821b bytes) 000800 1787143268.561102 trunc from 0x0175bd to 0x0150d2 (0x24ea bytes) 000801 1787143268.590153 read 0x00ba20 thru 0x0111f5 (0x57d6 bytes) 000802 1787143268.591718 trunc from 0x0150d3 to 0x010c0f (0x44c3 bytes) 000803 1787143268.613696 trunc from 0x010c10 to 0x0292d2 (0x186c3 bytes) 000804 1787143268.646114 trunc from 0x0292d3 to 0x02bbbb (0x28e9 bytes) 000806 1787143268.686433 read 0x005555 thru 0x00d008 (0x7ab4 bytes) 000810 1787143268.699074 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bbbc): Mirror 3: 0xb2605b14 (stale) Mirror 4: 0xb2605b14 (stale) Mirror 6: 0xb2605b14 (stale) Mirror 9: 0xb2605b14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbbc): Mirror 3: 0xb2605b14 (stale) Mirror 4: 0xb2605b14 (stale) Mirror 6: 0xb2605b14 (stale) Mirror 9: 0xb2605b14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbbc: skipped 000811 1787143269.222107 MIRROR_RESYNC 5 mirrors 000812 1787143270.950782 MIRROR_EXTEND to 6 mirrors 000814 1787143272.071026 read 0x001477 thru 0x00d24b (0xbdd5 bytes) 000816 1787143272.103479 read 0x00e43a thru 0x01af1b (0xcae2 bytes) 000817 1787143272.123658 read 0x010bc7 thru 0x01ce3d (0xc277 bytes) 000819 1787143272.125769 trunc from 0x02bbbc to 0x009f8b (0x21c30 bytes) 000820 1787143272.290924 mapread 0x00290c thru 0x009f8b (0x7680 bytes) 000823 1787143272.316737 write 0x0218ed thru 0x02fac1 (0xe1d5 bytes) 000824 1787143272.327792 mapread 0x00c81e thru 0x011bb4 (0x5397 bytes) 000825 1787143272.359296 mapwrite 0x03db8d thru 0x03ffff (0x2473 bytes) 000826 1787143272.451182 read 0x013433 thru 0x01f69d (0xc26b bytes) 000827 1787143272.456015 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x76fb4c47 (stale) Mirror 4: 0x76fb4c47 (stale) Mirror 6: 0x76fb4c47 (stale) Mirror 9: 0x76fb4c47 (stale) Mirror 10: 0x76fb4c47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x235e65e1 Mirror 11: 0x235e65e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000828 1787143274.388305 MIRROR_RESYNC 7 mirrors 000831 1787143277.211525 read 0x015593 thru 0x022ce0 (0xd74e bytes) 000833 1787143277.223169 mapread 0x01761b thru 0x018708 (0x10ee bytes) 000835 1787143277.228504 MIRROR_RESYNC 7 mirrors 000836 1787143277.868181 mapread 0x01a3ae thru 0x01b8d8 (0x152b bytes) 000837 1787143277.869223 mapread 0x01d6c2 thru 0x027eae (0xa7ed bytes) 000838 1787143277.889326 read 0x0039bc thru 0x00fbfb (0xc240 bytes) 000839 1787143277.902446 trunc from 0x040000 to 0x026c24 (0x193db bytes) 000841 1787143278.007107 trunc from 0x026c25 to 0x01ccc7 (0x9f5d bytes) 000845 1787143278.042469 MIRROR_RESYNC 7 mirrors 000846 1787143279.655410 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000849 1787143280.136106 write 0x02f267 thru 0x03187f (0x2619 bytes) 000850 1787143280.220799 mapread 0x00501b thru 0x007859 (0x283f bytes) 000851 1787143280.252094 trunc from 0x031880 to 0x012057 (0x1f828 bytes) 000853 1787143280.283556 mapread 0x00bd90 thru 0x00fb58 (0x3dc9 bytes) 000854 1787143280.285525 trunc from 0x012058 to 0x03ae11 (0x28dba bytes) 000856 1787143280.315275 MIRROR_RESYNC 6 mirrors 000859 1787143282.220996 mapread 0x02fe73 thru 0x036375 (0x6503 bytes) 000860 1787143282.249261 trunc from 0x03ae12 to 0x009345 (0x31acc bytes) 000861 1787143282.351704 trunc from 0x009346 to 0x001dcd (0x7578 bytes) 000862 1787143282.393786 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1dce): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dce): Mirror 3: 0x15324763 (stale) Mirror 4: 0x15324763 (stale) Mirror 6: 0x15324763 (stale) Mirror 10: 0x15324763 (stale) Mirror 11: 0x15324763 (stale) CRC-32 checksum value for chunk [0, 0x1dce): Mirror 1: 0x15324763 Mirror 12: 0x15324763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dce: skipped 000863 1787143283.489134 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1dce): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dce): Mirror 3: 0x15324763 (stale) Mirror 4: 0x15324763 (stale) Mirror 6: 0x15324763 (stale) Mirror 10: 0x15324763 (stale) Mirror 11: 0x15324763 (stale) CRC-32 checksum value for chunk [0, 0x1dce): Mirror 1: 0x15324763 Mirror 12: 0x15324763 Mirror 13: 0x15324763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dce: skipped 000864 1787143284.496331 write 0x000856 thru 0x009513 (0x8cbe bytes) 000867 1787143284.692659 mapwrite 0x02e2fb thru 0x038707 (0xa40d bytes) 000868 1787143284.909249 write 0x02facd thru 0x03467b (0x4baf bytes) 000871 1787143284.931475 MIRROR_RESYNC 8 mirrors 000872 1787143287.589694 write 0x01480a thru 0x01c777 (0x7f6e bytes) 000873 1787143287.671303 read 0x01f0f8 thru 0x027dc1 (0x8cca bytes) 000874 1787143287.688845 MIRROR_RESYNC 8 mirrors 000876 1787143290.003364 write 0x03d3f9 thru 0x03ffff (0x2c07 bytes) 000879 1787143290.077124 trunc from 0x040000 to 0x03db5c (0x24a3 bytes) 000881 1787143290.098741 write 0x00f456 thru 0x011ad7 (0x2682 bytes) 000882 1787143290.114037 write 0x032fad thru 0x03ffff (0xd053 bytes) 000883 1787143290.131935 read 0x03eb2f thru 0x03ffff (0x14d1 bytes) truncating to largest ever: 0x3f540 000884 1787143290.140882 trunc from 0x040000 to 0x03f53f (0xac0 bytes) 000885 1787143290.194400 mapread 0x01142c thru 0x0203b3 (0xef88 bytes) 000887 1787143290.234323 trunc from 0x03f540 to 0x01d82a (0x21d15 bytes) 000894 1787143290.258414 mapwrite 0x00e170 thru 0x01296f (0x4800 bytes) 000896 1787143290.308362 mapwrite 0x01b790 thru 0x0261c8 (0xaa39 bytes) 000897 1787143290.394014 write 0x007b7f thru 0x01280d (0xac8f bytes) 000898 1787143290.399986 trunc from 0x0261c9 to 0x028d40 (0x2b78 bytes) 000899 1787143290.419286 MIRROR_RESYNC 8 mirrors 000900 1787143292.687095 read 0x0125ee thru 0x01e3b5 (0xbdc8 bytes) 000902 1787143292.703007 read 0x023540 thru 0x028d40 (0x5801 bytes) 000903 1787143292.721741 mapread 0x01ab73 thru 0x028d40 (0xe1ce bytes) 000904 1787143292.723284 trunc from 0x028d41 to 0x008b11 (0x2022f bytes) 000906 1787143292.814820 mapread 0x006a2b thru 0x008b11 (0x20e7 bytes) 000907 1787143292.831556 mapwrite 0x014087 thru 0x021403 (0xd37d bytes) 000909 1787143292.975127 write 0x01efaf thru 0x029f7a (0xafcc bytes) 000913 1787143292.980215 write 0x01b680 thru 0x01fe4b (0x47cc bytes) 000915 1787143292.982013 mapwrite 0x016d38 thru 0x024c8f (0xdf58 bytes) 000916 1787143293.112003 trunc from 0x029f7b to 0x01a30b (0xfc6f bytes) 000921 1787143293.164286 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1a30c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a30c): Mirror 3: 0x4082ba76 (stale) Mirror 4: 0x4082ba76 (stale) Mirror 6: 0x4082ba76 (stale) Mirror 10: 0x4082ba76 (stale) Mirror 11: 0x4082ba76 (stale) Mirror 12: 0x4082ba76 (stale) Mirror 13: 0x4082ba76 (stale) CRC-32 checksum value for chunk [0, 0x1a30c): Mirror 1: 0x7c11d2b5 Mirror 14: 0x7c11d2b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a30c: skipped 000922 1787143294.807078 trunc from 0x01a30c to 0x02d99b (0x13690 bytes) 000923 1787143294.922389 read 0x00a50b thru 0x010ddc (0x68d2 bytes) 000925 1787143294.936349 write 0x01c4a1 thru 0x02116d (0x4ccd bytes) 000927 1787143294.954374 write 0x00f5b1 thru 0x01c8d9 (0xd329 bytes) 000928 1787143294.957805 mapwrite 0x037bbb thru 0x037e96 (0x2dc bytes) 000929 1787143295.152150 mapwrite 0x03c7e2 thru 0x03ffff (0x381e bytes) 000931 1787143295.346899 write 0x02ac4b thru 0x032696 (0x7a4c bytes) 000932 1787143295.373830 mapread 0x002927 thru 0x0121c7 (0xf8a1 bytes) 000933 1787143295.407172 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x48470084 (stale) Mirror 4: 0x48470084 (stale) Mirror 6: 0x48470084 (stale) Mirror 10: 0x48470084 (stale) Mirror 12: 0x48470084 (stale) Mirror 13: 0x48470084 (stale) Mirror 14: 0xc129592a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x48470084 (stale) Mirror 4: 0x48470084 (stale) Mirror 6: 0x48470084 (stale) Mirror 10: 0x48470084 (stale) Mirror 12: 0x48470084 (stale) Mirror 13: 0x48470084 (stale) Mirror 14: 0xc129592a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000934 1787143296.083942 mapread 0x027c5e thru 0x0307d0 (0x8b73 bytes) 000935 1787143296.109292 mapwrite 0x0262dc thru 0x0292c4 (0x2fe9 bytes) 000936 1787143296.218781 trunc from 0x040000 to 0x02442e (0x1bbd1 bytes) 000937 1787143296.250113 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2442f): Mirror 4: 0x4f76bff5 (stale) Mirror 6: 0x4f76bff5 (stale) Mirror 10: 0x4f76bff5 (stale) Mirror 12: 0x4f76bff5 (stale) Mirror 13: 0x4f76bff5 (stale) Mirror 14: 0x59ab6f86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2442f): Mirror 4: 0x4f76bff5 (stale) Mirror 6: 0x4f76bff5 (stale) Mirror 10: 0x4f76bff5 (stale) Mirror 12: 0x4f76bff5 (stale) Mirror 13: 0x4f76bff5 (stale) Mirror 14: 0x59ab6f86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2442f: skipped 000941 1787143296.769448 mapread 0x0040c2 thru 0x012f9e (0xeedd bytes) 000942 1787143296.822980 read 0x00cb86 thru 0x01811b (0xb596 bytes) 000943 1787143296.840913 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2442f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2442f): Mirror 4: 0x4f76bff5 (stale) Mirror 6: 0x4f76bff5 (stale) Mirror 10: 0x4f76bff5 (stale) Mirror 12: 0x4f76bff5 (stale) Mirror 13: 0x4f76bff5 (stale) Mirror 14: 0x59ab6f86 (stale) CRC-32 checksum value for chunk [0, 0x2442f): Mirror 1: 0x9c21257d Mirror 15: 0x9c21257d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2442f: skipped 000945 1787143297.967994 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2442f): Mirror 6: 0x4f76bff5 (stale) Mirror 10: 0x4f76bff5 (stale) Mirror 12: 0x4f76bff5 (stale) Mirror 13: 0x4f76bff5 (stale) Mirror 14: 0x59ab6f86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2442f): Mirror 6: 0x4f76bff5 (stale) Mirror 10: 0x4f76bff5 (stale) Mirror 12: 0x4f76bff5 (stale) Mirror 13: 0x4f76bff5 (stale) Mirror 14: 0x59ab6f86 (stale) CRC-32 checksum value for chunk [0, 0x2442f): Mirror 1: 0x9c21257d Mirror 15: 0x9c21257d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2442f: skipped 000946 1787143299.025314 trunc from 0x02442f to 0x00ef22 (0x1550c bytes) 000947 1787143299.211806 mapread 0x00b8f5 thru 0x00ef22 (0x362e bytes) 000948 1787143299.244865 MIRROR_RESYNC 7 mirrors 000951 1787143300.611105 mapread 0x0095c5 thru 0x00ef22 (0x595e bytes) 000952 1787143300.637153 write 0x009d37 thru 0x015ea5 (0xc16f bytes) 000957 1787143300.726516 trunc from 0x015ea6 to 0x004f28 (0x10f7d bytes) 000959 1787143300.750773 trunc from 0x004f29 to 0x037979 (0x32a51 bytes) 000960 1787143300.770177 write 0x034e49 thru 0x03a130 (0x52e8 bytes) 000961 1787143300.776298 mapwrite 0x02387c thru 0x0318fd (0xe082 bytes) 000963 1787143300.888372 read 0x02a6f8 thru 0x02a99f (0x2a8 bytes) 000964 1787143300.889111 mapwrite 0x020fc8 thru 0x0257f6 (0x482f bytes) 000965 1787143301.081274 read 0x00f67c thru 0x019519 (0x9e9e bytes) 000967 1787143301.109092 write 0x03d0c7 thru 0x03f31d (0x2257 bytes) 000968 1787143301.110131 write 0x005359 thru 0x006d3a (0x19e2 bytes) 000969 1787143301.119072 read 0x004641 thru 0x007ae7 (0x34a7 bytes) 000970 1787143301.120133 write 0x00fa3a thru 0x011275 (0x183c bytes) 000971 1787143301.121900 mapwrite 0x019af1 thru 0x0258f4 (0xbe04 bytes) 000972 1787143301.228088 write 0x01f3a0 thru 0x025654 (0x62b5 bytes) 000973 1787143301.229395 mapwrite 0x00ad99 thru 0x015647 (0xa8af bytes) 000974 1787143301.356535 mapread 0x00ddb4 thru 0x00fa07 (0x1c54 bytes) 000975 1787143301.361722 write 0x020d16 thru 0x0268ff (0x5bea bytes) 000977 1787143301.375128 MIRROR_RESYNC 7 mirrors 000981 1787143303.488359 mapread 0x01f8e1 thru 0x02b688 (0xbda8 bytes) 000983 1787143303.537950 trunc from 0x03f31e to 0x033afe (0xb81f bytes) 000986 1787143303.647371 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33aff): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33aff): Mirror 6: 0xe89b1bc4 (stale) Mirror 10: 0xe89b1bc4 (stale) Mirror 12: 0xe89b1bc4 (stale) Mirror 13: 0xe89b1bc4 (stale) Mirror 14: 0xe89b1bc4 (stale) Mirror 15: 0xe89b1bc4 (stale) CRC-32 checksum value for chunk [0, 0x33aff): Mirror 1: 0xe89b1bc4 Mirror 16: 0xe89b1bc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33aff: skipped 000987 1787143305.126427 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33aff): Mirror 10: 0xe89b1bc4 (stale) Mirror 12: 0xe89b1bc4 (stale) Mirror 13: 0xe89b1bc4 (stale) Mirror 14: 0xe89b1bc4 (stale) Mirror 15: 0xe89b1bc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33aff): Mirror 10: 0xe89b1bc4 (stale) Mirror 12: 0xe89b1bc4 (stale) Mirror 13: 0xe89b1bc4 (stale) Mirror 14: 0xe89b1bc4 (stale) Mirror 15: 0xe89b1bc4 (stale) CRC-32 checksum value for chunk [0, 0x33aff): Mirror 1: 0xe89b1bc4 Mirror 16: 0xe89b1bc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33aff: skipped 000988 1787143306.253216 mapwrite 0x01ff8f thru 0x0262ec (0x635e bytes) 000989 1787143306.646899 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33aff): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33aff): Mirror 10: 0xe89b1bc4 (stale) Mirror 12: 0xe89b1bc4 (stale) Mirror 13: 0xe89b1bc4 (stale) Mirror 14: 0xe89b1bc4 (stale) Mirror 15: 0xe89b1bc4 (stale) Mirror 16: 0xe89b1bc4 (stale) CRC-32 checksum value for chunk [0, 0x33aff): Mirror 1: 0x7f38d85c Mirror 17: 0x7f38d85c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33aff: skipped 000990 1787143307.947082 mapwrite 0x02d353 thru 0x036a3a (0x96e8 bytes) 000991 1787143308.295980 write 0x033f63 thru 0x03ec84 (0xad22 bytes) 000992 1787143308.303558 read 0x025855 thru 0x03434c (0xeaf8 bytes) 000994 1787143308.318048 MIRROR_RESYNC 8 mirrors 000995 1787143309.852974 mapwrite 0x017d2f thru 0x01afc9 (0x329b bytes) 000997 1787143310.064561 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ec85): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec85): Mirror 10: 0x57e427ae (stale) Mirror 12: 0x57e427ae (stale) Mirror 13: 0x57e427ae (stale) Mirror 14: 0x57e427ae (stale) Mirror 15: 0x57e427ae (stale) Mirror 16: 0x57e427ae (stale) Mirror 17: 0x57e427ae (stale) CRC-32 checksum value for chunk [0, 0x3ec85): Mirror 1: 0x715ed785 Mirror 18: 0x715ed785 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec85: skipped 000998 1787143311.402611 mapread 0x02a278 thru 0x032c45 (0x89ce bytes) All operations completed A-OK! PASS 70a (160s) == sanity-flr test 71: check mirror extend parent fid ==== 08:41:58 (1787143318) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 08:42:02 (1787143322) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.449214 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184578 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:42:13 (1787143333) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 885664 .. Extending file size to 2246880 .. done Extending file size to 3503456 .. Extending file size to 7931456 .. Extending file size to 4437408 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8137216 .. failed Extending file size to 8391712 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8417632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9095840 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8564192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8979328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57427 57428 57431 57433 57437 done 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:43:36 (1787143416) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1510208 .. Extending file size to 5220320 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5263616 .. Extending file size to 8712768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8051552 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9116224 .. Extending file size to 9208224 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9386016 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59027 59028 59030 59031 59032 done 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:44:57 (1787143497) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147036 s, 14.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324856 s, 6.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x972dcc98 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:45:14 (1787143514) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0988333 s, 21.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 08:45:22 (1787143522) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:45:29 (1787143529) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24407 s, 8.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:45:38 (1787143538) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236183 s, 8.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:45:47 (1787143547) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156876 s, 13.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:46:19 (1787143579) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107642 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:47:17 (1787143637) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179138 s, 11.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:48:23 (1787143703) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. Creating new pool test_204f oleg424-server: Pool lustre.test_204f created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261325 s, 8.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg424-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:49:42 (1787143782) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:49:49 (1787143789) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:49:57 (1787143797) oleg424-server: Pool lustre.test_206 created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg424-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:50:22 (1787143822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824718 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573886 s, 18.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5dadcb6e Mirror 3: 0x5dadcb6e PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:50:32 (1787143832) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:50:35 (1787143835) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:50:38 (1787143838) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.127732 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xeb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfb0000 Read this (4 bytes): BBBB 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:51:18 (1787143878) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0533306 s, 154 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:51:25 (1787143885) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496505 s, 21.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834445 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 08:51:27.000000000 -0400 Modify: 2026-08-19 08:51:27.000000000 -0400 Change: 2026-08-19 08:51:29.000000000 -0400 Birth: 2026-08-19 08:51:27.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:51:34 (1787143894) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:51:38 (1787143898) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.661564 s, 61.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0868648 s, 47.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 08:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 08:51 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:51:48 (1787143908) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:51:56 (1787143916) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1847 sec =========== 08:52:05 (1787143925) === sanity-flr: start cleanup 08:52:07 (1787143927) === === sanity-flr: finish cleanup 08:52:14 (1787143934) ===