-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 10:49:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg642-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:49:27 (1787582967) === oleg642-client.virtnet: executing check_config_client /mnt/lustre oleg642-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg642-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e60d0014000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e60d0014000.idle_timeout=debug disable quota as required oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:50:00 (1787583000) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 10:50:03 (1787583003) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:50:19 (1787583019) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:50:22 (1787583022) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:50:25 (1787583025) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:50:49 (1787583049) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:50:52 (1787583052) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:50:56 (1787583056) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e60d0014000.stats=clear osc.lustre-OST0001-osc-ffff8e60d0014000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:51:03 (1787583063) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:51:15 (1787583075) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:51:26 (1787583086) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:51:30 (1787583090) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:51:38 (1787583098) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:51:47 (1787583107) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:51:53 (1787583113) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:51:57 (1787583117) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:52:03 (1787583123) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:52:13 (1787583133) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:52:19 (1787583139) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.10659 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767 s, 13.7 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:52:30 (1787583150) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.31711 s, 49.6 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:52:38 (1787583158) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:52:47 (1787583167) Stopping /mnt/lustre-ost1 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg642-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg642-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:53:59 (1787583239) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.357816 s, 20.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg642-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:54:36 (1787583276) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg642-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg642-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:55:59 (1787583359) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.68633 s, 9.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 10:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.306s user 0m0.001s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.252s user 0m0.001s sys 0m0.196s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg642-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg642-server oleg642-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.290s user 0m0.001s sys 0m0.193s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg642-server: oleg642-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg642-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg642-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e60d0014000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:57:34 (1787583454) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:57:37 (1787583457) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:57:41 (1787583461) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:57:44 (1787583464) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:57:52 (1787583472) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.540449 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:58:06 (1787583486) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249978 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269739 s, 38.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:58:14 (1787583494) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:58:22 (1787583502) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00491642 s, 208 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00234026 s, 438 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00248435 s, 412 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00240643 s, 426 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 10:58:33 (1787583513) create test files with size 15752648 .. 17352e6deceb2413dd1e0084c5c0f828 - 052295ffad5ad9f0f449d6a76c4b624d - a38fc4479fbb69de293af84d51309ae7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg642-client.virtnet /mnt/lustre (opts:) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 10:58:59 (1787583539) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:59:17 (1787583557) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:59:23 (1787583563) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115073 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403396 s, 26.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd2:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd3:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:59:32 (1787583572) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229024 s, 9.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.447893 s, 9.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175745 s, 11.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 10:59:50 (1787583590) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 10:59:53 (1787583593) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:59:56 (1787583596) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669517 s, 15.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 11:00:05 (1787583605) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:00:08 (1787583608) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:00:18 (1787583618) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363786 s, 28.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg642-client.virtnet /mnt/lustre (opts:) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:00:30 (1787583630) llite.lustre-ffff8e60c9cde000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.240404 s, 43.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg642-client.virtnet /mnt/lustre (opts:) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e60c9cae800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:00:44 (1787583644) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:01:23 (1787583683) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:01:57 (1787583717) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 11:02:01 (1787583721) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.168057 s, 18.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256107 s, 12.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4ed63a1f (nosync,stale) Mirror 2: 0x4ed63a1f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 66e3b84d4de353e1a9c8eff42a8ebe78 - mirror 1 checksum: 2a13b226ed41ea5f20df976c21e655fc - mirror 2 checksum: 66e3b84d4de353e1a9c8eff42a8ebe78 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T11:02:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T11:02:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 66e3b84d4de353e1a9c8eff42a8ebe78 - mirror 1 checksum: 2a13b226ed41ea5f20df976c21e655fc - mirror 2 checksum: 2a13b226ed41ea5f20df976c21e655fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T11:02:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T11:02:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:02:16 (1787583736) Failing mds1 on oleg642-server Stopping /mnt/lustre-mds1 (opts:) on oleg642-server 11:02:23 (1787583743) shut down facet: mds1 facet_host: oleg642-server facet_failover_host: oleg642-server Failover mds1 to oleg642-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg642-server: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg642-client: oleg642-server: ssh exited with exit code 1 Started lustre-MDT0000 11:02:57 (1787583777) targets are mounted 11:02:57 (1787583777) facet_failover done oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303116 s, 34.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:03:21 (1787583801) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:03:24 (1787583804) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:03:28 (1787583808) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.01922 s, 852 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:03:38 (1787583818) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:03:42 (1787583822) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0267933 s, 153 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.104655 s, 39.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:03:51 (1787583831) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:04:16 (1787583856) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:04:47 (1787583887) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:05:10 (1787583910) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:05:34 (1787583934) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 11:05:40 (1787583940) mirror create pid 54335 mirror split pid 54336 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54335 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54336 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 11:06:47 (1787584007) Seed set to 4008 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x9cfd 000001 1787584009.103578 trunc from 00000000 to 0x009cfc (0x9cfd bytes) 000002 1787584009.124398 read 0x004683 thru 0x009cfc (0x567a bytes) 000003 1787584009.133168 read 0x002e2e thru 0x006dc2 (0x3f95 bytes) 000004 1787584009.154470 mapread 0x005866 thru 0x009cfc (0x4497 bytes) 000005 1787584009.155592 mapwrite 0x029d97 thru 0x03220f (0x8479 bytes) 000010 1787584009.362181 MIRROR_EXTEND to 2 mirrors 000012 1787584010.202732 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000013 1787584010.378961 mapread 0x025e3e thru 0x03220f (0xc3d2 bytes) 000014 1787584010.425158 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x22de3 000015 1787584011.448810 trunc from 0x032210 to 0x022de2 (0xf42d bytes) 000018 1787584011.544230 write 0x019496 thru 0x0261c5 (0xcd30 bytes) 000019 1787584011.552402 write 0x02172d thru 0x030570 (0xee44 bytes) 000021 1787584011.556395 mapread 0x028c87 thru 0x030570 (0x78ea bytes) 000022 1787584011.557529 read 0x00bf48 thru 0x014c7c (0x8d35 bytes) 000025 1787584011.572681 mapread 0x0233b1 thru 0x024c58 (0x18a8 bytes) 000026 1787584011.574325 read 0x0105bd thru 0x018d04 (0x8748 bytes) 000027 1787584011.584142 read 0x0268cc thru 0x02cf0a (0x663f bytes) 000032 1787584011.588574 write 0x02fb35 thru 0x03f74d (0xfc19 bytes) 000033 1787584011.591483 read 0x022ff8 thru 0x023c63 (0xc6c bytes) 000035 1787584011.597063 MIRROR_RESYNC 2 mirrors 000038 1787584012.477067 write 0x02e0c3 thru 0x0303fe (0x233c bytes) truncating to largest ever: 0x3b03b 000039 1787584012.552078 trunc from 0x03f74e to 0x03b03a (0x4713 bytes) 000044 1787584012.582335 trunc from 0x03b03b to 0x01855d (0x22add bytes) 000046 1787584012.605479 trunc from 0x01855e to 0x035df1 (0x1d894 bytes) 000047 1787584012.624336 mapwrite 0x01ad59 thru 0x02581d (0xaac5 bytes) 000049 1787584012.744928 mapread 0x0093fe thru 0x01768d (0xe290 bytes) 000053 1787584012.795416 MIRROR_RESYNC 2 mirrors 000054 1787584013.681644 mapwrite 0x00ea90 thru 0x0142f8 (0x5869 bytes) 000057 1787584013.894955 mapread 0x0180f2 thru 0x0246d4 (0xc5e3 bytes) 000058 1787584013.925086 read 0x014601 thru 0x01c0db (0x7adb bytes) 000060 1787584013.926283 mapwrite 0x023887 thru 0x02ed10 (0xb48a bytes) 000061 1787584014.019542 trunc from 0x035df2 to 0x00b290 (0x2ab61 bytes) 000064 1787584014.041327 mapwrite 0x009daa thru 0x0146ac (0xa903 bytes) 000067 1787584014.123150 MIRROR_RESYNC 2 mirrors 000068 1787584014.813884 read 0x004fb4 thru 0x00d6ea (0x8737 bytes) 000069 1787584014.824120 read 0x005454 thru 0x00a260 (0x4e0d bytes) 000070 1787584014.824797 trunc from 0x0146ad to 0x0103b6 (0x42f6 bytes) 000071 1787584014.924114 trunc from 0x0103b7 to 0x000cb5 (0xf701 bytes) 000073 1787584014.955025 mapread 0x0000a2 thru 0x000cb5 (0xc14 bytes) 000076 1787584014.962709 read 0x000870 thru 0x000cb5 (0x446 bytes) 000077 1787584014.967609 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xcb6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcb6): Mirror 2: 0xc09d743a (stale) CRC-32 checksum value for chunk [0, 0xcb6): Mirror 1: 0xc09d743a Mirror 3: 0xc09d743a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcb6: skipped 000078 1787584015.891073 read 0x00040d thru 0x000cb5 (0x8a9 bytes) 000079 1787584015.905013 write 0x0299ac thru 0x035ae7 (0xc13c bytes) 000080 1787584015.992884 mapwrite 0x019889 thru 0x01c3f6 (0x2b6e bytes) 000082 1787584016.068536 mapwrite 0x02ebbc thru 0x037d87 (0x91cc bytes) 000084 1787584016.179374 write 0x02a83d thru 0x02e827 (0x3feb bytes) 000085 1787584016.185597 read 0x02f247 thru 0x037d87 (0x8b41 bytes) 000090 1787584016.191401 read 0x027fb0 thru 0x02e1f0 (0x6241 bytes) 000091 1787584016.204697 mapread 0x01c9f7 thru 0x01d5fe (0xc08 bytes) 000092 1787584016.207780 write 0x0172a9 thru 0x01fac3 (0x881b bytes) 000093 1787584016.211704 trunc from 0x037d88 to 0x02eb2d (0x925a bytes) 000096 1787584016.243144 write 0x000a4b thru 0x00e432 (0xd9e8 bytes) 000097 1787584016.245519 write 0x032ae0 thru 0x035ab2 (0x2fd3 bytes) 000098 1787584016.246621 write 0x01a35b thru 0x01c7d9 (0x247f bytes) 000100 1787584016.250147 mapwrite 0x039e15 thru 0x03ffff (0x61eb bytes) 000101 1787584016.335902 MIRROR_RESYNC 3 mirrors 000103 1787584017.217447 trunc from 0x040000 to 0x008111 (0x37eee bytes) 000104 1787584017.317712 mapwrite 0x03ae84 thru 0x03ffff (0x517c bytes) 000105 1787584017.410292 write 0x038562 thru 0x03ffff (0x7a9e bytes) 000106 1787584017.419181 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xebddcb14 (stale) Mirror 3: 0xebddcb14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x298b0c3c Mirror 4: 0x298b0c3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000109 1787584018.436331 read 0x01ca1a thru 0x022ede (0x64c5 bytes) 000110 1787584018.443607 read 0x0070d5 thru 0x00fff0 (0x8f1c bytes) 000111 1787584018.454604 read 0x011366 thru 0x0161f4 (0x4e8f bytes) 000112 1787584018.455369 read 0x01a6ea thru 0x025d7b (0xb692 bytes) 000113 1787584018.456101 read 0x024a6e thru 0x025cec (0x127f bytes) 000115 1787584018.457266 write 0x002483 thru 0x00327c (0xdfa bytes) 000116 1787584018.535671 read 0x037aef thru 0x038495 (0x9a7 bytes) 000117 1787584018.541604 mapread 0x00b48b thru 0x00bbfb (0x771 bytes) 000118 1787584018.544385 write 0x00826b thru 0x011f08 (0x9c9e bytes) 000119 1787584018.545734 mapwrite 0x004d88 thru 0x00cd17 (0x7f90 bytes) 000121 1787584018.595991 write 0x010323 thru 0x018793 (0x8471 bytes) 000123 1787584018.598175 write 0x02b785 thru 0x033a9a (0x8316 bytes) 000124 1787584018.600846 mapwrite 0x01c37b thru 0x024ab6 (0x873c bytes) 000126 1787584018.660042 mapwrite 0x0371ef thru 0x03e73c (0x754e bytes) 000129 1787584018.710996 mapwrite 0x013212 thru 0x01c8ba (0x96a9 bytes) 000131 1787584018.809376 mapread 0x02ebc7 thru 0x038baf (0x9fe9 bytes) 000132 1787584018.816343 mapwrite 0x00d024 thru 0x010ff5 (0x3fd2 bytes) 000133 1787584018.949337 trunc from 0x040000 to 0x0389c2 (0x763d bytes) truncating to largest ever: 0x3ba9e 000135 1787584019.063777 trunc from 0x0389c3 to 0x03ba9d (0x30db bytes) 000137 1787584019.114062 mapread 0x015414 thru 0x0198b2 (0x449f bytes) 000140 1787584019.121197 read 0x027660 thru 0x02da5d (0x63fe bytes) 000141 1787584019.146049 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 2: 0x300e6e42 (stale) Mirror 3: 0x300e6e42 (stale) Mirror 4: 0x6d79d7d (stale) CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 1: 0xa928c1f3 Mirror 5: 0xa928c1f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba9e: skipped 000144 1787584019.943664 mapread 0x02d0a2 thru 0x033367 (0x62c6 bytes) 000146 1787584019.971918 mapread 0x038bc1 thru 0x03b66e (0x2aae bytes) 000147 1787584019.985430 write 0x029015 thru 0x032edf (0x9ecb bytes) 000148 1787584020.057312 mapread 0x032345 thru 0x03588f (0x354b bytes) 000149 1787584020.072074 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 2: 0x300e6e42 (stale) Mirror 3: 0x300e6e42 (stale) Mirror 4: 0x6d79d7d (stale) Mirror 5: 0xa928c1f3 (stale) CRC-32 checksum value for chunk [0, 0x3ba9e): Mirror 1: 0xa33b56f1 Mirror 6: 0xa33b56f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba9e: skipped 000150 1787584021.131106 read 0x00001b thru 0x0092ba (0x92a0 bytes) 000153 1787584021.182560 write 0x038bc4 thru 0x03ffff (0x743c bytes) 000154 1787584021.289726 mapread 0x03f386 thru 0x03ffff (0xc7a bytes) 000155 1787584021.295373 trunc from 0x040000 to 0x02f6e2 (0x1091d bytes) 000156 1787584021.372935 read 0x003946 thru 0x00583e (0x1ef9 bytes) 000157 1787584021.392450 MIRROR_RESYNC 6 mirrors 000162 1787584023.070118 trunc from 0x02f6e3 to 0x01e685 (0x1105d bytes) 000163 1787584023.144345 write 0x02f9d2 thru 0x0312e4 (0x1913 bytes) 000164 1787584023.146448 mapread 0x0042b6 thru 0x0098bf (0x560a bytes) 000167 1787584023.182810 trunc from 0x0312e5 to 0x036046 (0x4d62 bytes) 000169 1787584023.210535 mapwrite 0x0174df thru 0x0229d4 (0xb4f6 bytes) 000170 1787584023.304281 mapread 0x020da0 thru 0x02d23d (0xc49e bytes) 000172 1787584023.332591 write 0x000908 thru 0x00a165 (0x985e bytes) 000176 1787584023.341658 read 0x015984 thru 0x024672 (0xecef bytes) 000179 1787584023.343580 trunc from 0x036047 to 0x038ff3 (0x2fad bytes) 000181 1787584023.389416 trunc from 0x038ff4 to 0x00e6c6 (0x2a92d bytes) 000182 1787584023.413183 trunc from 0x00e6c7 to 0x0240c8 (0x15a02 bytes) 000185 1787584023.452535 trunc from 0x0240c9 to 0x01a5b8 (0x9b10 bytes) 000187 1787584023.479391 trunc from 0x01a5b9 to 0x00b20d (0xf3ab bytes) 000188 1787584023.498675 read 0x000429 thru 0x004065 (0x3c3d bytes) 000190 1787584023.500669 write 0x00a7d2 thru 0x01632a (0xbb59 bytes) 000191 1787584023.504627 mapread 0x00f9ff thru 0x01632a (0x692c bytes) 000194 1787584023.521266 mapread 0x005629 thru 0x00f5b9 (0x9f91 bytes) 000196 1787584023.523579 mapwrite 0x01b0ab thru 0x01da58 (0x29ae bytes) 000197 1787584023.600284 trunc from 0x01da59 to 0x038260 (0x1a808 bytes) 000198 1787584023.617644 mapwrite 0x0116d9 thru 0x01f353 (0xdc7b bytes) 000199 1787584023.695566 trunc from 0x038261 to 0x0085e2 (0x2fc7e bytes) 000200 1787584023.718894 mapwrite 0x02c297 thru 0x03578c (0x94f6 bytes) 000202 1787584023.847855 write 0x014701 thru 0x018706 (0x4006 bytes) 000203 1787584023.862574 mapread 0x0121ba thru 0x01354e (0x1395 bytes) 000207 1787584023.896855 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3578d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578d): Mirror 2: 0xf4def77 (stale) Mirror 3: 0xf4def77 (stale) Mirror 4: 0xf4def77 (stale) Mirror 5: 0xf4def77 (stale) Mirror 6: 0xf4def77 (stale) CRC-32 checksum value for chunk [0, 0x3578d): Mirror 1: 0xbb11ed35 Mirror 7: 0xbb11ed35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578d: skipped 000208 1787584025.054362 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3578d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578d): Mirror 2: 0xf4def77 (stale) Mirror 3: 0xf4def77 (stale) Mirror 4: 0xf4def77 (stale) Mirror 5: 0xf4def77 (stale) Mirror 6: 0xf4def77 (stale) CRC-32 checksum value for chunk [0, 0x3578d): Mirror 1: 0xbb11ed35 Mirror 7: 0xbb11ed35 Mirror 8: 0xbb11ed35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578d: skipped 000209 1787584026.313962 trunc from 0x03578d to 0x033de4 (0x19a8 bytes) 000211 1787584026.486392 write 0x039a46 thru 0x03b959 (0x1f14 bytes) 000212 1787584026.494250 MIRROR_RESYNC 8 mirrors 000216 1787584028.149156 mapread 0x0039d8 thru 0x008e3d (0x5466 bytes) 000220 1787584028.171639 write 0x0053f4 thru 0x011667 (0xc274 bytes) 000221 1787584028.282816 mapwrite 0x00a4ac thru 0x010bdf (0x6734 bytes) 000222 1787584028.387249 mapread 0x02a8bd thru 0x02e915 (0x4059 bytes) 000223 1787584028.415058 trunc from 0x03b95a to 0x00ce8d (0x2eacc bytes) 000224 1787584028.447289 write 0x015103 thru 0x01fcd3 (0xabd1 bytes) 000228 1787584028.453306 mapwrite 0x01c4c7 thru 0x029392 (0xcecc bytes) 000229 1787584028.562964 mapwrite 0x00e425 thru 0x010f61 (0x2b3d bytes) 000233 1787584028.617221 read 0x00a716 thru 0x013518 (0x8e03 bytes) 000234 1787584028.618435 mapwrite 0x0100b0 thru 0x014409 (0x435a bytes) 000237 1787584028.662866 trunc from 0x029393 to 0x022023 (0x736f bytes) 000239 1787584028.686015 trunc from 0x022024 to 0x02f9d6 (0xd9b3 bytes) 000241 1787584028.707255 trunc from 0x02f9d7 to 0x01e171 (0x11865 bytes) 000242 1787584028.734465 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e172): Mirror 2: 0x19210bd2 (stale) Mirror 3: 0x19210bd2 (stale) Mirror 5: 0x19210bd2 (stale) Mirror 6: 0x19210bd2 (stale) Mirror 7: 0x19210bd2 (stale) Mirror 8: 0x19210bd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e172): Mirror 2: 0x19210bd2 (stale) Mirror 3: 0x19210bd2 (stale) Mirror 5: 0x19210bd2 (stale) Mirror 6: 0x19210bd2 (stale) Mirror 7: 0x19210bd2 (stale) Mirror 8: 0x19210bd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e172: skipped 000243 1787584029.375332 mapwrite 0x036442 thru 0x03ffff (0x9bbe bytes) 000245 1787584029.468476 trunc from 0x040000 to 0x02f03a (0x10fc5 bytes) 000246 1787584029.503706 write 0x032910 thru 0x03bd02 (0x93f3 bytes) 000247 1787584029.521317 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: 0x102e2f88 (stale) Mirror 5: 0x102e2f88 (stale) Mirror 6: 0x102e2f88 (stale) Mirror 7: 0x102e2f88 (stale) Mirror 8: 0x102e2f88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: 0x102e2f88 (stale) Mirror 5: 0x102e2f88 (stale) Mirror 6: 0x102e2f88 (stale) Mirror 7: 0x102e2f88 (stale) Mirror 8: 0x102e2f88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd03: skipped 000250 1787584030.009581 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: 0x102e2f88 (stale) Mirror 5: 0x102e2f88 (stale) Mirror 6: 0x102e2f88 (stale) Mirror 8: 0x102e2f88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: 0x102e2f88 (stale) Mirror 5: 0x102e2f88 (stale) Mirror 6: 0x102e2f88 (stale) Mirror 8: 0x102e2f88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd03: skipped 000251 1787584030.500046 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 3: 0x102e2f88 (stale) Mirror 5: 0x102e2f88 (stale) Mirror 6: 0x102e2f88 (stale) Mirror 8: 0x102e2f88 (stale) CRC-32 checksum value for chunk [0, 0x3bd03): Mirror 1: 0x6851e82a Mirror 9: 0x6851e82a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd03: skipped 000252 1787584031.651219 write 0x015e3f thru 0x017e7f (0x2041 bytes) 000254 1787584031.775864 trunc from 0x03bd03 to 0x020216 (0x1baec bytes) 000256 1787584031.813277 write 0x03980f thru 0x03ffff (0x67f1 bytes) 000257 1787584031.815730 trunc from 0x040000 to 0x00622a (0x39dd5 bytes) 000258 1787584031.834388 trunc from 0x00622b to 0x0012da (0x4f50 bytes) 000259 1787584031.850520 write 0x034940 thru 0x03be34 (0x74f5 bytes) 000260 1787584031.853229 trunc from 0x03be35 to 0x020758 (0x1b6dc bytes) 000261 1787584031.872696 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x20759): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20759): Mirror 3: 0x793f717e (stale) Mirror 5: 0x793f717e (stale) Mirror 6: 0x793f717e (stale) Mirror 8: 0x793f717e (stale) Mirror 9: 0x43f618ad (stale) CRC-32 checksum value for chunk [0, 0x20759): Mirror 1: 0xcb0dc70a Mirror 10: 0xcb0dc70a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20759: skipped 000262 1787584032.939637 mapread 0x00858d thru 0x0102ae (0x7d22 bytes) 000264 1787584032.959547 read 0x00e42c thru 0x00e977 (0x54c bytes) 000266 1787584032.960332 write 0x0378b1 thru 0x03ffff (0x874f bytes) 000268 1787584033.055069 read 0x016215 thru 0x025ae0 (0xf8cc bytes) 000269 1787584033.076157 trunc from 0x040000 to 0x007b26 (0x384d9 bytes) 000271 1787584033.114487 MIRROR_RESYNC 7 mirrors 000272 1787584034.576848 trunc from 0x007b27 to 0x009c43 (0x211d bytes) 000273 1787584034.672311 trunc from 0x009c44 to 0x01d33f (0x136fc bytes) 000275 1787584034.695915 mapwrite 0x0395a0 thru 0x03ffff (0x6a60 bytes) 000276 1787584034.839212 read 0x03dd4b thru 0x03ffff (0x22b5 bytes) 000279 1787584034.841573 trunc from 0x040000 to 0x02f57f (0x10a80 bytes) 000281 1787584034.868210 MIRROR_RESYNC 7 mirrors 000283 1787584036.342728 mapwrite 0x00b584 thru 0x015c63 (0xa6e0 bytes) 000285 1787584036.551118 write 0x01313b thru 0x01713d (0x4003 bytes) 000287 1787584036.554057 mapread 0x01d5a1 thru 0x0234f6 (0x5f56 bytes) 000289 1787584036.583115 read 0x01bf68 thru 0x028ac7 (0xcb60 bytes) 000290 1787584036.595349 MIRROR_RESYNC 7 mirrors 000291 1787584038.008952 trunc from 0x02f580 to 0x0296ca (0x5eb5 bytes) 000292 1787584038.113246 write 0x039d06 thru 0x03ffff (0x62fa bytes) 000294 1787584038.115689 write 0x02a4ba thru 0x033b96 (0x96dd bytes) 000295 1787584038.123482 read 0x005f72 thru 0x00e17c (0x820b bytes) 000297 1787584038.133155 mapread 0x001d07 thru 0x004746 (0x2a40 bytes) 000299 1787584038.155454 MIRROR_RESYNC 7 mirrors 000301 1787584039.337931 read 0x035ec4 thru 0x03ffff (0xa13c bytes) 000302 1787584039.362202 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) 000304 1787584039.781524 write 0x01f290 thru 0x01f73b (0x4ac bytes) 000305 1787584039.893521 write 0x02aad9 thru 0x02f872 (0x4d9a bytes) 000306 1787584039.911496 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa02626ff (stale) Mirror 6: 0xa02626ff (stale) Mirror 8: 0xa02626ff (stale) Mirror 9: 0xa02626ff (stale) Mirror 10: 0xa02626ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7dcf235b Mirror 11: 0x7dcf235b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000308 1787584040.931542 mapread 0x01e303 thru 0x02ce6f (0xeb6d bytes) 000309 1787584040.988946 trunc from 0x040000 to 0x011ca7 (0x2e358 bytes) 000310 1787584041.157283 read 0x003cb7 thru 0x0095dc (0x5926 bytes) 000311 1787584041.188061 read 0x00cc22 thru 0x011ca7 (0x5086 bytes) 000313 1787584041.220350 write 0x01df71 thru 0x0260cf (0x815f bytes) 000314 1787584041.224151 write 0x000453 thru 0x001d4f (0x18fd bytes) 000315 1787584041.226122 read 0x007046 thru 0x010506 (0x94c1 bytes) 000317 1787584041.228781 trunc from 0x0260d0 to 0x01f3a4 (0x6d2b bytes) 000318 1787584041.265928 mapread 0x00efcf thru 0x0150b8 (0x60ea bytes) 000319 1787584041.283871 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1f3a5): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f3a5): Mirror 5: 0x7586b699 (stale) Mirror 6: 0x7586b699 (stale) Mirror 8: 0x7586b699 (stale) Mirror 9: 0x7586b699 (stale) Mirror 10: 0x7586b699 (stale) Mirror 11: 0x4dac014e (stale) CRC-32 checksum value for chunk [0, 0x1f3a5): Mirror 1: 0xf102861e Mirror 12: 0xf102861e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f3a5: skipped 000320 1787584042.248828 read 0x018e5f thru 0x01f3a4 (0x6546 bytes) 000321 1787584042.258120 mapwrite 0x02e2be thru 0x0345e7 (0x632a bytes) 000322 1787584042.415105 read 0x011835 thru 0x017f73 (0x673f bytes) 000323 1787584042.426972 mapwrite 0x00671a thru 0x00ae02 (0x46e9 bytes) 000324 1787584042.487429 MIRROR_RESYNC 8 mirrors 000326 1787584044.122340 write 0x027658 thru 0x034b07 (0xd4b0 bytes) 000327 1787584044.240990 mapwrite 0x00cee3 thru 0x01a4b0 (0xd5ce bytes) 000328 1787584044.349853 write 0x03c136 thru 0x03ffff (0x3eca bytes) 000331 1787584044.356575 read 0x038836 thru 0x03ac3d (0x2408 bytes) 000332 1787584044.362935 mapread 0x013519 thru 0x017288 (0x3d70 bytes) 000336 1787584044.367281 read 0x0236e7 thru 0x02c647 (0x8f61 bytes) 000338 1787584044.382257 mapwrite 0x0027f0 thru 0x00ab8d (0x839e bytes) 000340 1787584044.439893 mapwrite 0x01458f thru 0x02042e (0xbea0 bytes) 000342 1787584044.490877 mapwrite 0x026195 thru 0x0303f6 (0xa262 bytes) 000344 1787584044.552536 mapread 0x0236bf thru 0x028876 (0x51b8 bytes) 000349 1787584044.561943 trunc from 0x040000 to 0x00ce3d (0x331c2 bytes) 000350 1787584044.609461 write 0x03e8b9 thru 0x03ffff (0x1747 bytes) 000351 1787584044.611988 read 0x019432 thru 0x020e56 (0x7a25 bytes) 000353 1787584044.640931 write 0x01d3c4 thru 0x02266c (0x52a9 bytes) 000355 1787584044.644219 mapread 0x001ec8 thru 0x00887b (0x69b4 bytes) 000356 1787584044.646217 read 0x03e1e4 thru 0x03ffff (0x1e1c bytes) 000357 1787584044.647577 mapwrite 0x02a3a0 thru 0x02cae5 (0x2746 bytes) 000358 1787584044.735293 read 0x039c6c thru 0x03e78c (0x4b21 bytes) 000359 1787584044.736306 write 0x02574c thru 0x02c7f8 (0x70ad bytes) 000362 1787584044.753602 mapread 0x014775 thru 0x0204c7 (0xbd53 bytes) truncating to largest ever: 0x3e910 000363 1787584044.756975 trunc from 0x040000 to 0x03e90f (0x16f0 bytes) 000365 1787584044.820565 read 0x0113af thru 0x018357 (0x6fa9 bytes) 000366 1787584044.823586 mapwrite 0x00469b thru 0x0135e4 (0xef4a bytes) 000367 1787584044.926527 mapwrite 0x018018 thru 0x01c334 (0x431d bytes) 000368 1787584044.971328 mapread 0x01f6c5 thru 0x0215f4 (0x1f30 bytes) 000369 1787584044.972279 write 0x03bba9 thru 0x03ffff (0x4457 bytes) 000370 1787584044.974344 mapread 0x0214c8 thru 0x027ef9 (0x6a32 bytes) 000373 1787584044.978288 mapread 0x02685c thru 0x0337a0 (0xcf45 bytes) 000374 1787584044.980094 mapwrite 0x017875 thru 0x020d5e (0x94ea bytes) 000375 1787584045.037190 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4cce25e7 (stale) Mirror 6: 0x4cce25e7 (stale) Mirror 8: 0x4cce25e7 (stale) Mirror 9: 0x4cce25e7 (stale) Mirror 11: 0x4cce25e7 (stale) Mirror 12: 0x4cce25e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4cce25e7 (stale) Mirror 6: 0x4cce25e7 (stale) Mirror 8: 0x4cce25e7 (stale) Mirror 9: 0x4cce25e7 (stale) Mirror 11: 0x4cce25e7 (stale) Mirror 12: 0x4cce25e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1787584045.717936 write 0x0301a3 thru 0x032623 (0x2481 bytes) 000377 1787584045.728798 mapread 0x03045b thru 0x03e9cd (0xe573 bytes) 000378 1787584045.792270 MIRROR_RESYNC 7 mirrors 000379 1787584047.244207 mapwrite 0x03f76a thru 0x03ffff (0x896 bytes) 000380 1787584047.374462 mapread 0x026e14 thru 0x02d965 (0x6b52 bytes) 000381 1787584047.387163 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x57d8ed28 (stale) Mirror 6: 0x57d8ed28 (stale) Mirror 8: 0x57d8ed28 (stale) Mirror 9: 0x57d8ed28 (stale) Mirror 11: 0x57d8ed28 (stale) Mirror 12: 0x57d8ed28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3f9832d8 Mirror 13: 0x3f9832d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1787584048.127306 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x57d8ed28 (stale) Mirror 6: 0x57d8ed28 (stale) Mirror 8: 0x57d8ed28 (stale) Mirror 9: 0x57d8ed28 (stale) Mirror 11: 0x57d8ed28 (stale) Mirror 12: 0x57d8ed28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3f9832d8 Mirror 13: 0x3f9832d8 Mirror 14: 0x3f9832d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000383 1787584049.082290 read 0x03431f thru 0x03fb5e (0xb840 bytes) 000384 1787584049.093217 write 0x0082a5 thru 0x00d028 (0x4d84 bytes) 000385 1787584049.173116 trunc from 0x040000 to 0x01bb1f (0x244e0 bytes) 000386 1787584049.197212 trunc from 0x01bb20 to 0x02c1dd (0x106be bytes) 000387 1787584049.227516 read 0x01cf32 thru 0x02c1dd (0xf2ac bytes) 000388 1787584049.241117 write 0x01c2f9 thru 0x01c8c0 (0x5c8 bytes) 000389 1787584049.242995 write 0x03003a thru 0x03a4fc (0xa4c3 bytes) 000391 1787584049.246318 write 0x021be6 thru 0x0316b6 (0xfad1 bytes) 000394 1787584049.250298 read 0x001feb thru 0x007156 (0x516c bytes) 000395 1787584049.255140 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a4fd): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4fd): Mirror 5: 0x7004a36a (stale) Mirror 6: 0x7004a36a (stale) Mirror 8: 0x7004a36a (stale) Mirror 9: 0x7004a36a (stale) Mirror 11: 0x7004a36a (stale) Mirror 12: 0x7004a36a (stale) Mirror 13: 0x7004a36a (stale) Mirror 14: 0x7004a36a (stale) CRC-32 checksum value for chunk [0, 0x3a4fd): Mirror 1: 0x3781225 Mirror 15: 0x3781225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4fd: skipped 000396 1787584050.232626 read 0x03322e thru 0x03775a (0x452d bytes) 000398 1787584050.252425 write 0x003c3c thru 0x012727 (0xeaec bytes) 000399 1787584050.364131 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a4fd): Mirror 5: 0x7004a36a (stale) Mirror 6: 0x7004a36a (stale) Mirror 8: 0x7004a36a (stale) Mirror 9: 0x7004a36a (stale) Mirror 11: 0x7004a36a (stale) Mirror 12: 0x7004a36a (stale) Mirror 14: 0x7004a36a (stale) Mirror 15: 0x3781225 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4fd): Mirror 5: 0x7004a36a (stale) Mirror 6: 0x7004a36a (stale) Mirror 8: 0x7004a36a (stale) Mirror 9: 0x7004a36a (stale) Mirror 11: 0x7004a36a (stale) Mirror 12: 0x7004a36a (stale) Mirror 14: 0x7004a36a (stale) Mirror 15: 0x3781225 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4fd: skipped 000404 1787584050.876221 mapread 0x0378f5 thru 0x03a4fc (0x2c08 bytes) 000405 1787584050.894805 mapread 0x030554 thru 0x0327a4 (0x2251 bytes) 000408 1787584050.905113 read 0x038465 thru 0x03a4fc (0x2098 bytes) 000411 1787584050.907239 trunc from 0x03a4fd to 0x02bbeb (0xe911 bytes) 000413 1787584050.953591 mapwrite 0x023298 thru 0x02cd5d (0x9ac6 bytes) 000414 1787584051.104866 read 0x00f9d4 thru 0x017291 (0x78be bytes) 000415 1787584051.141546 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2cd5e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd5e): Mirror 5: 0x23c6e470 (stale) Mirror 6: 0x23c6e470 (stale) Mirror 8: 0x23c6e470 (stale) Mirror 9: 0x23c6e470 (stale) Mirror 11: 0x23c6e470 (stale) Mirror 12: 0x23c6e470 (stale) Mirror 14: 0x23c6e470 (stale) Mirror 15: 0x23c9af90 (stale) CRC-32 checksum value for chunk [0, 0x2cd5e): Mirror 1: 0x74bbb18a Mirror 16: 0x74bbb18a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd5e: skipped 000418 1787584052.271427 read 0x001577 thru 0x006f88 (0x5a12 bytes) 000423 1787584052.366062 trunc from 0x02cd5e to 0x02f02a (0x22cd bytes) 000424 1787584052.537316 mapwrite 0x03dfae thru 0x03ffff (0x2052 bytes) 000429 1787584052.638479 trunc from 0x040000 to 0x00bf76 (0x34089 bytes) 000431 1787584052.666549 MIRROR_RESYNC 10 mirrors 000432 1787584054.205147 write 0x013082 thru 0x01654e (0x34cd bytes) 000434 1787584054.286416 read 0x00bde0 thru 0x01654e (0xa76f bytes) 000436 1787584054.297894 mapwrite 0x01aeee thru 0x01d856 (0x2969 bytes) 000439 1787584054.422881 mapread 0x013401 thru 0x0193a8 (0x5fa8 bytes) 000440 1787584054.424352 trunc from 0x01d857 to 0x037b62 (0x1a30c bytes) 000442 1787584054.448269 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b63): Mirror 5: 0xaf5ee18e (stale) Mirror 8: 0xaf5ee18e (stale) Mirror 9: 0xaf5ee18e (stale) Mirror 11: 0xaf5ee18e (stale) Mirror 12: 0xaf5ee18e (stale) Mirror 14: 0xaf5ee18e (stale) Mirror 15: 0xaf5ee18e (stale) Mirror 16: 0xaf5ee18e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b63): Mirror 5: 0xaf5ee18e (stale) Mirror 8: 0xaf5ee18e (stale) Mirror 9: 0xaf5ee18e (stale) Mirror 11: 0xaf5ee18e (stale) Mirror 12: 0xaf5ee18e (stale) Mirror 14: 0xaf5ee18e (stale) Mirror 15: 0xaf5ee18e (stale) Mirror 16: 0xaf5ee18e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b63: skipped 000443 1787584054.925554 mapwrite 0x03fa38 thru 0x03ffff (0x5c8 bytes) 000448 1787584054.988279 read 0x035f66 thru 0x03a7de (0x4879 bytes) 000449 1787584054.996417 mapwrite 0x0092a0 thru 0x00eb28 (0x5889 bytes) 000451 1787584055.075622 mapwrite 0x026752 thru 0x02a423 (0x3cd2 bytes) 000453 1787584055.138493 mapread 0x02c4e7 thru 0x03a65e (0xe178 bytes) 000454 1787584055.152228 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf9b8752e (stale) Mirror 8: 0xf9b8752e (stale) Mirror 9: 0xf9b8752e (stale) Mirror 11: 0xf9b8752e (stale) Mirror 12: 0xf9b8752e (stale) Mirror 14: 0xf9b8752e (stale) Mirror 15: 0xf9b8752e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf9b8752e (stale) Mirror 8: 0xf9b8752e (stale) Mirror 9: 0xf9b8752e (stale) Mirror 11: 0xf9b8752e (stale) Mirror 12: 0xf9b8752e (stale) Mirror 14: 0xf9b8752e (stale) Mirror 15: 0xf9b8752e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1787584055.647174 mapread 0x036ec2 thru 0x03ffff (0x913e bytes) 000456 1787584055.678308 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf9b8752e (stale) Mirror 8: 0xf9b8752e (stale) Mirror 9: 0xf9b8752e (stale) Mirror 11: 0xf9b8752e (stale) Mirror 12: 0xf9b8752e (stale) Mirror 14: 0xf9b8752e (stale) Mirror 15: 0xf9b8752e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf86cc20b Mirror 16: 0xf86cc20b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1787584056.801928 mapwrite 0x00c527 thru 0x00dc58 (0x1732 bytes) 000460 1787584056.992092 read 0x02d32f thru 0x02daf9 (0x7cb bytes) 000461 1787584056.998486 write 0x027704 thru 0x02d18b (0x5a88 bytes) 000462 1787584057.005000 mapread 0x00f537 thru 0x01445a (0x4f24 bytes) 000463 1787584057.010502 read 0x00bd1b thru 0x012c2b (0x6f11 bytes) 000464 1787584057.015050 MIRROR_RESYNC 9 mirrors 000466 1787584058.628069 mapread 0x0187ee thru 0x02119b (0x89ae bytes) 000477 1787584058.662387 trunc from 0x040000 to 0x00fa3e (0x305c1 bytes) 000478 1787584058.803567 mapread 0x009263 thru 0x00ca38 (0x37d6 bytes) 000479 1787584058.816530 mapwrite 0x002ad0 thru 0x00c973 (0x9ea4 bytes) 000481 1787584058.890493 trunc from 0x00fa3f to 0x03b6bc (0x2bc7e bytes) 000482 1787584058.913024 read 0x02c17d thru 0x0335fb (0x747f bytes) 000489 1787584058.931157 read 0x010db7 thru 0x0195de (0x8828 bytes) 000491 1787584058.957819 write 0x00805c thru 0x013fe8 (0xbf8d bytes) 000494 1787584058.966912 write 0x008a9d thru 0x00ce89 (0x43ed bytes) 000495 1787584058.968553 read 0x0051c2 thru 0x008603 (0x3442 bytes) 000496 1787584058.974061 read 0x020e58 thru 0x02f5b4 (0xe75d bytes) 000497 1787584058.976480 write 0x014d4a thru 0x01cb41 (0x7df8 bytes) 000498 1787584058.979973 mapread 0x0167cb thru 0x023dc6 (0xd5fc bytes) 000500 1787584058.981501 write 0x03b1cb thru 0x03ffff (0x4e35 bytes) 000501 1787584058.983814 trunc from 0x040000 to 0x003900 (0x3c6ff bytes) 000503 1787584059.007310 trunc from 0x003901 to 0x0169f4 (0x130f4 bytes) 000504 1787584059.034448 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 8: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) Mirror 16: 0x23339f22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 8: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) Mirror 16: 0x23339f22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169f5: skipped 000508 1787584059.631235 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) Mirror 16: 0x23339f22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) Mirror 16: 0x23339f22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169f5: skipped 000509 1787584060.109137 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169f5): Mirror 5: 0x23339f22 (stale) Mirror 11: 0x23339f22 (stale) Mirror 12: 0x23339f22 (stale) Mirror 14: 0x23339f22 (stale) Mirror 15: 0x23339f22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169f5: skipped 000510 1787584060.534116 MIRROR_RESYNC 6 mirrors 000511 1787584061.878487 mapread 0x00b4e3 thru 0x00b9e6 (0x504 bytes) 000512 1787584061.883455 mapwrite 0x02ce28 thru 0x02d146 (0x31f bytes) 000514 1787584062.074217 write 0x02e453 thru 0x03732f (0x8edd bytes) 000515 1787584062.076418 read 0x0344d0 thru 0x03732f (0x2e60 bytes) 000517 1787584062.078223 read 0x00240c thru 0x003ff4 (0x1be9 bytes) 000520 1787584062.095333 trunc from 0x037330 to 0x001143 (0x361ec bytes) 000521 1787584062.125572 read 0x000caa thru 0x001143 (0x49a bytes) 000522 1787584062.126309 mapread 0x0003d8 thru 0x001143 (0xd6c bytes) 000523 1787584062.129570 mapread 0x000257 thru 0x001143 (0xeed bytes) 000524 1787584062.131100 read 0x000eac thru 0x001143 (0x298 bytes) 000525 1787584062.132576 write 0x00ac56 thru 0x019b3f (0xeeea bytes) 000526 1787584062.145115 MIRROR_RESYNC 6 mirrors 000527 1787584063.309670 mapwrite 0x0260aa thru 0x034be4 (0xeb3b bytes) 000530 1787584063.554678 trunc from 0x034be5 to 0x02ec93 (0x5f51 bytes) 000531 1787584063.578182 write 0x03bfe0 thru 0x03ffff (0x4020 bytes) 000533 1787584063.582284 write 0x01cd4c thru 0x02ab0e (0xddc3 bytes) 000534 1787584063.590515 write 0x00a25a thru 0x011bcf (0x7976 bytes) 000535 1787584063.602058 mapwrite 0x02993a thru 0x02c2bb (0x2982 bytes) 000541 1787584063.662423 mapread 0x02804c thru 0x0292fd (0x12b2 bytes) 000542 1787584063.668639 MIRROR_RESYNC 6 mirrors 000545 1787584064.795384 mapwrite 0x033405 thru 0x03ffff (0xcbfb bytes) 000546 1787584065.016512 trunc from 0x040000 to 0x02305d (0x1cfa2 bytes) 000547 1787584065.044380 mapwrite 0x0091fd thru 0x00ff57 (0x6d5b bytes) 000550 1787584065.111476 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2305e): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2305e): Mirror 5: 0xbd845ca4 (stale) Mirror 11: 0xbd845ca4 (stale) Mirror 12: 0xbd845ca4 (stale) Mirror 14: 0xbd845ca4 (stale) Mirror 15: 0xbd845ca4 (stale) CRC-32 checksum value for chunk [0, 0x2305e): Mirror 1: 0xcf949fb6 Mirror 16: 0xcf949fb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2305e: skipped 000551 1787584066.031826 read 0x01e770 thru 0x01f5da (0xe6b bytes) 000552 1787584066.043070 trunc from 0x02305e to 0x00abe0 (0x1847d bytes) 000553 1787584066.211308 mapwrite 0x0364f1 thru 0x03ffff (0x9b0f bytes) 000554 1787584066.387330 read 0x035455 thru 0x03c812 (0x73be bytes) 000555 1787584066.388221 mapread 0x02f126 thru 0x030f31 (0x1e0c bytes) 000556 1787584066.427995 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8c5db345 (stale) Mirror 11: 0x8c5db345 (stale) Mirror 12: 0x8c5db345 (stale) Mirror 14: 0x8c5db345 (stale) Mirror 15: 0x8c5db345 (stale) Mirror 16: 0x51aa3cd4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b09f24e Mirror 17: 0x1b09f24e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000557 1787584067.451029 write 0x029002 thru 0x02f74e (0x674d bytes) 000559 1787584067.554281 trunc from 0x040000 to 0x0215e3 (0x1ea1c bytes) 000562 1787584067.606601 mapwrite 0x0045d7 thru 0x013873 (0xf29d bytes) 000564 1787584067.723385 trunc from 0x0215e4 to 0x037c22 (0x1663f bytes) 000565 1787584067.785123 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37c23): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c23): Mirror 5: 0xc6fc2082 (stale) Mirror 11: 0xc6fc2082 (stale) Mirror 12: 0xc6fc2082 (stale) Mirror 14: 0xc6fc2082 (stale) Mirror 15: 0xc6fc2082 (stale) Mirror 16: 0x47ed7cdb (stale) Mirror 17: 0x4d42d591 (stale) CRC-32 checksum value for chunk [0, 0x37c23): Mirror 1: 0x75ffa950 Mirror 18: 0x75ffa950 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c23: skipped 000566 1787584068.920623 mapread 0x032aaa thru 0x037c22 (0x5179 bytes) 000568 1787584068.984993 write 0x00472b thru 0x00e3df (0x9cb5 bytes) 000571 1787584069.083616 read 0x02dc06 thru 0x02dc91 (0x8c bytes) 000572 1787584069.094097 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c23): Mirror 11: 0xc6fc2082 (stale) Mirror 12: 0xc6fc2082 (stale) Mirror 14: 0xc6fc2082 (stale) Mirror 15: 0xc6fc2082 (stale) Mirror 16: 0x47ed7cdb (stale) Mirror 17: 0x4d42d591 (stale) Mirror 18: 0x75ffa950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c23): Mirror 11: 0xc6fc2082 (stale) Mirror 12: 0xc6fc2082 (stale) Mirror 14: 0xc6fc2082 (stale) Mirror 15: 0xc6fc2082 (stale) Mirror 16: 0x47ed7cdb (stale) Mirror 17: 0x4d42d591 (stale) Mirror 18: 0x75ffa950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c23: skipped 000573 1787584069.685046 write 0x02aeae thru 0x02fde6 (0x4f39 bytes) 000574 1787584069.705865 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c23): Mirror 11: 0xc6fc2082 (stale) Mirror 12: 0xc6fc2082 (stale) Mirror 14: 0xc6fc2082 (stale) Mirror 15: 0xc6fc2082 (stale) Mirror 16: 0x47ed7cdb (stale) Mirror 17: 0x4d42d591 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c23): Mirror 11: 0xc6fc2082 (stale) Mirror 12: 0xc6fc2082 (stale) Mirror 14: 0xc6fc2082 (stale) Mirror 15: 0xc6fc2082 (stale) Mirror 16: 0x47ed7cdb (stale) Mirror 17: 0x4d42d591 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c23: skipped 000575 1787584070.171821 mapread 0x0373a5 thru 0x037c22 (0x87e bytes) 000577 1787584070.179802 write 0x02dc9c thru 0x02ecc2 (0x1027 bytes) 000578 1787584070.196564 MIRROR_RESYNC 7 mirrors 000580 1787584071.632390 mapread 0x0059bb thru 0x01534f (0xf995 bytes) 000581 1787584071.675249 write 0x03e612 thru 0x03e6ea (0xd9 bytes) 000582 1787584071.757230 mapread 0x009461 thru 0x013caf (0xa84f bytes) 000583 1787584071.790077 write 0x02f32c thru 0x03469d (0x5372 bytes) 000584 1787584071.806068 MIRROR_RESYNC 7 mirrors 000585 1787584072.950363 mapread 0x032fc8 thru 0x03bd34 (0x8d6d bytes) 000586 1787584072.965849 write 0x038552 thru 0x0389d9 (0x488 bytes) 000587 1787584073.072153 mapread 0x027888 thru 0x02ac27 (0x33a0 bytes) 000588 1787584073.081652 mapread 0x03be63 thru 0x03e5cb (0x2769 bytes) 000589 1787584073.092445 mapread 0x003528 thru 0x007fd0 (0x4aa9 bytes) 000590 1787584073.105372 MIRROR_RESYNC 7 mirrors 000592 1787584074.631574 read 0x028741 thru 0x033430 (0xacf0 bytes) 000594 1787584074.650630 trunc from 0x03e6eb to 0x00f5ff (0x2f0eb bytes) 000596 1787584074.735228 mapwrite 0x0179e8 thru 0x0218d1 (0x9eea bytes) 000597 1787584074.910329 mapwrite 0x0050a6 thru 0x00f201 (0xa15c bytes) 000598 1787584074.963454 write 0x01544c thru 0x015c62 (0x817 bytes) 000599 1787584074.965138 read 0x013379 thru 0x01fdb8 (0xca40 bytes) 000600 1787584074.966368 mapread 0x0080bb thru 0x013e6b (0xbdb1 bytes) 000601 1787584074.977440 read 0x01582e thru 0x0218d1 (0xc0a4 bytes) 000603 1787584074.979642 write 0x03b87e thru 0x03ffff (0x4782 bytes) 000605 1787584074.982296 mapread 0x0028d0 thru 0x00be73 (0x95a4 bytes) 000606 1787584074.989366 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 12: 0x349b5282 (stale) Mirror 14: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 12: 0x349b5282 (stale) Mirror 14: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1787584075.514005 write 0x03982a thru 0x03ffff (0x67d6 bytes) 000612 1787584075.524150 trunc from 0x040000 to 0x023eac (0x1c153 bytes) 000613 1787584075.556408 mapread 0x00c3c4 thru 0x013d43 (0x7980 bytes) 000614 1787584075.592598 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23ead): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ead): Mirror 11: 0x28e0de4a (stale) Mirror 12: 0x28e0de4a (stale) Mirror 14: 0x28e0de4a (stale) Mirror 15: 0x28e0de4a (stale) Mirror 16: 0x28e0de4a (stale) CRC-32 checksum value for chunk [0, 0x23ead): Mirror 1: 0xc3ae917b Mirror 17: 0xc3ae917b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ead: skipped 000617 1787584076.710981 mapwrite 0x02234b thru 0x02a4f0 (0x81a6 bytes) 000618 1787584076.957490 trunc from 0x02a4f1 to 0x0336c0 (0x91d0 bytes) 000619 1787584076.983446 read 0x00a0d4 thru 0x017bf7 (0xdb24 bytes) 000620 1787584076.996100 read 0x015590 thru 0x0248ec (0xf35d bytes) 000621 1787584077.016326 write 0x0168c9 thru 0x01db7e (0x72b6 bytes) 000623 1787584077.029020 write 0x03b676 thru 0x03ffff (0x498a bytes) 000624 1787584077.035085 read 0x03a542 thru 0x03ffff (0x5abe bytes) 000626 1787584077.054125 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 12: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) Mirror 17: 0xa984188a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 12: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) Mirror 17: 0xa984188a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000628 1787584077.536225 trunc from 0x040000 to 0x01a5c1 (0x25a3e bytes) 000629 1787584077.555168 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a5c2): Mirror 11: 0x946dd0aa (stale) Mirror 15: 0x946dd0aa (stale) Mirror 16: 0x946dd0aa (stale) Mirror 17: 0x3b30adfb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a5c2): Mirror 11: 0x946dd0aa (stale) Mirror 15: 0x946dd0aa (stale) Mirror 16: 0x946dd0aa (stale) Mirror 17: 0x3b30adfb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a5c2: skipped 000630 1787584077.933567 trunc from 0x01a5c2 to 0x0341bb (0x19bfa bytes) 000631 1787584077.949075 trunc from 0x0341bc to 0x02997e (0xa83d bytes) 000632 1787584077.967028 mapwrite 0x02f336 thru 0x03e143 (0xee0e bytes) 000633 1787584078.144504 mapread 0x010df1 thru 0x018d91 (0x7fa1 bytes) 000635 1787584078.160622 read 0x028c54 thru 0x034634 (0xb9e1 bytes) 000636 1787584078.166294 mapwrite 0x019dd7 thru 0x022734 (0x895e bytes) 000637 1787584078.295528 trunc from 0x03e144 to 0x017011 (0x27132 bytes) 000638 1787584078.331672 mapread 0x00bb86 thru 0x01610d (0xa588 bytes) 000639 1787584078.334553 read 0x0012bd thru 0x0109de (0xf722 bytes) 000640 1787584078.345972 read 0x010591 thru 0x017011 (0x6a81 bytes) 000641 1787584078.346806 mapwrite 0x0359c8 thru 0x03ffff (0xa638 bytes) 000643 1787584078.429150 trunc from 0x040000 to 0x00869c (0x37963 bytes) 000646 1787584078.452734 write 0x035e22 thru 0x03e5e8 (0x87c7 bytes) 000647 1787584078.455211 mapwrite 0x036967 thru 0x03ffff (0x9699 bytes) 000649 1787584078.563219 mapwrite 0x002180 thru 0x00d625 (0xb4a6 bytes) 000652 1787584078.685132 mapread 0x010f5d thru 0x01d910 (0xc9b4 bytes) 000653 1787584078.828818 read 0x039ab2 thru 0x03ffff (0x654e bytes) 000654 1787584078.829533 read 0x03866a thru 0x03fa6a (0x7401 bytes) 000658 1787584078.841165 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) Mirror 17: 0xa984188a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ff71220 Mirror 18: 0x2ff71220 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1787584080.016483 read 0x039b0e thru 0x03ffff (0x64f2 bytes) 000664 1787584080.066969 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x349b5282 (stale) Mirror 15: 0x349b5282 (stale) Mirror 16: 0x349b5282 (stale) Mirror 17: 0xa984188a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ff71220 Mirror 18: 0x2ff71220 Mirror 19: 0x2ff71220 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000666 1787584081.109828 write 0x03bb00 thru 0x03c09f (0x5a0 bytes) 000667 1787584081.212022 read 0x031a85 thru 0x032356 (0x8d2 bytes) 000668 1787584081.219473 mapread 0x03a193 thru 0x03ffff (0x5e6d bytes) 000670 1787584081.239065 trunc from 0x040000 to 0x001fb2 (0x3e04d bytes) 000672 1787584081.271565 trunc from 0x001fb3 to 0x00f41a (0xd468 bytes) 000674 1787584081.302403 read 0x00b212 thru 0x00b6d3 (0x4c2 bytes) 000675 1787584081.331135 write 0x02d15a thru 0x02e68d (0x1534 bytes) 000677 1787584081.332741 write 0x001840 thru 0x00947d (0x7c3e bytes) 000680 1787584081.355864 read 0x029bab thru 0x02e68d (0x4ae3 bytes) 000682 1787584081.372018 write 0x017142 thru 0x021217 (0xa0d6 bytes) 000683 1787584081.379662 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e68e): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e68e): Mirror 11: 0x644b3ed1 (stale) Mirror 15: 0x644b3ed1 (stale) Mirror 16: 0x644b3ed1 (stale) Mirror 17: 0x2459dacb (stale) Mirror 18: 0x69f3db93 (stale) Mirror 19: 0x69f3db93 (stale) CRC-32 checksum value for chunk [0, 0x2e68e): Mirror 1: 0x107af6e1 Mirror 20: 0x107af6e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e68e: skipped 000686 1787584082.622900 mapread 0x01b2c7 thru 0x02a332 (0xf06c bytes) truncating to largest ever: 0x3ff84 000687 1787584082.656304 trunc from 0x02e68e to 0x03ff83 (0x118f6 bytes) 000688 1787584082.758886 write 0x03df46 thru 0x03ffff (0x20ba bytes) 000690 1787584082.766252 mapwrite 0x00437c thru 0x00973a (0x53bf bytes) 000691 1787584082.839182 mapread 0x000d02 thru 0x010c7b (0xff7a bytes) 000693 1787584082.850173 trunc from 0x040000 to 0x026203 (0x19dfc bytes) 000694 1787584082.894362 read 0x01bc74 thru 0x01e4e4 (0x2871 bytes) 000695 1787584082.904155 write 0x006ea7 thru 0x012461 (0xb5bb bytes) 000697 1787584082.910895 trunc from 0x026204 to 0x00fdbb (0x16448 bytes) 000698 1787584082.931207 mapread 0x00c1c4 thru 0x00faa5 (0x38e2 bytes) 000699 1787584082.932845 mapwrite 0x02ea07 thru 0x02f720 (0xd1a bytes) 000700 1787584083.041394 read 0x01f334 thru 0x026820 (0x74ed bytes) 000701 1787584083.061681 mapread 0x02794b thru 0x02f720 (0x7dd6 bytes) 000702 1787584083.064193 trunc from 0x02f721 to 0x004c94 (0x2aa8c bytes) 000703 1787584083.098140 mapwrite 0x0080bd thru 0x00f9fe (0x7942 bytes) 000704 1787584083.237474 mapread 0x00d508 thru 0x00f9fe (0x24f7 bytes) 000706 1787584083.238974 read 0x00f1c3 thru 0x00f9fe (0x83c bytes) 000707 1787584083.240917 write 0x003a34 thru 0x006e27 (0x33f4 bytes) 000708 1787584083.249356 trunc from 0x00f9ff to 0x0025f1 (0xd40d bytes) 000711 1787584083.269853 trunc from 0x0025f2 to 0x030877 (0x2e286 bytes) 000712 1787584083.282225 mapwrite 0x027452 thru 0x03459b (0xd14a bytes) 000713 1787584083.369495 MIRROR_RESYNC 8 mirrors 000714 1787584085.580216 mapread 0x02c623 thru 0x03459b (0x7f79 bytes) 000716 1787584085.614121 read 0x02dc20 thru 0x03459b (0x697c bytes) 000719 1787584085.615435 mapwrite 0x007819 thru 0x0115e8 (0x9dd0 bytes) 000720 1787584085.790651 MIRROR_RESYNC 8 mirrors 000721 1787584087.357454 write 0x00ca4d thru 0x01b626 (0xebda bytes) 000723 1787584087.450108 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3459c): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3459c): Mirror 11: 0xd401dfb0 (stale) Mirror 15: 0xd401dfb0 (stale) Mirror 16: 0xd401dfb0 (stale) Mirror 17: 0xd401dfb0 (stale) Mirror 18: 0xd401dfb0 (stale) Mirror 19: 0xd401dfb0 (stale) Mirror 20: 0xd401dfb0 (stale) CRC-32 checksum value for chunk [0, 0x3459c): Mirror 1: 0xa43c6a5f Mirror 21: 0xa43c6a5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3459c: skipped 000726 1787584088.868464 trunc from 0x03459c to 0x00869b (0x2bf00 bytes) 000727 1787584089.036694 trunc from 0x00869c to 0x00efd0 (0x6935 bytes) 000728 1787584089.063810 read 0x00c789 thru 0x00cc4d (0x4c5 bytes) 000729 1787584089.073663 trunc from 0x00efd1 to 0x00fe95 (0xec5 bytes) 000731 1787584089.094191 trunc from 0x00fe96 to 0x00374e (0xc747 bytes) 000732 1787584089.115184 trunc from 0x00374f to 0x03499b (0x3124d bytes) 000733 1787584089.140830 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3499c): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3499c): Mirror 11: 0x9e5ba434 (stale) Mirror 15: 0x9e5ba434 (stale) Mirror 16: 0x9e5ba434 (stale) Mirror 17: 0x9e5ba434 (stale) Mirror 18: 0x9e5ba434 (stale) Mirror 19: 0x9e5ba434 (stale) Mirror 20: 0x9e5ba434 (stale) Mirror 21: 0xdb255635 (stale) CRC-32 checksum value for chunk [0, 0x3499c): Mirror 1: 0x5539ad86 Mirror 22: 0x5539ad86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3499c: skipped 000734 1787584090.529057 mapread 0x00f26e thru 0x013044 (0x3dd7 bytes) 000736 1787584090.585488 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3499c): Mirror 11: 0x9e5ba434 (stale) Mirror 15: 0x9e5ba434 (stale) Mirror 16: 0x9e5ba434 (stale) Mirror 17: 0x9e5ba434 (stale) Mirror 18: 0x9e5ba434 (stale) Mirror 19: 0x9e5ba434 (stale) Mirror 21: 0xdb255635 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3499c): Mirror 11: 0x9e5ba434 (stale) Mirror 15: 0x9e5ba434 (stale) Mirror 16: 0x9e5ba434 (stale) Mirror 17: 0x9e5ba434 (stale) Mirror 18: 0x9e5ba434 (stale) Mirror 19: 0x9e5ba434 (stale) Mirror 21: 0xdb255635 (stale) CRC-32 checksum value for chunk [0, 0x3499c): Mirror 1: 0x5539ad86 Mirror 22: 0x5539ad86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3499c: skipped 000738 1787584091.665548 trunc from 0x03499c to 0x00f418 (0x25583 bytes) 000740 1787584091.813975 mapread 0x000f0f thru 0x00f418 (0xe50a bytes) 000741 1787584091.866809 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xf419): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf419): Mirror 11: 0xa5815c0e (stale) Mirror 15: 0xa5815c0e (stale) Mirror 16: 0xa5815c0e (stale) Mirror 17: 0xa5815c0e (stale) Mirror 18: 0xa5815c0e (stale) Mirror 19: 0xa5815c0e (stale) Mirror 21: 0x7fcd866d (stale) Mirror 22: 0x9185c8f0 (stale) CRC-32 checksum value for chunk [0, 0xf419): Mirror 1: 0x9185c8f0 Mirror 23: 0x9185c8f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf419: skipped 000745 1787584093.024298 MIRROR_RESYNC 10 mirrors 000746 1787584094.837674 read 0x00f09f thru 0x00f418 (0x37a bytes) 000747 1787584094.842531 mapread 0x009333 thru 0x00f418 (0x60e6 bytes) 000749 1787584094.864934 MIRROR_RESYNC 10 mirrors 000751 1787584095.487731 trunc from 0x00f419 to 0x00dfb2 (0x1466 bytes) 000752 1787584095.584089 mapread 0x00cfa4 thru 0x00d7d9 (0x836 bytes) 000753 1787584095.595477 write 0x004f12 thru 0x011517 (0xc606 bytes) 000754 1787584095.602098 mapwrite 0x031922 thru 0x03360a (0x1ce9 bytes) 000756 1787584095.707191 mapread 0x024cb5 thru 0x0252e0 (0x62c bytes) 000757 1787584095.718048 trunc from 0x03360b to 0x013508 (0x20102 bytes) 000758 1787584095.734995 read 0x009d79 thru 0x00e84f (0x4ad7 bytes) 000759 1787584095.735974 mapwrite 0x00ca47 thru 0x010a4d (0x4007 bytes) 000760 1787584095.791880 MIRROR_RESYNC 10 mirrors 000763 1787584097.360117 read 0x0124bd thru 0x013508 (0x104c bytes) 000764 1787584097.367116 trunc from 0x013509 to 0x00d1d7 (0x6331 bytes) 000765 1787584097.491657 mapwrite 0x005987 thru 0x005b6a (0x1e4 bytes) 000766 1787584097.638332 read 0x003eb4 thru 0x00a266 (0x63b3 bytes) 000767 1787584097.663168 mapread 0x006187 thru 0x007fee (0x1e68 bytes) 000768 1787584097.666173 write 0x03e248 thru 0x03ffff (0x1db8 bytes) 000769 1787584097.700482 MIRROR_RESYNC 10 mirrors 000770 1787584100.305067 trunc from 0x040000 to 0x01f5cb (0x20a34 bytes) 000772 1787584100.388907 write 0x034bec thru 0x03e66e (0x9a83 bytes) 000773 1787584100.391480 trunc from 0x03e66f to 0x02708b (0x175e3 bytes) 000777 1787584100.416991 trunc from 0x02708c to 0x0274ab (0x420 bytes) 000781 1787584100.456258 mapwrite 0x00a67b thru 0x014aaf (0xa435 bytes) 000782 1787584100.572733 read 0x01b7ec thru 0x0274ab (0xbcc0 bytes) 000783 1787584100.583327 read 0x00a1d1 thru 0x014fac (0xaddc bytes) 000784 1787584100.584146 mapread 0x005a62 thru 0x01491b (0xeeba bytes) 000785 1787584100.589732 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x274ac): Mirror 11: 0x61e429f0 (stale) Mirror 15: 0x61e429f0 (stale) Mirror 16: 0x61e429f0 (stale) Mirror 18: 0x61e429f0 (stale) Mirror 19: 0x61e429f0 (stale) Mirror 21: 0x61e429f0 (stale) Mirror 22: 0x61e429f0 (stale) Mirror 23: 0x61e429f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x274ac): Mirror 11: 0x61e429f0 (stale) Mirror 15: 0x61e429f0 (stale) Mirror 16: 0x61e429f0 (stale) Mirror 18: 0x61e429f0 (stale) Mirror 19: 0x61e429f0 (stale) Mirror 21: 0x61e429f0 (stale) Mirror 22: 0x61e429f0 (stale) Mirror 23: 0x61e429f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x274ac: skipped 000788 1787584101.269527 mapread 0x013279 thru 0x022490 (0xf218 bytes) 000790 1787584101.311940 trunc from 0x0274ac to 0x01dc2b (0x9880 bytes) 000791 1787584101.325520 read 0x003d61 thru 0x0099de (0x5c7e bytes) 000792 1787584101.332911 trunc from 0x01dc2c to 0x031300 (0x136d5 bytes) 000795 1787584101.376822 mapread 0x01fe29 thru 0x02b34e (0xb526 bytes) 000796 1787584101.413094 mapread 0x0198fc thru 0x019929 (0x2e bytes) 000799 1787584101.417159 trunc from 0x031301 to 0x0098f3 (0x27a0d bytes) 000801 1787584101.446017 trunc from 0x0098f4 to 0x01c54b (0x12c58 bytes) 000803 1787584101.468272 mapread 0x01313e thru 0x0188c3 (0x5786 bytes) 000804 1787584101.475960 mapread 0x00125c thru 0x009c18 (0x89bd bytes) 000805 1787584101.483875 mapread 0x00742c thru 0x008328 (0xefd bytes) 000806 1787584101.484795 write 0x021fef thru 0x0230a7 (0x10b9 bytes) 000807 1787584101.486268 read 0x0045bd thru 0x0143f4 (0xfe38 bytes) 000810 1787584101.488552 write 0x00047a thru 0x00e3a2 (0xdf29 bytes) 000812 1787584101.497612 write 0x0031bd thru 0x003819 (0x65d bytes) 000813 1787584101.498282 mapwrite 0x020665 thru 0x0274e6 (0x6e82 bytes) 000814 1787584101.565592 mapread 0x021b66 thru 0x0258ee (0x3d89 bytes) 000816 1787584101.567849 mapwrite 0x0084cb thru 0x00fb0e (0x7644 bytes) 000821 1787584101.613240 read 0x01a60b thru 0x01e34a (0x3d40 bytes) 000822 1787584101.618601 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x274e7): Mirror 11: 0xd1a0042b (stale) Mirror 15: 0xd1a0042b (stale) Mirror 16: 0xd1a0042b (stale) Mirror 19: 0xd1a0042b (stale) Mirror 21: 0xd1a0042b (stale) Mirror 22: 0xd1a0042b (stale) Mirror 23: 0xd1a0042b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x274e7): Mirror 11: 0xd1a0042b (stale) Mirror 15: 0xd1a0042b (stale) Mirror 16: 0xd1a0042b (stale) Mirror 19: 0xd1a0042b (stale) Mirror 21: 0xd1a0042b (stale) Mirror 22: 0xd1a0042b (stale) Mirror 23: 0xd1a0042b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x274e7: skipped 000823 1787584102.027954 read 0x00e0f2 thru 0x0170c1 (0x8fd0 bytes) 000824 1787584102.039029 read 0x0196f9 thru 0x0201fb (0x6b03 bytes) 000826 1787584102.052573 trunc from 0x0274e7 to 0x024799 (0x2d4d bytes) 000827 1787584102.068430 mapread 0x00b052 thru 0x019e0d (0xedbc bytes) 000828 1787584102.074547 MIRROR_RESYNC 8 mirrors 000829 1787584104.119906 mapread 0x016203 thru 0x024799 (0xe597 bytes) 000831 1787584104.154357 write 0x006e0c thru 0x0131c7 (0xc3bc bytes) 000832 1787584104.288583 mapread 0x0161ab thru 0x018f06 (0x2d5c bytes) 000834 1787584104.300519 read 0x012a85 thru 0x01de9f (0xb41b bytes) 000835 1787584104.306932 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2479a): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2479a): Mirror 11: 0x762184f6 (stale) Mirror 15: 0x762184f6 (stale) Mirror 16: 0x762184f6 (stale) Mirror 19: 0x762184f6 (stale) Mirror 21: 0x762184f6 (stale) Mirror 22: 0x762184f6 (stale) Mirror 23: 0x762184f6 (stale) CRC-32 checksum value for chunk [0, 0x2479a): Mirror 1: 0x431290fc Mirror 24: 0x431290fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2479a: skipped 000837 1787584105.198717 trunc from 0x02479a to 0x0320db (0xd942 bytes) 000838 1787584105.313221 mapread 0x012010 thru 0x01588d (0x387e bytes) 000839 1787584105.333733 mapwrite 0x019d98 thru 0x027535 (0xd79e bytes) 000841 1787584105.419431 mapwrite 0x035d7d thru 0x037b02 (0x1d86 bytes) 000843 1787584105.570183 mapread 0x01d0d5 thru 0x01fd92 (0x2cbe bytes) 000844 1787584105.571037 mapwrite 0x014738 thru 0x01ae25 (0x66ee bytes) 000845 1787584105.702567 trunc from 0x037b03 to 0x02196f (0x16193 bytes) 000847 1787584105.737216 write 0x01d24b thru 0x02901a (0xbdd0 bytes) 000849 1787584105.744716 trunc from 0x02901b to 0x00b289 (0x1dd91 bytes) 000850 1787584105.780016 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xb28a): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb28a): Mirror 11: 0x5336b088 (stale) Mirror 15: 0x5336b088 (stale) Mirror 16: 0x5336b088 (stale) Mirror 19: 0x5336b088 (stale) Mirror 21: 0x5336b088 (stale) Mirror 22: 0x5336b088 (stale) Mirror 23: 0x5336b088 (stale) Mirror 24: 0xd3512fea (stale) CRC-32 checksum value for chunk [0, 0xb28a): Mirror 1: 0xd3512fea Mirror 25: 0xd3512fea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb28a: skipped 000851 1787584107.112945 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb28a): Mirror 15: 0x5336b088 (stale) Mirror 16: 0x5336b088 (stale) Mirror 19: 0x5336b088 (stale) Mirror 21: 0x5336b088 (stale) Mirror 22: 0x5336b088 (stale) Mirror 23: 0x5336b088 (stale) Mirror 24: 0xd3512fea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb28a): Mirror 15: 0x5336b088 (stale) Mirror 16: 0x5336b088 (stale) Mirror 19: 0x5336b088 (stale) Mirror 21: 0x5336b088 (stale) Mirror 22: 0x5336b088 (stale) Mirror 23: 0x5336b088 (stale) Mirror 24: 0xd3512fea (stale) CRC-32 checksum value for chunk [0, 0xb28a): Mirror 1: 0xd3512fea Mirror 25: 0xd3512fea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb28a: skipped 000853 1787584108.341164 mapwrite 0x0323a5 thru 0x03ca1f (0xa67b bytes) 000856 1787584108.676335 mapread 0x001e67 thru 0x00d514 (0xb6ae bytes) 000858 1787584108.700785 write 0x002e20 thru 0x00c386 (0x9567 bytes) 000859 1787584108.707934 MIRROR_RESYNC 9 mirrors 000863 1787584111.917427 mapread 0x03aecb thru 0x03ca1f (0x1b55 bytes) 000865 1787584111.967201 write 0x03735d thru 0x03d03b (0x5cdf bytes) 000868 1787584112.227142 read 0x0383d9 thru 0x03b108 (0x2d30 bytes) 000869 1787584112.230166 trunc from 0x03d03c to 0x0150de (0x27f5d bytes) 000870 1787584112.253728 read 0x014f23 thru 0x0150de (0x1bc bytes) 000874 1787584112.267552 trunc from 0x0150df to 0x03a55a (0x2547c bytes) 000876 1787584112.292022 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a55b): Mirror 16: 0xcd64807f (stale) Mirror 19: 0xcd64807f (stale) Mirror 21: 0xcd64807f (stale) Mirror 22: 0xcd64807f (stale) Mirror 23: 0xcd64807f (stale) Mirror 24: 0xcd64807f (stale) Mirror 25: 0xcd64807f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a55b): Mirror 16: 0xcd64807f (stale) Mirror 19: 0xcd64807f (stale) Mirror 21: 0xcd64807f (stale) Mirror 22: 0xcd64807f (stale) Mirror 23: 0xcd64807f (stale) Mirror 24: 0xcd64807f (stale) Mirror 25: 0xcd64807f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a55b: skipped 000877 1787584113.195618 trunc from 0x03a55b to 0x025648 (0x14f12 bytes) 000878 1787584113.221846 trunc from 0x025649 to 0x01a14a (0xb4fe bytes) 000880 1787584113.251248 trunc from 0x01a14b to 0x00a20f (0xff3b bytes) 000881 1787584113.271184 trunc from 0x00a210 to 0x0025f8 (0x7c17 bytes) 000882 1787584113.288894 mapread 0x001f40 thru 0x0025f8 (0x6b9 bytes) 000884 1787584113.302060 trunc from 0x0025f9 to 0x027f35 (0x2593d bytes) 000886 1787584113.317120 mapwrite 0x00d9a0 thru 0x019f8f (0xc5f0 bytes) 000887 1787584113.392255 mapwrite 0x020a79 thru 0x02221b (0x17a3 bytes) 000888 1787584113.439156 write 0x03b4b9 thru 0x03ffff (0x4b47 bytes) 000892 1787584113.443915 read 0x0237f0 thru 0x02f07b (0xb88c bytes) 000893 1787584113.449463 read 0x01c003 thru 0x02672e (0xa72c bytes) 000895 1787584113.451862 mapwrite 0x02287f thru 0x03204d (0xf7cf bytes) 000896 1787584113.512318 mapread 0x02d330 thru 0x039576 (0xc247 bytes) 000898 1787584113.513684 trunc from 0x040000 to 0x0289bf (0x17640 bytes) 000899 1787584113.533125 mapread 0x01103b thru 0x01d2db (0xc2a1 bytes) 000901 1787584113.534559 mapwrite 0x00154d thru 0x009f6c (0x8a20 bytes) 000902 1787584113.586508 mapread 0x009bc0 thru 0x012d8e (0x91cf bytes) 000903 1787584113.588641 trunc from 0x0289c0 to 0x03bd2e (0x1336f bytes) 000904 1787584113.609190 trunc from 0x03bd2f to 0x033706 (0x8628 bytes) 000905 1787584113.633569 MIRROR_RESYNC 8 mirrors 000907 1787584115.778222 trunc from 0x033707 to 0x02a7b4 (0x8f52 bytes) 000909 1787584115.998800 MIRROR_RESYNC 8 mirrors 000910 1787584117.573097 mapwrite 0x0225a4 thru 0x02895f (0x63bc bytes) 000913 1787584117.788824 trunc from 0x02a7b5 to 0x00dc99 (0x1cb1b bytes) 000917 1787584117.829627 write 0x02d48c thru 0x02ec33 (0x17a8 bytes) 000918 1787584117.831292 mapwrite 0x022952 thru 0x03281a (0xfec9 bytes) 000919 1787584117.949143 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3281b): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3281b): Mirror 16: 0x8367c781 (stale) Mirror 19: 0x8367c781 (stale) Mirror 21: 0x8367c781 (stale) Mirror 22: 0x8367c781 (stale) Mirror 23: 0x8367c781 (stale) Mirror 24: 0x8367c781 (stale) Mirror 25: 0x8367c781 (stale) CRC-32 checksum value for chunk [0, 0x3281b): Mirror 1: 0x1a86c607 Mirror 26: 0x1a86c607 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3281b: skipped 000920 1787584119.681285 write 0x016d5c thru 0x024b03 (0xdda8 bytes) 000924 1787584119.789234 read 0x0167c9 thru 0x0232ca (0xcb02 bytes) 000927 1787584119.792555 mapwrite 0x02393b thru 0x02a15a (0x6820 bytes) 000928 1787584119.951145 write 0x01caa3 thru 0x0202aa (0x3808 bytes) 000929 1787584119.953907 trunc from 0x03281b to 0x0243c8 (0xe452 bytes) 000931 1787584120.003168 mapwrite 0x0077a1 thru 0x009aa1 (0x2301 bytes) 000932 1787584120.184627 mapwrite 0x00a0f6 thru 0x011391 (0x729c bytes) 000935 1787584120.347597 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x243c9): Mirror 16: 0x1d460d5a (stale) Mirror 19: 0x1d460d5a (stale) Mirror 21: 0x1d460d5a (stale) Mirror 22: 0x1d460d5a (stale) Mirror 24: 0x1d460d5a (stale) Mirror 25: 0x1d460d5a (stale) Mirror 26: 0x85a2aa01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x243c9): Mirror 16: 0x1d460d5a (stale) Mirror 19: 0x1d460d5a (stale) Mirror 21: 0x1d460d5a (stale) Mirror 22: 0x1d460d5a (stale) Mirror 24: 0x1d460d5a (stale) Mirror 25: 0x1d460d5a (stale) Mirror 26: 0x85a2aa01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x243c9: skipped 000937 1787584121.077054 write 0x0115c6 thru 0x01e1ef (0xcc2a bytes) 000938 1787584121.087732 mapwrite 0x026b3c thru 0x0299e7 (0x2eac bytes) 000940 1787584121.182542 mapread 0x00f6f1 thru 0x01769a (0x7faa bytes) 000941 1787584121.197004 mapread 0x012872 thru 0x0135fe (0xd8d bytes) 000942 1787584121.199235 read 0x01121d thru 0x01242e (0x1212 bytes) 000943 1787584121.200608 read 0x00db9a thru 0x0171c8 (0x962f bytes) 000946 1787584121.203183 read 0x00517d thru 0x010a6c (0xb8f0 bytes) 000948 1787584121.221871 MIRROR_RESYNC 8 mirrors 000949 1787584123.257880 mapwrite 0x039781 thru 0x03ffff (0x687f bytes) 000952 1787584123.561209 read 0x02576e thru 0x02e01d (0x88b0 bytes) 000953 1787584123.581691 mapwrite 0x01f34d thru 0x0251c0 (0x5e74 bytes) 000954 1787584123.675158 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x2abcd7c4 (stale) Mirror 19: 0x2abcd7c4 (stale) Mirror 21: 0x2abcd7c4 (stale) Mirror 22: 0x2abcd7c4 (stale) Mirror 24: 0x2abcd7c4 (stale) Mirror 26: 0x2abcd7c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x2abcd7c4 (stale) Mirror 19: 0x2abcd7c4 (stale) Mirror 21: 0x2abcd7c4 (stale) Mirror 22: 0x2abcd7c4 (stale) Mirror 24: 0x2abcd7c4 (stale) Mirror 26: 0x2abcd7c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1787584124.702617 write 0x01710f thru 0x022639 (0xb52b bytes) 000957 1787584124.733548 read 0x02cc34 thru 0x033f76 (0x7343 bytes) 000958 1787584124.746629 trunc from 0x040000 to 0x003154 (0x3ceab bytes) 000960 1787584124.775493 trunc from 0x003155 to 0x019342 (0x161ee bytes) 000961 1787584124.803169 write 0x01b240 thru 0x01b7e4 (0x5a5 bytes) 000962 1787584124.815626 write 0x01390e thru 0x01a2dc (0x69cf bytes) 000964 1787584124.882598 write 0x017a44 thru 0x01ecc5 (0x7282 bytes) 000965 1787584124.885549 write 0x01ba23 thru 0x022e70 (0x744e bytes) 000966 1787584124.892660 trunc from 0x022e71 to 0x022091 (0xddf bytes) 000968 1787584124.915996 trunc from 0x022092 to 0x03014d (0xe0bc bytes) 000969 1787584124.946070 mapread 0x008057 thru 0x0089ba (0x964 bytes) 000970 1787584124.964484 trunc from 0x03014e to 0x023f2a (0xc223 bytes) 000971 1787584125.014147 mapwrite 0x0165f9 thru 0x018463 (0x1e6b bytes) 000972 1787584125.095653 trunc from 0x023f2b to 0x01b43f (0x8aeb bytes) 000973 1787584125.131629 mapread 0x005698 thru 0x006eb9 (0x1822 bytes) 000974 1787584125.162018 mapread 0x000e6f thru 0x002bfd (0x1d8f bytes) 000975 1787584125.171531 write 0x027397 thru 0x02e28e (0x6ef8 bytes) 000976 1787584125.173321 read 0x02ca0c thru 0x02e28e (0x1883 bytes) 000977 1787584125.176125 trunc from 0x02e28f to 0x00c4b3 (0x21ddb bytes) 000980 1787584125.215718 read 0x008afe thru 0x00c4b3 (0x39b6 bytes) 000983 1787584125.229509 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 16: 0xc9bc11f8 (stale) Mirror 19: 0xc9bc11f8 (stale) Mirror 21: 0xc9bc11f8 (stale) Mirror 22: 0xc9bc11f8 (stale) Mirror 24: 0xc9bc11f8 (stale) Mirror 26: 0xc9bc11f8 (stale) CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 1: 0x574f544c Mirror 27: 0x574f544c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc4b4: skipped 000984 1787584126.447098 mapread 0x009c46 thru 0x00c4b3 (0x286e bytes) 000985 1787584126.473383 mapread 0x000a99 thru 0x00c1fd (0xb765 bytes) 000986 1787584126.486615 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 16: 0xc9bc11f8 (stale) Mirror 19: 0xc9bc11f8 (stale) Mirror 21: 0xc9bc11f8 (stale) Mirror 22: 0xc9bc11f8 (stale) Mirror 24: 0xc9bc11f8 (stale) Mirror 26: 0xc9bc11f8 (stale) CRC-32 checksum value for chunk [0, 0xc4b4): Mirror 1: 0x574f544c Mirror 27: 0x574f544c Mirror 28: 0x574f544c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc4b4: skipped truncating to largest ever: 0x3ffdb 000987 1787584128.044912 trunc from 0x00c4b4 to 0x03ffda (0x33b27 bytes) 000989 1787584128.170151 mapwrite 0x031c9f thru 0x034431 (0x2793 bytes) 000990 1787584128.228669 write 0x027b16 thru 0x035432 (0xd91d bytes) 000992 1787584128.239942 write 0x0369bc thru 0x03809f (0x16e4 bytes) 000993 1787584128.242230 read 0x01f6b8 thru 0x021e6e (0x27b7 bytes) 000994 1787584128.251587 read 0x009583 thru 0x00a5db (0x1059 bytes) 000995 1787584128.274077 read 0x01ea02 thru 0x01f5cb (0xbca bytes) 000997 1787584128.275581 read 0x00c739 thru 0x01b50d (0xedd5 bytes) 000998 1787584128.279211 read 0x011a79 thru 0x018c44 (0x71cc bytes) 000999 1787584128.279936 mapwrite 0x00d524 thru 0x00e0a4 (0xb81 bytes) 001000 1787584128.375046 read 0x024fbb thru 0x029ede (0x4f24 bytes) All operations completed A-OK! PASS 70a (126s) == sanity-flr test 71: check mirror extend parent fid ==== 11:08:54 (1787584134) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 11:08:57 (1787584137) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.379171 s, 5.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323119 s, 3.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:09:07 (1787584147) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre2 Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8851104 .. Extending file size to 648960 .. failed Extending file size to 3235104 .. Extending file size to 7378880 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8939936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57494 57495 57498 57499 57501 done 192.168.206.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre2 (opts:) 192.168.206.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:10:28 (1787584228) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre2 Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8652704 .. Extending file size to 2552000 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7553344 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8865696 .. Extending file size to 8038272 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8746848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59023 59024 59025 59028 59029 done 192.168.206.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre2 (opts:) 192.168.206.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (92s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:12:01 (1787584321) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188909 s, 11.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.312707 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x48b3ad63 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:12:25 (1787584345) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21576 s, 9.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 11:12:34 (1787584354) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:12:41 (1787584361) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190511 s, 11.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:12:50 (1787584370) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193273 s, 10.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:12:59 (1787584379) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.412466 s, 5.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:13:33 (1787584413) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg642-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167404 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:14:39 (1787584479) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg642-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.268783 s, 7.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:15:43 (1787584543) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg642-server mds-ost sync done. Creating new pool test_204f oleg642-server: Pool lustre.test_204f created Adding targets to pool oleg642-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg642-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257224 s, 8.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg642-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg642-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg642-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:17:01 (1787584621) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:17:09 (1787584629) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:17:17 (1787584637) oleg642-server: Pool lustre.test_206 created Adding targets to pool oleg642-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg642-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg642-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg642-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg642-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:17:38 (1787584658) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886571 s, 11.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033609 s, 31.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x47700151 Mirror 3: 0x47700151 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:17:48 (1787584668) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:17:52 (1787584672) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:17:56 (1787584676) Starting client: oleg642-client.virtnet: -o user_xattr,flock 192.168.206.142@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106903 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1eb0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16b0000 Read this (4 bytes): BBBB 192.168.206.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg642-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:18:33 (1787584713) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0257464 s, 318 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:18:39 (1787584719) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576444 s, 18.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x210 lmm_fid: [0x200000404:0x210:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x210 lmm_fid: [0x200000404:0x210:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834448 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 11:18:41.000000000 -0400 Modify: 2026-08-24 11:18:41.000000000 -0400 Change: 2026-08-24 11:18:42.000000000 -0400 Birth: 2026-08-24 11:18:41.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:18:47 (1787584727) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:18:50 (1787584730) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.708949 s, 57.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0540839 s, 75.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 11:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x212:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 11:18 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:19:00 (1787584740) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:19:09 (1787584749) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1811 sec =========== 11:19:20 (1787584760) === sanity-flr: start cleanup 11:19:22 (1787584762) === === sanity-flr: finish cleanup 11:19:28 (1787584768) ===