-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 03:34:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg319-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:35:02 (1787729702) === oleg319-client.virtnet: executing check_config_client /mnt/lustre oleg319-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg319-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e184694b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e184694b800.idle_timeout=debug disable quota as required oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:35:41 (1787729741) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8420] == sanity-flr test 0a: lfs mirror create with -N option == 03:35:43 (1787729743) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:36:03 (1787729763) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:36:07 (1787729767) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:36:10 (1787729770) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:36:35 (1787729795) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:36:38 (1787729798) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 03:36:41 (1787729801) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e184694b800.stats=clear osc.lustre-OST0001-osc-ffff8e184694b800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:36:51 (1787729811) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:37:01 (1787729821) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:37:10 (1787729830) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:37:14 (1787729834) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:37:21 (1787729841) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 03:37:31 (1787729851) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:37:38 (1787729858) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:37:46 (1787729866) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:37:54 (1787729874) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:38:05 (1787729885) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:38:12 (1787729892) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.95354 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245704 s, 4.3 MB/s PASS 21 (16s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:38:29 (1787729909) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.14062 s, 18.4 MB/s PASS 22 (14s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:38:43 (1787729923) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:38:54 (1787729934) Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:40:03 (1787730003) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.38326 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:40:43 (1787730043) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:42:03 (1787730123) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.82776 s, 10.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 03:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.239s user 0m0.001s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.379s user 0m0.000s sys 0m0.167s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg319-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg319-server oleg319-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.312s user 0m0.001s sys 0m0.199s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg319-server: oleg319-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg319-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e184694b800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:43:39 (1787730219) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:43:42 (1787730222) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:43:46 (1787730226) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 03:43:49 (1787730229) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 03:43:57 (1787730237) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.563774 s, 22.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:44:11 (1787730251) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253475 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392948 s, 26.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:44:18 (1787730258) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:44:26 (1787730266) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108554 s, 943 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0013864 s, 739 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.002136 s, 479 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00680711 s, 150 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 03:44:36 (1787730276) create test files with size 15755336 .. 9be4ae2b182449a9abebf26c8a9b1634 - 66d00b54a3fdd7dd67acb2111b8ef2b1 - 18063927520535c554826db4740f4f95 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 03:45:02 (1787730302) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:45:16 (1787730316) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:45:23 (1787730323) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108061 s, 9.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400959 s, 26.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 03:45:33 (1787730333) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1884 s, 11.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.421893 s, 9.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.278859 s, 7.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 03:45:49 (1787730349) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:45:53 (1787730353) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:45:57 (1787730357) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445722 s, 23.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 03:46:06 (1787730366) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.390735 s, 8.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:46:17 (1787730377) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:46:27 (1787730387) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.316074 s, 33.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:46:38 (1787730398) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8e185171d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271984 s, 38.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e18521b0000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:46:53 (1787730413) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:47:37 (1787730457) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:48:18 (1787730498) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:48:21 (1787730501) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.291369 s, 10.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.165613 s, 19.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd6b9e632 (nosync,stale) Mirror 2: 0xd6b9e632 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 66d37aac107b3144d0de050cb391544b - mirror 1 checksum: 7be901ed56ba3db0a29f504f789c66d6 - mirror 2 checksum: 66d37aac107b3144d0de050cb391544b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T03:48:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T03:48:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 66d37aac107b3144d0de050cb391544b - mirror 1 checksum: 7be901ed56ba3db0a29f504f789c66d6 - mirror 2 checksum: 7be901ed56ba3db0a29f504f789c66d6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T03:48:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T03:48:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:48:36 (1787730516) Failing mds1 on oleg319-server Stopping /mnt/lustre-mds1 (opts:) on oleg319-server 03:48:43 (1787730523) shut down facet: mds1 facet_host: oleg319-server facet_failover_host: oleg319-server Failover mds1 to oleg319-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 03:49:08 (1787730548) targets are mounted 03:49:08 (1787730548) facet_failover done oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410291 s, 25.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:49:24 (1787730564) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:49:28 (1787730568) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:49:31 (1787730571) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00731139 s, 2.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:49:39 (1787730579) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:49:43 (1787730583) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00609187 s, 672 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0762451 s, 53.7 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:49:51 (1787730591) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:50:14 (1787730614) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:50:44 (1787730644) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xd:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:51:08 (1787730668) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:51:30 (1787730690) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 03:51:37 (1787730697) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 56362 mirror split pid 56363 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56362 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56363 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 03:52:45 (1787730765) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 767 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1787730767.277620 write 0x01f582 thru 0x022523 (0x2fa2 bytes) truncating to largest ever: 0x3fccd 000004 1787730767.280439 trunc from 0x022524 to 0x03fccc (0x1d7a9 bytes) 000006 1787730767.300133 write 0x01d697 thru 0x01f7ed (0x2157 bytes) 000007 1787730767.307444 read 0x01d6cd thru 0x01e1a2 (0xad6 bytes) 000009 1787730767.309270 write 0x021955 thru 0x0272f1 (0x599d bytes) 000013 1787730767.323183 mapwrite 0x013beb thru 0x01589e (0x1cb4 bytes) 000019 1787730767.410618 trunc from 0x03fccd to 0x00a61e (0x356ae bytes) 000020 1787730767.432422 mapwrite 0x03dd46 thru 0x03ffff (0x22ba bytes) 000022 1787730767.531141 trunc from 0x040000 to 0x013146 (0x2ceb9 bytes) 000023 1787730767.558078 write 0x001e57 thru 0x00804d (0x61f7 bytes) 000024 1787730767.584848 read 0x008cea thru 0x013146 (0xa45d bytes) 000025 1787730767.616638 mapread 0x005606 thru 0x006445 (0xe40 bytes) 000026 1787730767.618836 read 0x010fa3 thru 0x013146 (0x21a4 bytes) 000027 1787730767.622263 write 0x0074be thru 0x00d07e (0x5bc1 bytes) 000031 1787730767.632611 mapread 0x00e626 thru 0x013146 (0x4b21 bytes) 000032 1787730767.635017 write 0x02e93f thru 0x03cd4d (0xe40f bytes) 000033 1787730767.636961 read 0x0169dc thru 0x016df9 (0x41e bytes) 000038 1787730767.649645 write 0x01e96d thru 0x027bfe (0x9292 bytes) 000041 1787730767.653683 read 0x037d6d thru 0x03cd4d (0x4fe1 bytes) 000042 1787730767.655206 write 0x01e14e thru 0x01f5f6 (0x14a9 bytes) 000043 1787730767.656468 read 0x029127 thru 0x038ee1 (0xfdbb bytes) 000044 1787730767.657550 read 0x00dcd0 thru 0x01d3a3 (0xf6d4 bytes) 000045 1787730767.660950 mapread 0x0147d6 thru 0x01931d (0x4b48 bytes) 000046 1787730767.662515 trunc from 0x03cd4e to 0x002163 (0x3abea bytes) 000049 1787730767.701602 write 0x024102 thru 0x02cab7 (0x89b6 bytes) 000051 1787730767.706289 mapwrite 0x031c0b thru 0x0362b6 (0x46ac bytes) 000054 1787730767.812028 read 0x02cf17 thru 0x0362b6 (0x93a0 bytes) 000056 1787730767.814094 read 0x008e73 thru 0x00f5f3 (0x6781 bytes) 000057 1787730767.817702 trunc from 0x0362b7 to 0x037608 (0x1352 bytes) 000058 1787730767.853081 mapread 0x02a05c thru 0x037608 (0xd5ad bytes) 000059 1787730767.872101 write 0x00c010 thru 0x017dcf (0xbdc0 bytes) 000060 1787730767.875680 trunc from 0x037609 to 0x0239f4 (0x13c14 bytes) 000061 1787730767.899657 read 0x00230d thru 0x00f31b (0xd00f bytes) 000063 1787730767.901386 mapwrite 0x029eed thru 0x030df4 (0x6f08 bytes) 000065 1787730767.990489 mapread 0x00330e thru 0x006e21 (0x3b14 bytes) 000066 1787730767.992303 mapwrite 0x036125 thru 0x03c1fa (0x60d6 bytes) 000069 1787730768.075810 read 0x02c717 thru 0x03a97d (0xe267 bytes) 000070 1787730768.076508 mapread 0x018acd thru 0x022e3c (0xa370 bytes) 000072 1787730768.080079 write 0x02cafd thru 0x033a05 (0x6f09 bytes) 000073 1787730768.086475 trunc from 0x03c1fb to 0x01f681 (0x1cb79 bytes) 000074 1787730768.105085 read 0x004e2f thru 0x00f858 (0xaa2a bytes) 000075 1787730768.106157 read 0x015af2 thru 0x01f681 (0x9b90 bytes) 000076 1787730768.107164 mapread 0x00b06d thru 0x0115ca (0x655e bytes) 000077 1787730768.108540 mapread 0x015724 thru 0x01f681 (0x9f5e bytes) 000079 1787730768.110707 mapread 0x0046c6 thru 0x0080e8 (0x3a23 bytes) 000081 1787730768.112595 write 0x03a992 thru 0x03ffff (0x566e bytes) 000082 1787730768.114474 read 0x01532a thru 0x01a041 (0x4d18 bytes) 000083 1787730768.117576 read 0x027082 thru 0x033362 (0xc2e1 bytes) 000084 1787730768.129627 mapread 0x03172e thru 0x036e95 (0x5768 bytes) 000085 1787730768.133373 mapread 0x02e5e2 thru 0x036115 (0x7b34 bytes) 000086 1787730768.136379 mapwrite 0x034b15 thru 0x03a397 (0x5883 bytes) 000089 1787730768.189968 mapread 0x03adfa thru 0x03ffff (0x5206 bytes) 000090 1787730768.190980 trunc from 0x040000 to 0x01a0a3 (0x25f5c bytes) 000091 1787730768.210155 trunc from 0x01a0a4 to 0x037251 (0x1d1ae bytes) 000093 1787730768.232659 write 0x03cd8e thru 0x03ffff (0x3272 bytes) 000094 1787730768.233694 mapwrite 0x00be5f thru 0x013c56 (0x7df8 bytes) 000095 1787730768.298970 write 0x02a1cf thru 0x031aa1 (0x78d3 bytes) 000097 1787730768.316509 read 0x0121aa thru 0x01e8da (0xc731 bytes) 000098 1787730768.335463 read 0x02344b thru 0x02e0c2 (0xac78 bytes) 000099 1787730768.336215 mapwrite 0x001805 thru 0x00a9b8 (0x91b4 bytes) 000101 1787730768.422418 mapwrite 0x0016c5 thru 0x00f4fb (0xde37 bytes) 000103 1787730768.478840 read 0x008324 thru 0x00bc9b (0x3978 bytes) 000104 1787730768.479464 mapwrite 0x03e3c1 thru 0x03ffff (0x1c3f bytes) 000107 1787730768.536230 read 0x02930c thru 0x0346b9 (0xb3ae bytes) 000108 1787730768.536866 write 0x0125e7 thru 0x0198cf (0x72e9 bytes) 000110 1787730768.542719 trunc from 0x040000 to 0x02b6b5 (0x1494a bytes) 000111 1787730768.566726 read 0x0232bf thru 0x0280c9 (0x4e0b bytes) 000113 1787730768.568029 write 0x00b7d0 thru 0x00e47e (0x2caf bytes) 000114 1787730768.571667 read 0x022f89 thru 0x025e59 (0x2ed1 bytes) 000115 1787730768.572725 write 0x00ed77 thru 0x016e01 (0x808b bytes) 000117 1787730768.575330 mapwrite 0x01bfb1 thru 0x024e95 (0x8ee5 bytes) 000120 1787730768.658700 mapread 0x025a43 thru 0x027fee (0x25ac bytes) 000121 1787730768.663129 write 0x00ec0f thru 0x01b7fe (0xcbf0 bytes) 000122 1787730768.664872 read 0x00f1b2 thru 0x0169de (0x782d bytes) 000123 1787730768.665486 read 0x0226da thru 0x022caa (0x5d1 bytes) 000128 1787730768.671629 write 0x009857 thru 0x01233f (0x8ae9 bytes) 000129 1787730768.673256 mapwrite 0x02f99a thru 0x0317d7 (0x1e3e bytes) 000130 1787730768.824356 mapwrite 0x003431 thru 0x00b6c8 (0x8298 bytes) 000131 1787730768.909258 trunc from 0x0317d8 to 0x00536e (0x2c469 bytes) 000133 1787730768.953933 mapread 0x0024eb thru 0x00536e (0x2e84 bytes) 000135 1787730768.957487 read 0x0033d2 thru 0x00536e (0x1f9d bytes) 000138 1787730768.959090 mapread 0x00039c thru 0x004311 (0x3f76 bytes) 000142 1787730768.962937 read 0x0042e1 thru 0x00536e (0x108e bytes) 000143 1787730768.963478 trunc from 0x00536f to 0x0383e3 (0x33075 bytes) 000144 1787730768.993493 mapwrite 0x01d0c4 thru 0x0201e0 (0x311d bytes) 000146 1787730769.063560 trunc from 0x0383e4 to 0x0387a6 (0x3c3 bytes) 000153 1787730769.096604 trunc from 0x0387a7 to 0x005fa3 (0x32803 bytes) 000154 1787730769.116871 mapwrite 0x002854 thru 0x004c43 (0x23f0 bytes) 000155 1787730769.168016 read 0x0023ec thru 0x005fa3 (0x3bb8 bytes) 000156 1787730769.169229 read 0x001bd9 thru 0x005fa3 (0x43cb bytes) 000158 1787730769.171561 read 0x0026ce thru 0x005fa3 (0x38d6 bytes) 000160 1787730769.173211 write 0x037307 thru 0x03e77d (0x7477 bytes) 000162 1787730769.175358 write 0x02371b thru 0x030b97 (0xd47d bytes) 000168 1787730769.193451 read 0x035187 thru 0x03e77d (0x95f7 bytes) 000169 1787730769.194239 read 0x01e850 thru 0x0213e5 (0x2b96 bytes) 000172 1787730769.196093 mapwrite 0x01eca1 thru 0x01f630 (0x990 bytes) 000173 1787730769.285154 write 0x02b8d2 thru 0x036b51 (0xb280 bytes) 000174 1787730769.286483 mapwrite 0x00ac8f thru 0x0192e4 (0xe656 bytes) 000175 1787730769.370871 MIRROR_EXTEND to 2 mirrors 000176 1787730769.986160 mapwrite 0x0261de thru 0x026748 (0x56b bytes) 000177 1787730770.143547 mapread 0x031f0a thru 0x036b02 (0x4bf9 bytes) 000178 1787730770.157602 mapread 0x01a51c thru 0x029e26 (0xf90b bytes) 000185 1787730770.185022 write 0x01a8f4 thru 0x023f74 (0x9681 bytes) 000187 1787730770.189189 mapread 0x0348ae thru 0x03903c (0x478f bytes) 000188 1787730770.190301 mapwrite 0x03dfdc thru 0x03ffff (0x2024 bytes) 000190 1787730770.357729 mapread 0x005e53 thru 0x00d4f0 (0x769e bytes) 000191 1787730770.371556 mapwrite 0x039ade thru 0x03c48c (0x29af bytes) 000193 1787730770.422045 write 0x0236e9 thru 0x028a0b (0x5323 bytes) 000194 1787730770.423223 write 0x01762a thru 0x022be5 (0xb5bc bytes) 000197 1787730770.430146 read 0x000bd1 thru 0x00ee99 (0xe2c9 bytes) 000198 1787730770.439117 mapread 0x02c7be thru 0x03c626 (0xfe69 bytes) 000200 1787730770.455399 write 0x026a5c thru 0x03283a (0xbddf bytes) 000203 1787730770.467149 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x966124c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6da91571 Mirror 3: 0x6da91571 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000204 1787730771.558825 trunc from 0x040000 to 0x01c21a (0x23de5 bytes) 000205 1787730771.721484 MIRROR_RESYNC 3 mirrors 000206 1787730772.728119 write 0x03905a thru 0x03ce82 (0x3e29 bytes) 000208 1787730772.851717 write 0x02c70b thru 0x0313c6 (0x4cbc bytes) 000212 1787730772.868063 mapwrite 0x017492 thru 0x01c852 (0x53c1 bytes) 000213 1787730772.967630 read 0x02f9ab thru 0x03236c (0x29c2 bytes) 000214 1787730772.974018 write 0x0320a7 thru 0x03fc06 (0xdb60 bytes) 000215 1787730772.976363 mapwrite 0x01fcab thru 0x0257fc (0x5b52 bytes) 000216 1787730773.028289 write 0x0387e2 thru 0x03f00b (0x682a bytes) 000217 1787730773.030514 read 0x013aab thru 0x01a1ee (0x6744 bytes) 000218 1787730773.031292 mapread 0x016d82 thru 0x01eea8 (0x8127 bytes) 000219 1787730773.032888 read 0x01b896 thru 0x02197b (0x60e6 bytes) 000220 1787730773.033629 read 0x0021c2 thru 0x00ca63 (0xa8a2 bytes) 000221 1787730773.050537 mapwrite 0x02d364 thru 0x03b6d9 (0xe376 bytes) 000225 1787730773.081525 trunc from 0x03fc07 to 0x01f178 (0x20a8e bytes) 000226 1787730773.111898 write 0x03a131 thru 0x03a9da (0x8aa bytes) 000229 1787730773.121684 mapwrite 0x007325 thru 0x013b79 (0xc855 bytes) 000230 1787730773.290725 MIRROR_RESYNC 3 mirrors 000232 1787730774.186124 read 0x010901 thru 0x014cfc (0x43fc bytes) 000235 1787730774.191456 trunc from 0x03a9db to 0x039675 (0x1365 bytes) 000237 1787730774.339924 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39676): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39676): Mirror 2: 0x442d148e (stale) Mirror 3: 0x442d148e (stale) CRC-32 checksum value for chunk [0, 0x39676): Mirror 1: 0x442d148e Mirror 4: 0x442d148e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39676: skipped 000238 1787730775.673329 mapread 0x02e148 thru 0x03126b (0x3124 bytes) 000239 1787730775.737619 mapread 0x03422a thru 0x039675 (0x544c bytes) 000242 1787730775.821222 mapread 0x02a705 thru 0x037940 (0xd23c bytes) 000243 1787730775.835728 mapwrite 0x02a729 thru 0x0331eb (0x8ac3 bytes) 000247 1787730776.074405 mapread 0x0248fd thru 0x0323b6 (0xdaba bytes) 000248 1787730776.080959 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39676): Mirror 3: 0x442d148e (stale) Mirror 4: 0x442d148e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39676): Mirror 3: 0x442d148e (stale) Mirror 4: 0x442d148e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39676: skipped 000249 1787730776.535296 mapwrite 0x02456e thru 0x027251 (0x2ce4 bytes) 000252 1787730776.630503 MIRROR_RESYNC 3 mirrors 000253 1787730777.684562 mapread 0x03737c thru 0x039675 (0x22fa bytes) 000254 1787730777.710491 read 0x024d86 thru 0x02cdd1 (0x804c bytes) 000256 1787730777.725846 mapwrite 0x03f277 thru 0x03ffff (0xd89 bytes) 000257 1787730777.860072 mapwrite 0x02a0ce thru 0x02cfc4 (0x2ef7 bytes) 000259 1787730777.923004 write 0x010517 thru 0x01c90b (0xc3f5 bytes) 000260 1787730777.934226 read 0x013390 thru 0x013f35 (0xba6 bytes) 000261 1787730777.934964 write 0x018463 thru 0x01e857 (0x63f5 bytes) 000262 1787730777.960880 trunc from 0x040000 to 0x02f5bc (0x10a43 bytes) 000263 1787730777.989190 mapwrite 0x028eaa thru 0x034e68 (0xbfbf bytes) 000265 1787730778.224496 write 0x00676f thru 0x014418 (0xdcaa bytes) 000266 1787730778.225721 read 0x00e027 thru 0x0148d2 (0x68ac bytes) 000267 1787730778.227199 mapwrite 0x014a35 thru 0x017621 (0x2bed bytes) 000268 1787730778.304445 write 0x021692 thru 0x029b7d (0x84ec bytes) 000270 1787730778.306062 write 0x00e6ad thru 0x00eda1 (0x6f5 bytes) 000272 1787730778.316376 MIRROR_RESYNC 3 mirrors 000273 1787730779.473281 trunc from 0x034e69 to 0x00d880 (0x275e8 bytes) 000274 1787730779.589335 write 0x030e0f thru 0x034a8e (0x3c80 bytes) 000275 1787730779.591510 mapwrite 0x02e4cf thru 0x0313af (0x2ee1 bytes) 000277 1787730779.685017 read 0x034926 thru 0x034a8e (0x169 bytes) 000278 1787730779.689453 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 3: 0x3441bf76 (stale) Mirror 4: 0x3441bf76 (stale) CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 1: 0x7c183aca Mirror 5: 0x7c183aca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a8f: skipped 000279 1787730781.180794 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 3: 0x3441bf76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 3: 0x3441bf76 (stale) CRC-32 checksum value for chunk [0, 0x34a8f): Mirror 1: 0x7c183aca Mirror 5: 0x7c183aca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a8f: skipped 000280 1787730782.031371 MIRROR_RESYNC 3 mirrors 000281 1787730783.319388 MIRROR_RESYNC 3 mirrors 000282 1787730783.821087 read 0x02258f thru 0x029bf8 (0x766a bytes) 000284 1787730783.831325 mapwrite 0x00759f thru 0x016906 (0xf368 bytes) 000285 1787730784.052492 mapwrite 0x01b44d thru 0x027f48 (0xcafc bytes) 000286 1787730784.130681 trunc from 0x034a8f to 0x011732 (0x2335c bytes) 000287 1787730784.159992 write 0x032ffc thru 0x03f83a (0xc83f bytes) 000288 1787730784.166969 mapread 0x0305ca thru 0x031975 (0x13ac bytes) 000290 1787730784.180675 mapread 0x004a04 thru 0x00d52c (0x8b29 bytes) 000293 1787730784.182702 mapread 0x01d2f5 thru 0x027ad3 (0xa7df bytes) 000296 1787730784.196615 trunc from 0x03f83b to 0x01ebbf (0x20c7b bytes) 000297 1787730784.228214 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ebc0): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebc0): Mirror 3: 0x35996dcf (stale) Mirror 5: 0x35996dcf (stale) CRC-32 checksum value for chunk [0, 0x1ebc0): Mirror 1: 0xd012696f Mirror 6: 0xd012696f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebc0: skipped 000299 1787730785.238069 mapwrite 0x03ad00 thru 0x03ffff (0x5300 bytes) 000300 1787730785.460926 read 0x01fc98 thru 0x0271a1 (0x750a bytes) 000301 1787730785.471368 trunc from 0x040000 to 0x023335 (0x1ccca bytes) 000302 1787730785.515583 trunc from 0x023336 to 0x039ca1 (0x1696c bytes) 000304 1787730785.569229 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ca2): Mirror 3: 0x3ccf2c24 (stale) Mirror 5: 0x3ccf2c24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ca2): Mirror 3: 0x3ccf2c24 (stale) Mirror 5: 0x3ccf2c24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ca2: skipped 000305 1787730785.967962 mapwrite 0x02476c thru 0x025d67 (0x15fc bytes) 000308 1787730786.031948 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39ca2): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ca2): Mirror 3: 0x3ccf2c24 (stale) Mirror 5: 0x3ccf2c24 (stale) CRC-32 checksum value for chunk [0, 0x39ca2): Mirror 1: 0x5eb81e09 Mirror 6: 0x5eb81e09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ca2: skipped 000311 1787730787.454455 mapwrite 0x01b6d1 thru 0x0245aa (0x8eda bytes) 000313 1787730787.738096 write 0x030f86 thru 0x034457 (0x34d2 bytes) 000314 1787730787.775489 mapwrite 0x00188a thru 0x00569d (0x3e14 bytes) 000315 1787730787.862422 mapwrite 0x0386b1 thru 0x03cd70 (0x46c0 bytes) 000317 1787730787.972906 mapwrite 0x02c76e thru 0x033604 (0x6e97 bytes) 000318 1787730788.039082 trunc from 0x03cd71 to 0x01b632 (0x2173e bytes) 000321 1787730788.056155 mapwrite 0x038257 thru 0x03c106 (0x3eb0 bytes) 000323 1787730788.195200 read 0x01f18f thru 0x02a13b (0xafad bytes) 000325 1787730788.208229 write 0x03df1a thru 0x03ffff (0x20e6 bytes) 000327 1787730788.210493 mapwrite 0x01c609 thru 0x025c07 (0x95ff bytes) 000329 1787730788.257933 write 0x00a6e5 thru 0x014d73 (0xa68f bytes) 000330 1787730788.263367 read 0x00ca83 thru 0x018988 (0xbf06 bytes) 000332 1787730788.265024 mapread 0x023c52 thru 0x026015 (0x23c4 bytes) 000334 1787730788.266671 mapwrite 0x01b452 thru 0x01e72f (0x32de bytes) 000335 1787730788.303234 trunc from 0x040000 to 0x011bc0 (0x2e43f bytes) 000337 1787730788.326434 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11bc1): Mirror 3: 0x7539c96e (stale) Mirror 6: 0x35011fbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bc1): Mirror 3: 0x7539c96e (stale) Mirror 6: 0x35011fbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bc1: skipped 000339 1787730788.618155 mapwrite 0x017991 thru 0x01f43e (0x7aae bytes) 000341 1787730788.776314 trunc from 0x01f43f to 0x02b070 (0xbc32 bytes) 000342 1787730788.813220 read 0x008409 thru 0x013023 (0xac1b bytes) 000343 1787730788.827143 mapread 0x00deac thru 0x0104b4 (0x2609 bytes) 000345 1787730788.830534 mapwrite 0x01055d thru 0x016af9 (0x659d bytes) 000348 1787730788.935225 read 0x0023f7 thru 0x0050ac (0x2cb6 bytes) 000349 1787730788.972276 mapwrite 0x00b532 thru 0x0160bc (0xab8b bytes) 000351 1787730789.050345 mapwrite 0x016e64 thru 0x01a061 (0x31fe bytes) 000354 1787730789.097533 mapread 0x01648a thru 0x01ff0c (0x9a83 bytes) 000355 1787730789.098595 read 0x024874 thru 0x02a2e0 (0x5a6d bytes) 000357 1787730789.101145 mapread 0x00c91d thru 0x015ff0 (0x96d4 bytes) 000358 1787730789.102511 trunc from 0x02b071 to 0x00e1cc (0x1cea4 bytes) 000360 1787730789.125662 mapread 0x002776 thru 0x00e1cc (0xba57 bytes) 000365 1787730789.129693 write 0x0269aa thru 0x034226 (0xd87d bytes) 000366 1787730789.132298 mapread 0x00367d thru 0x00ae5b (0x77df bytes) 000369 1787730789.134391 trunc from 0x034227 to 0x00fb74 (0x246b2 bytes) 000370 1787730789.168460 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfb75): Mirror 6: 0xdffe9775 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfb75): Mirror 6: 0xdffe9775 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfb75: skipped 000372 1787730789.547065 write 0x011b63 thru 0x012254 (0x6f2 bytes) 000375 1787730789.561651 mapread 0x0086c8 thru 0x00b5e2 (0x2f1b bytes) 000376 1787730789.606603 trunc from 0x012255 to 0x022976 (0x10722 bytes) 000379 1787730789.631172 write 0x03659f thru 0x03c1f1 (0x5c53 bytes) 000381 1787730789.644699 MIRROR_RESYNC 2 mirrors 000383 1787730791.204652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000384 1787730791.613485 read 0x0109e3 thru 0x016952 (0x5f70 bytes) 000386 1787730791.650296 read 0x02677c thru 0x02ccb6 (0x653b bytes) 000387 1787730791.690503 trunc from 0x03c1f2 to 0x0346bc (0x7b35 bytes) 000389 1787730791.769308 write 0x026a89 thru 0x02d6bd (0x6c35 bytes) 000390 1787730791.792602 write 0x01ef27 thru 0x02269e (0x3778 bytes) 000392 1787730791.817220 read 0x026da4 thru 0x02766a (0x8c7 bytes) 000393 1787730791.817865 trunc from 0x0346bd to 0x01f620 (0x1509c bytes) 000395 1787730791.849655 mapread 0x00f87f thru 0x01dd21 (0xe4a3 bytes) 000396 1787730791.892353 MIRROR_EXTEND to 2 mirrors 000399 1787730793.361483 trunc from 0x01f621 to 0x03ab2f (0x1b50f bytes) 000404 1787730793.515473 trunc from 0x03ab30 to 0x000368 (0x3a7c7 bytes) 000405 1787730793.584205 MIRROR_RESYNC 2 mirrors 000407 1787730794.758415 mapread 0x00015c thru 0x000368 (0x20d bytes) 000409 1787730794.768500 trunc from 0x000369 to 0x02db1d (0x2d7b5 bytes) 000412 1787730794.934872 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2db1e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db1e): Mirror 2: 0x6049cd39 (stale) CRC-32 checksum value for chunk [0, 0x2db1e): Mirror 1: 0x6049cd39 Mirror 3: 0x6049cd39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db1e: skipped 000413 1787730796.628700 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2db1e): Mirror 2: 0x6049cd39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2db1e): Mirror 2: 0x6049cd39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2db1e: skipped 000414 1787730797.216028 trunc from 0x02db1e to 0x008d4e (0x24dcf bytes) 000416 1787730797.303136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000419 1787730797.563093 trunc from 0x008d4f to 0x038a08 (0x2fcba bytes) 000422 1787730797.615225 read 0x023f6d thru 0x0298cb (0x595f bytes) 000424 1787730797.628501 trunc from 0x038a09 to 0x038256 (0x7b2 bytes) 000428 1787730797.648384 trunc from 0x038257 to 0x02a5e1 (0xdc75 bytes) 000431 1787730797.664530 trunc from 0x02a5e2 to 0x0055f9 (0x24fe8 bytes) 000432 1787730797.699402 read 0x001794 thru 0x0055f9 (0x3e66 bytes) 000433 1787730797.728512 mapwrite 0x031354 thru 0x03ffff (0xecac bytes) 000434 1787730797.971330 mapread 0x012290 thru 0x01a4d6 (0x8247 bytes) 000435 1787730797.976189 trunc from 0x040000 to 0x01b6ae (0x24951 bytes) 000436 1787730798.015466 mapwrite 0x01ea38 thru 0x02be7e (0xd447 bytes) 000438 1787730798.395285 trunc from 0x02be7f to 0x03df7b (0x120fd bytes) 000439 1787730798.419206 mapwrite 0x0239da thru 0x02f130 (0xb757 bytes) 000440 1787730798.530286 mapwrite 0x037f68 thru 0x03ffff (0x8098 bytes) 000442 1787730798.701171 trunc from 0x040000 to 0x03fba2 (0x45d bytes) 000443 1787730798.745876 read 0x00ad7a thru 0x00e3b7 (0x363e bytes) 000444 1787730798.746487 write 0x0326ff thru 0x03ffff (0xd901 bytes) 000446 1787730798.765870 write 0x010dd8 thru 0x011ee0 (0x1109 bytes) 000448 1787730798.767167 mapwrite 0x03f2a7 thru 0x03ffff (0xd59 bytes) 000451 1787730798.856217 trunc from 0x040000 to 0x01e215 (0x21dea bytes) 000452 1787730798.880142 write 0x014f1d thru 0x01c907 (0x79eb bytes) 000453 1787730798.881653 mapread 0x00bed3 thru 0x00d072 (0x11a0 bytes) 000454 1787730798.883956 mapwrite 0x035e49 thru 0x03ffff (0xa1b7 bytes) 000456 1787730799.150726 mapwrite 0x013407 thru 0x01f93b (0xc535 bytes) 000458 1787730799.327099 read 0x032486 thru 0x0351bd (0x2d38 bytes) 000459 1787730799.404985 MIRROR_EXTEND to 2 mirrors 000460 1787730800.866837 read 0x0213dc thru 0x02215d (0xd82 bytes) 000461 1787730800.876681 write 0x002504 thru 0x005b17 (0x3614 bytes) 000462 1787730801.091300 write 0x031399 thru 0x0360fd (0x4d65 bytes) 000464 1787730801.112281 mapread 0x029351 thru 0x02e15f (0x4e0f bytes) 000465 1787730801.159809 MIRROR_RESYNC 2 mirrors 000466 1787730802.344947 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000468 1787730802.455629 mapread 0x025d44 thru 0x028b9e (0x2e5b bytes) 000469 1787730802.483317 write 0x020a43 thru 0x02feba (0xf478 bytes) 000470 1787730802.521918 write 0x02e099 thru 0x032fcb (0x4f33 bytes) 000473 1787730802.533142 write 0x003a5a thru 0x00c6a2 (0x8c49 bytes) 000474 1787730802.554543 mapread 0x005a63 thru 0x006fc1 (0x155f bytes) 000475 1787730802.555483 trunc from 0x040000 to 0x0067fd (0x39802 bytes) 000476 1787730802.646711 write 0x00bfc8 thru 0x00ed03 (0x2d3c bytes) 000478 1787730802.656828 trunc from 0x00ed04 to 0x000977 (0xe38c bytes) 000479 1787730802.710481 MIRROR_EXTEND to 2 mirrors 000480 1787730803.268206 mapread 0x00051c thru 0x00086e (0x353 bytes) 000482 1787730803.272190 read 0x000385 thru 0x000977 (0x5f3 bytes) 000483 1787730803.272535 mapwrite 0x03f600 thru 0x03ffff (0xa00 bytes) 000486 1787730803.416992 read 0x030db1 thru 0x03a5eb (0x983b bytes) 000488 1787730803.428664 read 0x034b76 thru 0x0392db (0x4766 bytes) 000489 1787730803.429632 read 0x02bd65 thru 0x037bf6 (0xbe92 bytes) 000492 1787730803.436077 MIRROR_RESYNC 2 mirrors 000494 1787730804.252509 mapwrite 0x0050a9 thru 0x00c55c (0x74b4 bytes) 000495 1787730804.463836 trunc from 0x040000 to 0x0192c6 (0x26d39 bytes) 000499 1787730804.555651 mapread 0x00d1cd thru 0x017030 (0x9e64 bytes) 000504 1787730804.601072 trunc from 0x0192c7 to 0x02ba8f (0x127c9 bytes) 000505 1787730804.632518 mapwrite 0x021c59 thru 0x02fd5b (0xe103 bytes) 000507 1787730804.774649 mapwrite 0x02750d thru 0x035033 (0xdb27 bytes) 000512 1787730804.988955 MIRROR_RESYNC 2 mirrors 000513 1787730806.177921 mapwrite 0x03cae5 thru 0x03cc64 (0x180 bytes) 000517 1787730806.398832 MIRROR_RESYNC 2 mirrors 000519 1787730807.220397 MIRROR_RESYNC 2 mirrors 000520 1787730807.652273 trunc from 0x03cc65 to 0x0012bf (0x3b9a5 bytes) 000522 1787730807.780305 mapread 0x000a50 thru 0x0012bf (0x870 bytes) 000523 1787730807.792975 read 0x0005ed thru 0x001034 (0xa48 bytes) 000525 1787730807.797497 MIRROR_RESYNC 2 mirrors 000526 1787730808.569567 write 0x022e71 thru 0x02a9be (0x7b4e bytes) 000528 1787730808.663265 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000529 1787730808.884259 mapread 0x00a769 thru 0x00cae4 (0x237c bytes) 000532 1787730808.923048 MIRROR_EXTEND to 2 mirrors 000533 1787730810.151188 mapwrite 0x019454 thru 0x01c778 (0x3325 bytes) 000534 1787730810.342591 mapread 0x006ee9 thru 0x00e32e (0x7446 bytes) 000535 1787730810.365000 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a9bf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9bf): Mirror 2: 0xd4348ea8 (stale) CRC-32 checksum value for chunk [0, 0x2a9bf): Mirror 1: 0x283fe921 Mirror 3: 0x283fe921 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9bf: skipped 000536 1787730811.128326 write 0x01393b thru 0x01af5c (0x7622 bytes) 000537 1787730811.187584 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a9bf): Mirror 2: 0xd4348ea8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9bf): Mirror 2: 0xd4348ea8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9bf: skipped 000538 1787730811.648033 trunc from 0x02a9bf to 0x011b75 (0x18e49 bytes) 000539 1787730811.713177 mapread 0x00fc65 thru 0x011b75 (0x1f11 bytes) 000541 1787730811.744000 mapread 0x00fa29 thru 0x011b75 (0x214d bytes) 000542 1787730811.744867 trunc from 0x011b76 to 0x02feb6 (0x1e341 bytes) 000546 1787730811.769814 mapread 0x00237a thru 0x0075ac (0x5233 bytes) 000549 1787730811.782836 mapwrite 0x007242 thru 0x00c752 (0x5511 bytes) 000550 1787730811.835048 mapread 0x00e148 thru 0x0190ab (0xaf64 bytes) 000551 1787730811.845048 trunc from 0x02feb7 to 0x037389 (0x74d3 bytes) 000553 1787730811.877193 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 000554 1787730812.077399 read 0x00ec44 thru 0x00fc7c (0x1039 bytes) 000555 1787730812.086611 write 0x0186fe thru 0x020c34 (0x8537 bytes) 000556 1787730812.105298 trunc from 0x03738a to 0x01c08d (0x1b2fc bytes) 000557 1787730812.132665 mapwrite 0x03fb9a thru 0x03ffff (0x466 bytes) 000558 1787730812.206949 read 0x018eb3 thru 0x0285dc (0xf72a bytes) 000561 1787730812.228570 write 0x03d794 thru 0x03ffff (0x286c bytes) 000563 1787730812.230367 write 0x01ba69 thru 0x02a698 (0xec30 bytes) 000564 1787730812.232171 mapread 0x026165 thru 0x029423 (0x32bf bytes) 000565 1787730812.233706 trunc from 0x040000 to 0x027650 (0x189af bytes) 000566 1787730812.264844 mapwrite 0x010998 thru 0x01652c (0x5b95 bytes) 000570 1787730812.325122 trunc from 0x027651 to 0x02963c (0x1fec bytes) 000571 1787730812.343562 mapwrite 0x017cb3 thru 0x02301c (0xb36a bytes) 000572 1787730812.414099 read 0x007541 thru 0x009617 (0x20d7 bytes) 000579 1787730812.419679 mapread 0x003e91 thru 0x008076 (0x41e6 bytes) 000582 1787730812.429443 write 0x02fdf5 thru 0x0388d4 (0x8ae0 bytes) 000583 1787730812.432953 read 0x0132bf thru 0x021f6d (0xecaf bytes) 000584 1787730812.435050 write 0x00fe75 thru 0x013d0b (0x3e97 bytes) 000586 1787730812.441287 trunc from 0x0388d5 to 0x02b49a (0xd43a bytes) 000587 1787730812.464375 trunc from 0x02b49b to 0x0160c5 (0x153d5 bytes) 000594 1787730812.501951 read 0x006777 thru 0x00eaa9 (0x8333 bytes) 000596 1787730812.504731 mapread 0x0017f1 thru 0x009855 (0x8065 bytes) 000598 1787730812.510083 mapread 0x0053f6 thru 0x01276d (0xd378 bytes) 000599 1787730812.511550 mapread 0x00046d thru 0x0013b1 (0xf45 bytes) 000601 1787730812.516451 mapread 0x005efc thru 0x00c772 (0x6877 bytes) 000602 1787730812.519693 read 0x000a62 thru 0x008ff1 (0x8590 bytes) 000603 1787730812.520548 mapread 0x010adf thru 0x015140 (0x4662 bytes) 000605 1787730812.526189 trunc from 0x0160c6 to 0x026303 (0x1023e bytes) 000606 1787730812.554031 mapread 0x015df8 thru 0x01ef4b (0x9154 bytes) 000609 1787730812.578180 write 0x02a51a thru 0x02b532 (0x1019 bytes) 000618 1787730812.587435 mapread 0x0224e8 thru 0x026f40 (0x4a59 bytes) 000619 1787730812.588935 mapread 0x02962f thru 0x02b532 (0x1f04 bytes) 000623 1787730812.604982 write 0x022185 thru 0x02a4e5 (0x8361 bytes) 000625 1787730812.607210 write 0x0371f7 thru 0x03950a (0x2314 bytes) 000628 1787730812.619990 mapwrite 0x00840b thru 0x009bf0 (0x17e6 bytes) 000629 1787730812.701433 trunc from 0x03950b to 0x038148 (0x13c2 bytes) 000632 1787730812.728192 mapread 0x00be43 thru 0x013ea6 (0x8064 bytes) 000633 1787730812.731508 mapread 0x0179df thru 0x0237c7 (0xbde9 bytes) 000634 1787730812.733862 mapread 0x0006e8 thru 0x009cfd (0x9616 bytes) truncating to largest ever: 0x3fdb2 000635 1787730812.736677 trunc from 0x038149 to 0x03fdb1 (0x7c69 bytes) 000636 1787730812.763074 mapwrite 0x03f743 thru 0x03ffff (0x8bd bytes) 000637 1787730812.839607 trunc from 0x040000 to 0x02d9e7 (0x12618 bytes) 000640 1787730812.873944 write 0x0057df thru 0x00bf76 (0x6798 bytes) 000643 1787730812.877056 read 0x01345e thru 0x015dd6 (0x2979 bytes) 000644 1787730812.883901 MIRROR_EXTEND to 2 mirrors 000646 1787730813.686614 MIRROR_RESYNC 2 mirrors 000647 1787730814.086640 mapread 0x02a8db thru 0x02d9e7 (0x310d bytes) 000649 1787730814.125236 write 0x003856 thru 0x00ce06 (0x95b1 bytes) 000651 1787730814.222928 trunc from 0x02d9e8 to 0x01d102 (0x108e5 bytes) 000652 1787730814.241215 write 0x0252d9 thru 0x034cb9 (0xf9e1 bytes) 000653 1787730814.244009 mapread 0x0128fb thru 0x01b531 (0x8c37 bytes) 000655 1787730814.268181 mapwrite 0x028712 thru 0x029af9 (0x13e8 bytes) 000656 1787730814.334326 trunc from 0x034cba to 0x01ee78 (0x15e41 bytes) 000658 1787730814.360230 trunc from 0x01ee79 to 0x03ddd5 (0x1ef5d bytes) 000659 1787730814.394383 MIRROR_RESYNC 2 mirrors 000660 1787730815.509126 trunc from 0x03ddd6 to 0x01c82e (0x215a7 bytes) 000663 1787730815.588936 write 0x03e367 thru 0x03ffff (0x1c99 bytes) 000664 1787730815.596362 MIRROR_RESYNC 2 mirrors 000665 1787730816.573276 read 0x0234f0 thru 0x03180a (0xe31b bytes) 000669 1787730816.591279 write 0x02af9a thru 0x02e63c (0x36a3 bytes) 000672 1787730816.673381 mapread 0x0022cf thru 0x00f34e (0xd080 bytes) 000673 1787730816.698018 trunc from 0x040000 to 0x03555d (0xaaa2 bytes) 000674 1787730816.722223 read 0x007a8a thru 0x00ba95 (0x400c bytes) 000676 1787730816.724213 write 0x01c983 thru 0x025368 (0x89e6 bytes) 000677 1787730816.735381 write 0x004f2f thru 0x012d32 (0xde04 bytes) 000678 1787730816.736872 mapwrite 0x03e868 thru 0x03ffff (0x1798 bytes) 000679 1787730816.815995 trunc from 0x040000 to 0x0365b6 (0x9a49 bytes) 000680 1787730816.842352 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x365b7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x365b7): Mirror 2: 0xfc1b516b (stale) CRC-32 checksum value for chunk [0, 0x365b7): Mirror 1: 0x43355a3 Mirror 3: 0x43355a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x365b7: skipped 000687 1787730817.647310 trunc from 0x0365b7 to 0x01ed61 (0x17855 bytes) 000689 1787730817.796168 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ed62): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed62): Mirror 2: 0x4bc9f558 (stale) Mirror 3: 0x4a0c4732 (stale) CRC-32 checksum value for chunk [0, 0x1ed62): Mirror 1: 0x4a0c4732 Mirror 4: 0x4a0c4732 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed62: skipped 000690 1787730818.907202 read 0x00fd68 thru 0x01ed61 (0xeffa bytes) 000691 1787730818.924348 trunc from 0x01ed62 to 0x03a952 (0x1bbf1 bytes) 000692 1787730819.001111 mapwrite 0x037f6e thru 0x03b0e5 (0x3178 bytes) 000693 1787730819.079137 read 0x019e4c thru 0x021b6f (0x7d24 bytes) 000694 1787730819.090111 MIRROR_RESYNC 4 mirrors 000696 1787730820.452423 MIRROR_RESYNC 4 mirrors 000697 1787730820.991284 mapwrite 0x005275 thru 0x008b84 (0x3910 bytes) 000703 1787730821.212152 read 0x00d403 thru 0x01410b (0x6d09 bytes) 000704 1787730821.235433 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b0e6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b0e6): Mirror 2: 0x792a2c84 (stale) Mirror 3: 0x792a2c84 (stale) Mirror 4: 0x792a2c84 (stale) CRC-32 checksum value for chunk [0, 0x3b0e6): Mirror 1: 0x324251c9 Mirror 5: 0x324251c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b0e6: skipped 000706 1787730822.369449 trunc from 0x03b0e6 to 0x01b38b (0x1fd5a bytes) 000708 1787730822.560036 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b38c): Mirror 3: 0xe2c8afc4 (stale) Mirror 4: 0xe2c8afc4 (stale) Mirror 5: 0x51ba8873 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b38c): Mirror 3: 0xe2c8afc4 (stale) Mirror 4: 0xe2c8afc4 (stale) Mirror 5: 0x51ba8873 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b38c: skipped 000709 1787730823.023704 trunc from 0x01b38c to 0x01443c (0x6f4f bytes) 000710 1787730823.050927 mapwrite 0x03ab30 thru 0x03ffff (0x54d0 bytes) 000711 1787730823.145708 trunc from 0x040000 to 0x03dd6e (0x2291 bytes) 000712 1787730823.166119 write 0x0160de thru 0x020620 (0xa543 bytes) 000713 1787730823.180931 read 0x031878 thru 0x031f1c (0x6a5 bytes) 000714 1787730823.186374 write 0x02c317 thru 0x038568 (0xc252 bytes) 000715 1787730823.193109 write 0x03f681 thru 0x03ffff (0x97f bytes) 000717 1787730823.196509 mapwrite 0x00538b thru 0x01130b (0xbf81 bytes) 000719 1787730823.372707 write 0x00fd64 thru 0x0115d3 (0x1870 bytes) 000722 1787730823.382713 trunc from 0x040000 to 0x00f278 (0x30d87 bytes) 000723 1787730823.431233 mapread 0x003bf2 thru 0x00f278 (0xb687 bytes) 000725 1787730823.437901 mapwrite 0x021ded thru 0x02be05 (0xa019 bytes) 000726 1787730823.610365 read 0x02b8b8 thru 0x02be05 (0x54e bytes) 000727 1787730823.612723 mapwrite 0x020c28 thru 0x030457 (0xf830 bytes) 000729 1787730823.719408 mapwrite 0x034880 thru 0x03cf9f (0x8720 bytes) 000730 1787730823.806307 MIRROR_RESYNC 4 mirrors 000732 1787730825.179010 write 0x02e78a thru 0x031206 (0x2a7d bytes) 000734 1787730825.257668 read 0x013039 thru 0x013c84 (0xc4c bytes) 000736 1787730825.264319 trunc from 0x03cfa0 to 0x011e4a (0x2b155 bytes) 000737 1787730825.282184 trunc from 0x011e4b to 0x006a13 (0xb437 bytes) 000738 1787730825.299109 write 0x00bca9 thru 0x019223 (0xd57b bytes) 000741 1787730825.303270 read 0x005002 thru 0x010824 (0xb823 bytes) 000742 1787730825.313473 write 0x011e03 thru 0x01d1f6 (0xb3f4 bytes) 000743 1787730825.316031 mapwrite 0x025bcb thru 0x02a454 (0x488a bytes) 000744 1787730825.384547 trunc from 0x02a455 to 0x01a572 (0xfee2 bytes) 000746 1787730825.408165 mapwrite 0x03c448 thru 0x03ffff (0x3bb8 bytes) 000750 1787730825.563522 mapwrite 0x020c41 thru 0x0284fc (0x78bc bytes) 000751 1787730825.844131 read 0x03f41c thru 0x03ffff (0xbe4 bytes) 000752 1787730825.844957 mapwrite 0x034105 thru 0x034f87 (0xe83 bytes) 000753 1787730825.919298 mapread 0x020ac2 thru 0x02e28e (0xd7cd bytes) 000754 1787730825.942027 read 0x011bed thru 0x013a1c (0x1e30 bytes) 000756 1787730825.943490 write 0x004fa5 thru 0x007b92 (0x2bee bytes) 000757 1787730825.951081 mapread 0x03fd83 thru 0x03ffff (0x27d bytes) 000758 1787730825.955063 read 0x0387b2 thru 0x03e598 (0x5de7 bytes) 000760 1787730825.956355 write 0x02fa29 thru 0x037b33 (0x810b bytes) 000762 1787730825.967134 mapread 0x0207a8 thru 0x02436f (0x3bc8 bytes) 000763 1787730825.968500 write 0x02702c thru 0x02f002 (0x7fd7 bytes) 000764 1787730825.976419 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d03cb7 (stale) Mirror 4: 0x88d03cb7 (stale) Mirror 5: 0x88d03cb7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x885f7d2e Mirror 6: 0x885f7d2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000765 1787730827.198608 read 0x025bb4 thru 0x032356 (0xc7a3 bytes) 000766 1787730827.207950 mapwrite 0x0105a5 thru 0x0159e3 (0x543f bytes) 000767 1787730827.385266 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d03cb7 (stale) Mirror 4: 0x88d03cb7 (stale) Mirror 6: 0x885f7d2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d03cb7 (stale) Mirror 4: 0x88d03cb7 (stale) Mirror 6: 0x885f7d2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000768 1787730827.850443 mapwrite 0x00ab1a thru 0x00c403 (0x18ea bytes) 000770 1787730827.933332 trunc from 0x040000 to 0x00b7e5 (0x3481a bytes) 000771 1787730827.947424 write 0x02e417 thru 0x02f869 (0x1453 bytes) 000773 1787730827.953468 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f86a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f86a): Mirror 3: 0x1e783de0 (stale) Mirror 4: 0x1e783de0 (stale) Mirror 6: 0x9feacd2 (stale) CRC-32 checksum value for chunk [0, 0x2f86a): Mirror 1: 0x7fc09471 Mirror 7: 0x7fc09471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f86a: skipped 000774 1787730829.306635 read 0x019eaf thru 0x01e331 (0x4483 bytes) 000775 1787730829.315558 read 0x004c33 thru 0x01339b (0xe769 bytes) 000777 1787730829.328881 mapread 0x00dab6 thru 0x01d75a (0xfca5 bytes) 000778 1787730829.329853 mapwrite 0x00c5b1 thru 0x01603a (0x9a8a bytes) 000779 1787730829.541097 read 0x00ca81 thru 0x00e6ca (0x1c4a bytes) 000783 1787730829.554891 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f86a): Mirror 4: 0x1e783de0 (stale) Mirror 6: 0x9feacd2 (stale) Mirror 7: 0x7fc09471 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f86a): Mirror 4: 0x1e783de0 (stale) Mirror 6: 0x9feacd2 (stale) Mirror 7: 0x7fc09471 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f86a: skipped 000784 1787730829.879795 write 0x0046ff thru 0x00a83b (0x613d bytes) 000787 1787730829.893467 write 0x01da85 thru 0x01e7af (0xd2b bytes) 000788 1787730829.905472 write 0x01281f thru 0x01c580 (0x9d62 bytes) 000790 1787730829.919840 mapread 0x009a5c thru 0x00ddb6 (0x435b bytes) 000791 1787730829.944220 mapwrite 0x030018 thru 0x03d944 (0xd92d bytes) 000794 1787730830.112304 trunc from 0x03d945 to 0x016a0c (0x26f38 bytes) 000796 1787730830.137029 read 0x004813 thru 0x0119c2 (0xd1b0 bytes) 000797 1787730830.138212 trunc from 0x016a0d to 0x0178e8 (0xedc bytes) 000798 1787730830.157392 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178e9): Mirror 4: 0xf13f27d3 (stale) Mirror 6: 0x8e5062ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178e9): Mirror 4: 0xf13f27d3 (stale) Mirror 6: 0x8e5062ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178e9: skipped 000800 1787730830.508281 trunc from 0x0178e9 to 0x0111b8 (0x6730 bytes) 000802 1787730830.539909 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x111b9): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x111b9): Mirror 4: 0xeb4674ff (stale) Mirror 6: 0x8cdb3e56 (stale) CRC-32 checksum value for chunk [0, 0x111b9): Mirror 1: 0x40bff008 Mirror 7: 0x40bff008 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x111b9: skipped 000805 1787730831.888798 mapwrite 0x0258a7 thru 0x031e45 (0xc59f bytes) 000806 1787730832.254495 mapwrite 0x003561 thru 0x004184 (0xc24 bytes) 000807 1787730832.342224 MIRROR_RESYNC 4 mirrors 000808 1787730833.930818 write 0x003555 thru 0x010741 (0xd1ed bytes) 000809 1787730834.051569 write 0x01684d thru 0x0176d2 (0xe86 bytes) 000811 1787730834.060986 mapread 0x01b9fa thru 0x01e11f (0x2726 bytes) 000812 1787730834.075508 mapread 0x012200 thru 0x01f6a8 (0xd4a9 bytes) 000814 1787730834.084703 read 0x002190 thru 0x0087a2 (0x6613 bytes) 000817 1787730834.102095 MIRROR_RESYNC 4 mirrors 000819 1787730835.099388 write 0x02e080 thru 0x036829 (0x87aa bytes) 000820 1787730835.205153 write 0x02b6d6 thru 0x0379a1 (0xc2cc bytes) 000821 1787730835.209528 write 0x0310fc thru 0x036daa (0x5caf bytes) 000822 1787730835.210456 write 0x01a45d thru 0x0265a9 (0xc14d bytes) 000823 1787730835.218359 read 0x00d638 thru 0x01a850 (0xd219 bytes) 000824 1787730835.236072 trunc from 0x0379a2 to 0x034c60 (0x2d41 bytes) 000827 1787730835.302965 MIRROR_RESYNC 4 mirrors 000828 1787730836.210882 mapwrite 0x038e10 thru 0x03ffff (0x71f0 bytes) 000830 1787730836.497207 read 0x02f9ac thru 0x02ffb5 (0x60a bytes) 000832 1787730836.505325 write 0x016c51 thru 0x01dc24 (0x6fd4 bytes) 000833 1787730836.516194 trunc from 0x040000 to 0x018974 (0x2768b bytes) 000836 1787730836.538594 mapread 0x0016ca thru 0x006b4b (0x5482 bytes) 000840 1787730836.561932 write 0x017231 thru 0x026874 (0xf644 bytes) 000841 1787730836.575230 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26875): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26875): Mirror 4: 0xf912a3f7 (stale) Mirror 6: 0xf912a3f7 (stale) Mirror 7: 0xf912a3f7 (stale) CRC-32 checksum value for chunk [0, 0x26875): Mirror 1: 0x923787b7 Mirror 8: 0x923787b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26875: skipped 000844 1787730838.036871 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26875): Mirror 6: 0xf912a3f7 (stale) Mirror 7: 0xf912a3f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26875): Mirror 6: 0xf912a3f7 (stale) Mirror 7: 0xf912a3f7 (stale) CRC-32 checksum value for chunk [0, 0x26875): Mirror 1: 0x923787b7 Mirror 8: 0x923787b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26875: skipped 000846 1787730839.246771 read 0x011751 thru 0x01727f (0x5b2f bytes) 000849 1787730839.382919 read 0x01dbfb thru 0x0217d9 (0x3bdf bytes) 000850 1787730839.434944 read 0x002f64 thru 0x0055c1 (0x265e bytes) 000852 1787730839.489045 read 0x013bda thru 0x019549 (0x5970 bytes) 000853 1787730839.503542 write 0x039da5 thru 0x03ffff (0x625b bytes) 000854 1787730839.580881 read 0x02c36e thru 0x033de4 (0x7a77 bytes) 000855 1787730839.594205 read 0x00e635 thru 0x0153c2 (0x6d8e bytes) 000857 1787730839.595449 mapwrite 0x02bdad thru 0x033657 (0x78ab bytes) 000858 1787730839.648813 write 0x02d885 thru 0x03bac3 (0xe23f bytes) 000859 1787730839.650141 write 0x036320 thru 0x03ffff (0x9ce0 bytes) 000862 1787730839.658143 MIRROR_RESYNC 4 mirrors 000865 1787730840.633677 trunc from 0x040000 to 0x016b33 (0x294cc bytes) 000866 1787730840.788647 write 0x00210d thru 0x006cd9 (0x4bcd bytes) 000868 1787730840.804970 read 0x001d7d thru 0x0024ee (0x772 bytes) 000869 1787730840.815863 mapwrite 0x03bece thru 0x03ffff (0x4132 bytes) 000870 1787730840.999308 MIRROR_RESYNC 4 mirrors 000871 1787730842.546330 MIRROR_EXTEND to 5 mirrors 000872 1787730843.604969 mapwrite 0x0106aa thru 0x01faf6 (0xf44d bytes) 000873 1787730843.834668 mapwrite 0x00e367 thru 0x016d5c (0x89f6 bytes) 000875 1787730843.926970 mapread 0x03ec5a thru 0x03ffff (0x13a6 bytes) 000878 1787730843.946123 MIRROR_RESYNC 5 mirrors 000879 1787730845.897977 read 0x036455 thru 0x03ffff (0x9bab bytes) 000882 1787730845.918558 MIRROR_RESYNC 5 mirrors 000883 1787730846.422997 write 0x011e91 thru 0x015667 (0x37d7 bytes) 000884 1787730846.523949 write 0x02dfe3 thru 0x03c5fa (0xe618 bytes) 000885 1787730846.541416 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x85e90929 (stale) Mirror 7: 0x85e90929 (stale) Mirror 8: 0x85e90929 (stale) Mirror 9: 0x85e90929 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x497522f0 Mirror 10: 0x497522f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1787730847.842407 mapwrite 0x00a12b thru 0x0183b1 (0xe287 bytes) 000887 1787730848.099034 trunc from 0x040000 to 0x03e1a5 (0x1e5a bytes) 000888 1787730848.122565 trunc from 0x03e1a6 to 0x00dcce (0x304d7 bytes) 000891 1787730848.143161 mapread 0x0017df thru 0x007f86 (0x67a8 bytes) 000892 1787730848.154362 read 0x006726 thru 0x006b24 (0x3ff bytes) 000893 1787730848.155939 trunc from 0x00dccf to 0x0004a7 (0xd827 bytes) 000894 1787730848.182303 read 0x00044e thru 0x0004a7 (0x5a bytes) 000895 1787730848.190963 read 0x000446 thru 0x0004a7 (0x62 bytes) 000896 1787730848.192359 write 0x03449b thru 0x037149 (0x2caf bytes) 000897 1787730848.194867 mapwrite 0x02ca4b thru 0x02f67b (0x2c31 bytes) 000899 1787730848.286320 mapwrite 0x02c395 thru 0x0316eb (0x5357 bytes) 000900 1787730848.373493 write 0x02be9f thru 0x02db14 (0x1c76 bytes) 000903 1787730848.385599 MIRROR_RESYNC 6 mirrors 000904 1787730850.553079 read 0x01c674 thru 0x0212e5 (0x4c72 bytes) 000905 1787730850.564789 mapread 0x011245 thru 0x019c35 (0x89f1 bytes) 000906 1787730850.584241 read 0x030b55 thru 0x037149 (0x65f5 bytes) 000907 1787730850.594204 mapread 0x02405d thru 0x02c1fc (0x81a0 bytes) 000909 1787730850.596102 trunc from 0x03714a to 0x005f1d (0x3122c bytes) 000911 1787730850.693689 trunc from 0x005f1e to 0x01b657 (0x1573a bytes) 000914 1787730850.722655 read 0x015838 thru 0x01a879 (0x5042 bytes) 000917 1787730850.737995 write 0x021c32 thru 0x02ab9e (0x8f6d bytes) 000920 1787730850.745526 mapread 0x00f42a thru 0x018cc2 (0x9899 bytes) 000924 1787730850.756943 MIRROR_RESYNC 6 mirrors 000925 1787730852.856529 mapread 0x001036 thru 0x0097c3 (0x878e bytes) 000926 1787730852.883015 mapwrite 0x03ea47 thru 0x03ffff (0x15b9 bytes) 000927 1787730853.080578 mapread 0x00f434 thru 0x01051a (0x10e7 bytes) 000928 1787730853.103261 mapread 0x014828 thru 0x01da3c (0x9215 bytes) 000929 1787730853.132160 read 0x014695 thru 0x015f7c (0x18e8 bytes) 000930 1787730853.132788 mapwrite 0x02c890 thru 0x02d409 (0xb7a bytes) 000931 1787730853.215621 trunc from 0x040000 to 0x017ef8 (0x28107 bytes) 000934 1787730853.257058 read 0x00c604 thru 0x016700 (0xa0fd bytes) 000936 1787730853.258361 write 0x01cb29 thru 0x02ca25 (0xfefd bytes) 000937 1787730853.262418 write 0x005acc thru 0x009050 (0x3585 bytes) 000938 1787730853.282974 write 0x014d31 thru 0x017c17 (0x2ee7 bytes) 000941 1787730853.284991 write 0x02d4cf thru 0x03006b (0x2b9d bytes) 000944 1787730853.289119 read 0x005317 thru 0x010582 (0xb26c bytes) 000946 1787730853.291205 trunc from 0x03006c to 0x00f403 (0x20c68 bytes) 000948 1787730853.315910 write 0x0049c3 thru 0x0084bc (0x3afa bytes) 000949 1787730853.318156 mapread 0x006bfe thru 0x00f403 (0x8806 bytes) 000950 1787730853.320173 mapwrite 0x03fe7c thru 0x03ffff (0x184 bytes) 000952 1787730853.519385 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fcff70e (stale) Mirror 7: 0x1fcff70e (stale) Mirror 8: 0x1fcff70e (stale) Mirror 10: 0x1fcff70e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fcff70e (stale) Mirror 7: 0x1fcff70e (stale) Mirror 8: 0x1fcff70e (stale) Mirror 10: 0x1fcff70e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1787730854.076810 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fcff70e (stale) Mirror 8: 0x1fcff70e (stale) Mirror 10: 0x1fcff70e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fcff70e (stale) Mirror 8: 0x1fcff70e (stale) Mirror 10: 0x1fcff70e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000954 1787730854.472857 read 0x0361f6 thru 0x03ffff (0x9e0a bytes) 000955 1787730854.489682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1fcff70e (stale) Mirror 8: 0x1fcff70e (stale) Mirror 10: 0x1fcff70e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4af0af8 Mirror 11: 0xc4af0af8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1787730855.832869 write 0x00442e thru 0x00e910 (0xa4e3 bytes) 000957 1787730855.952188 trunc from 0x040000 to 0x01b926 (0x246d9 bytes) 000958 1787730855.979122 write 0x0023af thru 0x00af31 (0x8b83 bytes) 000960 1787730855.986525 write 0x021b5c thru 0x024621 (0x2ac6 bytes) 000965 1787730855.991274 write 0x025112 thru 0x032b79 (0xda68 bytes) 000966 1787730855.995205 mapwrite 0x01d6e2 thru 0x0206f4 (0x3013 bytes) 000967 1787730856.088647 trunc from 0x032b7a to 0x00866d (0x2a50c bytes) 000969 1787730856.115048 read 0x005bfa thru 0x00866d (0x2a74 bytes) 000970 1787730856.116281 read 0x006540 thru 0x00866d (0x212e bytes) 000971 1787730856.117248 mapwrite 0x027a10 thru 0x02fbb0 (0x81a1 bytes) 000972 1787730856.209262 mapwrite 0x03b0d1 thru 0x03ffff (0x4f2f bytes) 000973 1787730856.301237 MIRROR_RESYNC 5 mirrors 000975 1787730857.768312 mapread 0x02165c thru 0x02e0f7 (0xca9c bytes) 000978 1787730857.797861 mapwrite 0x03c8ba thru 0x03ffff (0x3746 bytes) 000979 1787730857.925262 mapread 0x00230c thru 0x003bd8 (0x18cd bytes) 000980 1787730857.945673 MIRROR_RESYNC 5 mirrors 000983 1787730859.701252 mapread 0x03aa75 thru 0x03ffff (0x558b bytes) 000986 1787730859.737940 write 0x039899 thru 0x03eacd (0x5235 bytes) 000988 1787730859.928710 read 0x03713d thru 0x039705 (0x25c9 bytes) 000990 1787730859.942839 mapwrite 0x00431f thru 0x011ccc (0xd9ae bytes) 000991 1787730860.078403 read 0x01a06d thru 0x02936a (0xf2fe bytes) 000992 1787730860.090599 mapread 0x0242e6 thru 0x02ca27 (0x8742 bytes) 000994 1787730860.092445 mapread 0x007938 thru 0x00850c (0xbd5 bytes) 000995 1787730860.097700 MIRROR_RESYNC 5 mirrors 000997 1787730861.712396 mapread 0x035bdb thru 0x037d9e (0x21c4 bytes) 000998 1787730861.805667 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16e:0x0]: Invalid argument (22) 001000 1787730862.283907 write 0x03b1a8 thru 0x03ffff (0x4e58 bytes) All operations completed A-OK! PASS 70a (103s) == sanity-flr test 71: check mirror extend parent fid ==== 03:54:28 (1787730868) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 03:54:32 (1787730872) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.386688 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161619 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 03:54:41 (1787730881) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 724224 .. done Extending file size to 6421184 .. Extending file size to 5441568 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8654112 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8053888 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8057344 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8710144 .. Extending file size to 8869472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 59444 59445 59447 59450 59451 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) 192.168.203.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:56:00 (1787730960) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4910816 .. Extending file size to 8511520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6208064 .. Extending file size to 8231584 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8876448 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8349504 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8449376 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9315808 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60899 60900 60902 60903 60907 done 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) 192.168.203.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:57:18 (1787731038) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135852 s, 15.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.319103 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdf1ebc9f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:57:36 (1787731056) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111853 s, 18.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 03:57:44 (1787731064) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:57:51 (1787731071) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186704 s, 11.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1de:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:57:59 (1787731079) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195739 s, 10.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:58:06 (1787731086) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253153 s, 8.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:58:40 (1787731120) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149704 s, 7.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1dd:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:59:35 (1787731175) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315607 s, 6.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:00:39 (1787731239) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg319-server: Pool lustre.test_204f created Adding targets to pool oleg319-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg319-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.270388 s, 7.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg319-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg319-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg319-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:02:00 (1787731320) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:02:09 (1787731329) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:02:17 (1787731337) oleg319-server: Pool lustre.test_206 created Adding targets to pool oleg319-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg319-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg319-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg319-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg319-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:02:46 (1787731366) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994267 s, 10.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577823 s, 18.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x69fa0c81 Mirror 3: 0x69fa0c81 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:02:57 (1787731377) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:03:01 (1787731381) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:03:05 (1787731385) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.124922 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2050000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2490000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2490000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2490000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e60000 Read this (4 bytes): BBBB 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:03:43 (1787731423) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0429855 s, 191 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:03:53 (1787731433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462256 s, 22.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 04:03:54.000000000 -0400 Modify: 2026-08-26 04:03:54.000000000 -0400 Change: 2026-08-26 04:03:56.000000000 -0400 Birth: 2026-08-26 04:03:55.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:04:01 (1787731441) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:04:04 (1787731444) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.571868 s, 71.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.092657 s, 44.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 04:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 04:04 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:04:13 (1787731453) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:04:21 (1787731461) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1787 sec =========== 04:04:31 (1787731471) === sanity-flr: start cleanup 04:04:33 (1787731473) === === sanity-flr: finish cleanup 04:04:41 (1787731481) ===