-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 16:39:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg214-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:39:46 (1787603986) === oleg214-client.virtnet: executing check_config_client /mnt/lustre oleg214-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg214-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c6ac5109000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c6ac5109000.idle_timeout=debug disable quota as required oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:40:25 (1787604025) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8445] == sanity-flr test 0a: lfs mirror create with -N option == 16:40:28 (1787604028) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:40:44 (1787604044) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:40:47 (1787604047) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:40:50 (1787604050) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:41:14 (1787604074) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:41:18 (1787604078) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:41:21 (1787604081) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c6ac5109000.stats=clear osc.lustre-OST0001-osc-ffff9c6ac5109000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:41:30 (1787604090) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:41:43 (1787604103) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:41:52 (1787604112) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:41:56 (1787604116) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:42:04 (1787604124) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 16:42:11 (1787604131) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:42:18 (1787604138) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:42:25 (1787604145) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:42:32 (1787604152) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:42:41 (1787604161) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:42:49 (1787604169) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.58906 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121335 s, 8.6 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:43:03 (1787604183) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.538118 s, 29.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:43:12 (1787604192) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:43:23 (1787604203) Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (87s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:44:50 (1787604290) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.444328 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:45:34 (1787604334) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (93s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:47:08 (1787604428) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.55446 s, 6.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 16:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.652s user 0m0.001s sys 0m0.194s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.460s user 0m0.000s sys 0m0.166s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.499s user 0m0.003s sys 0m0.209s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6ac5109000.ost_server_uuid in FULL state after 0 sec PASS 33b (116s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:49:05 (1787604545) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:49:09 (1787604549) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:49:12 (1787604552) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 16:49:17 (1787604557) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 16:49:26 (1787604566) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.540103 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:49:39 (1787604579) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450124 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650393 s, 16.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:49:47 (1787604587) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:49:56 (1787604596) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000977672 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132655 s, 772 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000752804 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00170178 s, 602 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 16:50:07 (1787604607) create test files with size 15751263 .. e89d8f3966d86e69e3c0507e6e5d1ad4 - 036873a7e37dfd7cdfee8676bde2487c - 08e6156daf58c921e3a12f478a17f086 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 16:50:31 (1787604631) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:50:50 (1787604650) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:50:59 (1787604659) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759348 s, 13.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552671 s, 19.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 16:51:09 (1787604669) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167845 s, 12.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.426905 s, 9.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228018 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 16:51:29 (1787604689) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 16:51:33 (1787604693) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:51:37 (1787604697) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335833 s, 31.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 16:51:45 (1787604705) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.277616 s, 11.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:51:56 (1787604716) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:52:08 (1787604728) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.34782 s, 30.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:52:21 (1787604741) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c6ac3a39000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300339 s, 34.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c6ac90af800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:52:39 (1787604759) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:53:19 (1787604799) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:54:00 (1787604840) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 16:54:05 (1787604845) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.48307 s, 6.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.230316 s, 13.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x89c0ce41 (nosync,stale) Mirror 2: 0x89c0ce41 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c083dd5751675d96e7ba6c5f24edda5a - mirror 1 checksum: 12d57b6575b2617275e900f7259211b5 - mirror 2 checksum: c083dd5751675d96e7ba6c5f24edda5a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T16:54:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T16:54:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c083dd5751675d96e7ba6c5f24edda5a - mirror 1 checksum: 12d57b6575b2617275e900f7259211b5 - mirror 2 checksum: 12d57b6575b2617275e900f7259211b5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T16:54:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T16:54:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:54:22 (1787604862) Failing mds1 on oleg214-server Stopping /mnt/lustre-mds1 (opts:) on oleg214-server 16:54:31 (1787604871) shut down facet: mds1 facet_host: oleg214-server facet_failover_host: oleg214-server Failover mds1 to oleg214-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 16:55:08 (1787604908) targets are mounted 16:55:08 (1787604908) facet_failover done oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xe:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286418 s, 36.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xe lmm_fid: [0x200000405:0xe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:55:28 (1787604928) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:55:31 (1787604931) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:55:35 (1787604935) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00816492 s, 2.0 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:55:44 (1787604944) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:55:49 (1787604949) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123233 s, 332 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0867545 s, 47.2 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:56:01 (1787604961) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:56:30 (1787604990) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:57:04 (1787605024) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xd:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:57:29 (1787605049) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:57:57 (1787605077) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 16:58:07 (1787605087) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 56419 mirror split pid 56420 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56419 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56420 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 16:59:16 (1787605156) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5157 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787605158.345543 mapwrite 0x02dd7f thru 0x0321a2 (0x4424 bytes) 000002 1787605158.547473 read 0x022b53 thru 0x026007 (0x34b5 bytes) 000003 1787605158.575601 MIRROR_EXTEND to 2 mirrors 000005 1787605159.676147 mapwrite 0x020b60 thru 0x02be8d (0xb32e bytes) 000007 1787605160.033314 mapwrite 0x019aff thru 0x01f517 (0x5a19 bytes) truncating to largest ever: 0x400b 000008 1787605160.116276 trunc from 0x0321a3 to 0x00400a (0x2e198 bytes) 000009 1787605160.173359 mapread 0x0037c8 thru 0x00400a (0x843 bytes) 000010 1787605160.181828 mapwrite 0x0057be thru 0x00b646 (0x5e89 bytes) truncating to largest ever: 0x2690b 000012 1787605160.246190 trunc from 0x00b647 to 0x02690a (0x1b2c4 bytes) 000014 1787605160.266055 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000015 1787605160.379263 mapread 0x002048 thru 0x0060db (0x4094 bytes) 000023 1787605160.396360 read 0x00afd6 thru 0x01ade6 (0xfe11 bytes) 000024 1787605160.404055 trunc from 0x02690b to 0x02141f (0x54eb bytes) 000025 1787605160.423264 MIRROR_EXTEND to 2 mirrors 000028 1787605161.080672 write 0x03dc29 thru 0x03ffff (0x23d7 bytes) 000029 1787605161.124686 mapwrite 0x02a5ae thru 0x03433d (0x9d90 bytes) 000030 1787605161.173865 write 0x004282 thru 0x00ff38 (0xbcb7 bytes) 000031 1787605161.183440 read 0x0049bf thru 0x00efc6 (0xa608 bytes) 000032 1787605161.184357 mapwrite 0x02b918 thru 0x02e8bb (0x2fa4 bytes) 000033 1787605161.243074 MIRROR_RESYNC 2 mirrors 000034 1787605162.219806 read 0x00b1b2 thru 0x018bd1 (0xda20 bytes) 000037 1787605162.245459 read 0x021a89 thru 0x0317ad (0xfd25 bytes) 000038 1787605162.275771 write 0x006aa6 thru 0x00866b (0x1bc6 bytes) 000039 1787605162.458384 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xffc7b983 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x46f6aff8 Mirror 3: 0x46f6aff8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000042 1787605163.893360 mapwrite 0x035eb7 thru 0x03d2be (0x7408 bytes) truncating to largest ever: 0x34d2b 000044 1787605164.150795 trunc from 0x040000 to 0x034d2a (0xb2d5 bytes) 000045 1787605164.175812 mapread 0x001d65 thru 0x0064a1 (0x473d bytes) 000046 1787605164.188308 mapread 0x004b2b thru 0x00f3c6 (0xa89c bytes) 000047 1787605164.196177 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d2b): Mirror 3: 0xa9ffe1b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d2b): Mirror 3: 0xa9ffe1b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d2b: skipped 000048 1787605164.510992 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34d2b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d2b): Mirror 3: 0xa9ffe1b6 (stale) CRC-32 checksum value for chunk [0, 0x34d2b): Mirror 1: 0xa9ffe1b6 Mirror 4: 0xa9ffe1b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d2b: skipped 000049 1787605165.909473 read 0x002375 thru 0x00e1e9 (0xbe75 bytes) 000050 1787605165.933420 MIRROR_RESYNC 3 mirrors 000051 1787605167.230644 trunc from 0x034d2b to 0x005184 (0x2fba6 bytes) 000052 1787605167.377936 trunc from 0x005185 to 0x030625 (0x2b4a1 bytes) 000053 1787605167.420848 write 0x0225b0 thru 0x0285a2 (0x5ff3 bytes) 000054 1787605167.431916 write 0x01313f thru 0x01bb2b (0x89ed bytes) 000055 1787605167.441990 write 0x022659 thru 0x030474 (0xde1c bytes) 000057 1787605167.452945 mapread 0x025c12 thru 0x029068 (0x3457 bytes) 000058 1787605167.456105 mapwrite 0x0273d0 thru 0x03048c (0x90bd bytes) 000059 1787605167.520037 mapwrite 0x02ca10 thru 0x03b282 (0xe873 bytes) 000060 1787605167.721463 MIRROR_RESYNC 3 mirrors 000062 1787605169.094801 trunc from 0x03b283 to 0x00793b (0x33947 bytes) 000063 1787605169.210052 MIRROR_RESYNC 3 mirrors 000065 1787605170.016072 mapread 0x0052bc thru 0x00793b (0x2680 bytes) 000067 1787605170.035185 mapwrite 0x037df1 thru 0x03ccc4 (0x4ed4 bytes) 000068 1787605170.193730 MIRROR_RESYNC 3 mirrors 000069 1787605171.257284 mapwrite 0x02fcaa thru 0x03212a (0x2481 bytes) 000070 1787605171.500597 trunc from 0x03ccc5 to 0x02d514 (0xf7b0 bytes) 000072 1787605171.590547 mapwrite 0x02aef2 thru 0x031d63 (0x6e72 bytes) 000075 1787605171.778628 write 0x02db36 thru 0x038a00 (0xaecb bytes) truncating to largest ever: 0x3ba0b 000076 1787605171.786058 trunc from 0x038a01 to 0x03ba0a (0x300a bytes) 000080 1787605171.835077 read 0x018540 thru 0x01f340 (0x6e01 bytes) 000082 1787605171.866227 read 0x014edc thru 0x0227fb (0xd920 bytes) 000083 1787605171.905375 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba0b): Mirror 3: 0x5eae9128 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba0b): Mirror 3: 0x5eae9128 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba0b: skipped 000087 1787605172.330418 mapread 0x008472 thru 0x015b41 (0xd6d0 bytes) 000088 1787605172.358071 mapwrite 0x013076 thru 0x01d6ae (0xa639 bytes) 000089 1787605172.420795 trunc from 0x03ba0b to 0x01ef78 (0x1ca92 bytes) 000090 1787605172.446005 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ef79): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef79): Mirror 3: 0x4721835 (stale) CRC-32 checksum value for chunk [0, 0x1ef79): Mirror 1: 0xdd7843f0 Mirror 4: 0xdd7843f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef79: skipped 000092 1787605173.835908 trunc from 0x01ef79 to 0x00bb49 (0x1342f bytes) 000093 1787605173.952023 write 0x0067aa thru 0x010d51 (0xa5a8 bytes) 000095 1787605173.980875 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x10d52): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d52): Mirror 3: 0x4b975e52 (stale) Mirror 4: 0x4b975e52 (stale) CRC-32 checksum value for chunk [0, 0x10d52): Mirror 1: 0x6e271dd7 Mirror 5: 0x6e271dd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d52: skipped 000096 1787605175.182619 mapwrite 0x005171 thru 0x0103e7 (0xb277 bytes) 000099 1787605175.381667 mapread 0x00d86d thru 0x010d51 (0x34e5 bytes) 000100 1787605175.382823 mapwrite 0x013bcb thru 0x016982 (0x2db8 bytes) 000101 1787605175.459297 read 0x011f93 thru 0x014f34 (0x2fa2 bytes) 000102 1787605175.460364 trunc from 0x016983 to 0x02da56 (0x170d4 bytes) 000104 1787605175.495100 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da57): Mirror 3: 0x21c818bf (stale) Mirror 5: 0x1e4f7e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da57): Mirror 3: 0x21c818bf (stale) Mirror 5: 0x1e4f7e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da57: skipped 000108 1787605175.943801 mapread 0x009b8b thru 0x017f30 (0xe3a6 bytes) 000112 1787605175.976568 trunc from 0x02da57 to 0x01424b (0x1980b bytes) 000113 1787605175.993355 write 0x02559f thru 0x02a852 (0x52b4 bytes) 000114 1787605175.995450 mapread 0x008483 thru 0x00ef57 (0x6ad5 bytes) 000116 1787605175.998218 write 0x02d37a thru 0x02fdaf (0x2a36 bytes) 000117 1787605176.003880 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fdb0): Mirror 5: 0xbaec7888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdb0): Mirror 5: 0xbaec7888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdb0: skipped 000119 1787605176.368715 write 0x0302b6 thru 0x0320a0 (0x1deb bytes) 000120 1787605176.370015 read 0x006bbb thru 0x009f61 (0x33a7 bytes) 000121 1787605176.399659 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000125 1787605176.587972 trunc from 0x0320a1 to 0x008c1a (0x29486 bytes) 000127 1787605176.606200 trunc from 0x008c1b to 0x0137c0 (0xaba6 bytes) 000128 1787605176.624702 MIRROR_EXTEND to 2 mirrors 000129 1787605177.550521 trunc from 0x0137c1 to 0x02b876 (0x180b6 bytes) 000130 1787605177.627216 write 0x014325 thru 0x015205 (0xee1 bytes) 000131 1787605177.638015 mapwrite 0x026068 thru 0x02f39a (0x9333 bytes) 000138 1787605177.801181 mapread 0x008282 thru 0x009011 (0xd90 bytes) 000139 1787605177.817876 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000140 1787605178.004208 read 0x0151c6 thru 0x015a57 (0x892 bytes) 000143 1787605178.013579 read 0x00fec6 thru 0x012ccf (0x2e0a bytes) 000146 1787605178.030579 write 0x0058bd thru 0x0073ea (0x1b2e bytes) 000147 1787605178.032018 read 0x00e118 thru 0x00e14a (0x33 bytes) 000150 1787605178.034349 mapread 0x0020bb thru 0x00f5cb (0xd511 bytes) 000152 1787605178.036600 write 0x00348e thru 0x00ba36 (0x85a9 bytes) 000154 1787605178.043741 write 0x035e74 thru 0x03655e (0x6eb bytes) 000155 1787605178.048918 mapwrite 0x01b416 thru 0x020c35 (0x5820 bytes) 000157 1787605178.102748 trunc from 0x03655f to 0x01c966 (0x19bf8 bytes) 000160 1787605178.118137 read 0x002a01 thru 0x006dcb (0x43cb bytes) 000161 1787605178.118778 write 0x00f008 thru 0x0133ea (0x43e3 bytes) 000167 1787605178.123325 mapread 0x0092d8 thru 0x018834 (0xf55d bytes) 000168 1787605178.124965 trunc from 0x01c967 to 0x017741 (0x5225 bytes) 000169 1787605178.137840 write 0x022a7c thru 0x0263bd (0x3942 bytes) 000172 1787605178.143436 trunc from 0x0263be to 0x00b5d3 (0x1adea bytes) 000173 1787605178.161616 read 0x0061af thru 0x00a55d (0x43af bytes) 000176 1787605178.168906 MIRROR_EXTEND to 2 mirrors 000178 1787605178.924561 trunc from 0x00b5d4 to 0x027d00 (0x1c72d bytes) 000179 1787605179.077264 read 0x0127b2 thru 0x01def3 (0xb742 bytes) 000181 1787605179.109801 mapread 0x01aa43 thru 0x0269a3 (0xbf61 bytes) 000182 1787605179.144381 mapwrite 0x010251 thru 0x014842 (0x45f2 bytes) 000184 1787605179.214699 write 0x01fa58 thru 0x020f66 (0x150f bytes) truncating to largest ever: 0x3de70 000185 1787605179.232529 trunc from 0x027d01 to 0x03de6f (0x1616f bytes) 000186 1787605179.256172 trunc from 0x03de70 to 0x006368 (0x37b07 bytes) 000187 1787605179.278741 write 0x0167a1 thru 0x0241e0 (0xda40 bytes) 000188 1787605179.289316 MIRROR_RESYNC 2 mirrors 000189 1787605180.106769 MIRROR_EXTEND to 3 mirrors 000190 1787605180.981951 mapwrite 0x00e39d thru 0x01ade2 (0xca46 bytes) 000191 1787605181.215516 write 0x012349 thru 0x01d01b (0xacd3 bytes) 000193 1787605181.223358 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x241e1): Mirror 3: 0x2278b4e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241e1): Mirror 3: 0x2278b4e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241e1: skipped 000194 1787605181.614088 write 0x039dcc thru 0x03ffff (0x6234 bytes) 000195 1787605181.615734 trunc from 0x040000 to 0x009f67 (0x36098 bytes) 000196 1787605181.659737 trunc from 0x009f68 to 0x0153ee (0xb487 bytes) 000199 1787605181.681623 mapwrite 0x022018 thru 0x023701 (0x16ea bytes) 000200 1787605181.753793 read 0x01373c thru 0x013bd1 (0x496 bytes) 000202 1787605181.760218 mapread 0x00bb3a thru 0x0192e4 (0xd7ab bytes) 000203 1787605181.783931 read 0x00e63c thru 0x015be7 (0x75ac bytes) 000204 1787605181.785020 read 0x00e48c thru 0x016694 (0x8209 bytes) 000206 1787605181.786467 mapread 0x017f68 thru 0x01a291 (0x232a bytes) 000207 1787605181.788253 mapread 0x00bbab thru 0x0107e8 (0x4c3e bytes) 000211 1787605181.791894 read 0x01eeff thru 0x023701 (0x4803 bytes) 000213 1787605181.807255 read 0x007944 thru 0x0177e6 (0xfea3 bytes) 000214 1787605181.808111 read 0x009a99 thru 0x014abe (0xb026 bytes) 000215 1787605181.808737 mapread 0x00d8b6 thru 0x014d84 (0x74cf bytes) 000216 1787605181.810836 trunc from 0x023702 to 0x039729 (0x16028 bytes) 000217 1787605181.837743 trunc from 0x03972a to 0x032699 (0x7090 bytes) 000218 1787605181.863180 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000222 1787605181.991126 mapread 0x00b90e thru 0x01394e (0x8041 bytes) 000223 1787605182.013535 read 0x01a4c1 thru 0x01af96 (0xad6 bytes) 000226 1787605182.020689 write 0x00dd9a thru 0x0149f1 (0x6c58 bytes) 000227 1787605182.026680 MIRROR_EXTEND to 2 mirrors 000228 1787605182.884812 read 0x029e12 thru 0x0313e7 (0x75d6 bytes) 000229 1787605182.902144 mapread 0x00a76c thru 0x00c24d (0x1ae2 bytes) 000231 1787605182.966207 mapwrite 0x01f25a thru 0x01f613 (0x3ba bytes) 000232 1787605183.196230 mapwrite 0x03b489 thru 0x03ffff (0x4b77 bytes) 000233 1787605183.298426 read 0x006e6d thru 0x00c6d5 (0x5869 bytes) 000234 1787605183.311030 read 0x02760e thru 0x03283d (0xb230 bytes) 000236 1787605183.313629 read 0x000c24 thru 0x00bdf7 (0xb1d4 bytes) 000240 1787605183.317519 mapread 0x00120d thru 0x008096 (0x6e8a bytes) 000241 1787605183.319006 mapwrite 0x026672 thru 0x0313ef (0xad7e bytes) 000242 1787605183.365644 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000243 1787605183.511770 mapwrite 0x0390b2 thru 0x03ffff (0x6f4e bytes) 000245 1787605183.585812 trunc from 0x040000 to 0x02a862 (0x1579d bytes) 000246 1787605183.604262 write 0x03fb9d thru 0x03ffff (0x463 bytes) 000247 1787605183.605538 write 0x036c5d thru 0x03917d (0x2521 bytes) 000249 1787605183.620030 read 0x003e1a thru 0x0115de (0xd7c5 bytes) 000250 1787605183.637963 trunc from 0x040000 to 0x01f8f8 (0x20707 bytes) 000251 1787605183.662626 read 0x00f676 thru 0x01ca9a (0xd425 bytes) 000254 1787605183.682949 trunc from 0x01f8f9 to 0x039314 (0x19a1c bytes) 000255 1787605183.712707 write 0x001434 thru 0x00dfa3 (0xcb70 bytes) 000259 1787605183.720289 mapread 0x006307 thru 0x010f9c (0xac96 bytes) 000260 1787605183.721740 read 0x007876 thru 0x00bc64 (0x43ef bytes) 000261 1787605183.723968 write 0x01b55e thru 0x02a17e (0xec21 bytes) 000262 1787605183.733551 read 0x019dfa thru 0x021e8a (0x8091 bytes) 000264 1787605183.736955 mapwrite 0x03aaf4 thru 0x03ffff (0x550c bytes) 000265 1787605183.834597 read 0x01467e thru 0x01db33 (0x94b6 bytes) 000266 1787605183.835876 trunc from 0x040000 to 0x00e69a (0x31965 bytes) 000269 1787605183.864217 mapwrite 0x0024be thru 0x00fd42 (0xd885 bytes) 000270 1787605183.936268 read 0x00d8d1 thru 0x00e2e2 (0xa12 bytes) 000271 1787605183.937335 mapread 0x00662d thru 0x00cc52 (0x6626 bytes) 000273 1787605183.941625 read 0x004afe thru 0x00fd42 (0xb245 bytes) 000275 1787605183.943570 write 0x01e0ef thru 0x02bf0c (0xde1e bytes) 000276 1787605183.946354 mapread 0x01377a thru 0x01a610 (0x6e97 bytes) 000277 1787605183.955388 mapread 0x01ddbe thru 0x02bf0c (0xe14f bytes) 000279 1787605183.958023 read 0x006068 thru 0x008d72 (0x2d0b bytes) 000281 1787605183.964139 MIRROR_EXTEND to 2 mirrors 000284 1787605184.891344 write 0x027819 thru 0x02ef80 (0x7768 bytes) 000285 1787605184.988820 mapread 0x021e01 thru 0x02a5be (0x87be bytes) 000287 1787605185.036715 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 2: 0x76037bdf (stale) CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 1: 0x54dc3166 Mirror 3: 0x54dc3166 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef81: skipped 000288 1787605186.687304 mapread 0x0172e3 thru 0x01f95d (0x867b bytes) 000289 1787605186.741042 MIRROR_RESYNC 3 mirrors 000291 1787605187.741030 mapread 0x02b9de thru 0x02ef80 (0x35a3 bytes) 000295 1787605187.758760 write 0x03dfa6 thru 0x03ffff (0x205a bytes) 000296 1787605187.833304 write 0x020899 thru 0x02723a (0x69a2 bytes) 000297 1787605187.851814 write 0x03f933 thru 0x03ffff (0x6cd bytes) 000298 1787605187.852800 mapread 0x00a877 thru 0x00f191 (0x491b bytes) 000299 1787605187.866113 read 0x03190b thru 0x038c29 (0x731f bytes) 000301 1787605187.876225 mapread 0x00383a thru 0x0105ae (0xcd75 bytes) 000303 1787605187.878950 trunc from 0x040000 to 0x023ace (0x1c531 bytes) 000304 1787605187.917606 read 0x017668 thru 0x023ace (0xc467 bytes) 000305 1787605187.930878 mapread 0x01ffe7 thru 0x023ace (0x3ae8 bytes) 000306 1787605187.934006 mapread 0x00c37d thru 0x016ba9 (0xa82d bytes) 000309 1787605187.937686 mapwrite 0x03bb06 thru 0x03ffff (0x44fa bytes) 000310 1787605188.035406 read 0x0369da thru 0x039105 (0x272c bytes) 000314 1787605188.038968 write 0x01916e thru 0x022ae2 (0x9975 bytes) 000315 1787605188.040710 read 0x02a36e thru 0x02e373 (0x4006 bytes) 000317 1787605188.042594 mapread 0x02b30f thru 0x03ae4c (0xfb3e bytes) 000319 1787605188.044320 read 0x03489e thru 0x03ffff (0xb762 bytes) 000320 1787605188.051654 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x81358527 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x81358527 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1787605188.478194 MIRROR_RESYNC 2 mirrors 000322 1787605190.499137 read 0x01cc0a thru 0x02c153 (0xf54a bytes) 000327 1787605190.510059 mapwrite 0x01b0fe thru 0x01b32c (0x22f bytes) 000329 1787605190.689599 mapread 0x01e11a thru 0x01e960 (0x847 bytes) 000330 1787605190.699544 mapread 0x025f18 thru 0x02ffc5 (0xa0ae bytes) 000331 1787605190.709927 mapwrite 0x03489a thru 0x03c014 (0x777b bytes) 000332 1787605190.796313 trunc from 0x040000 to 0x020434 (0x1fbcb bytes) 000333 1787605190.823719 write 0x010332 thru 0x017e2c (0x7afb bytes) 000334 1787605190.832296 trunc from 0x020435 to 0x02d529 (0xd0f5 bytes) 000335 1787605190.852692 read 0x001dac thru 0x0034ab (0x1700 bytes) 000337 1787605190.884118 mapwrite 0x01319c thru 0x0142a8 (0x110d bytes) 000338 1787605190.939890 mapwrite 0x030c81 thru 0x034c09 (0x3f89 bytes) 000339 1787605191.010857 mapread 0x0158b5 thru 0x019635 (0x3d81 bytes) 000341 1787605191.012691 trunc from 0x034c0a to 0x00f13f (0x25aca bytes) 000342 1787605191.029787 mapread 0x002134 thru 0x009b38 (0x7a05 bytes) 000344 1787605191.032864 mapread 0x0075eb thru 0x007b85 (0x59b bytes) truncating to largest ever: 0x3efbd 000345 1787605191.034467 trunc from 0x00f140 to 0x03efbc (0x2fe7d bytes) 000347 1787605191.053644 mapwrite 0x00d20f thru 0x00ec40 (0x1a32 bytes) 000348 1787605191.093289 write 0x00cb24 thru 0x016065 (0x9542 bytes) 000349 1787605191.103817 mapread 0x028dea thru 0x03527d (0xc494 bytes) 000350 1787605191.167269 read 0x02e2a6 thru 0x036d49 (0x8aa4 bytes) 000351 1787605191.168328 read 0x031b0b thru 0x03efbc (0xd4b2 bytes) 000355 1787605191.173103 read 0x011272 thru 0x0195b0 (0x833f bytes) 000357 1787605191.195876 write 0x00cb73 thru 0x0178f0 (0xad7e bytes) 000358 1787605191.198727 read 0x035798 thru 0x03e4de (0x8d47 bytes) 000360 1787605191.201324 trunc from 0x03efbd to 0x02ed64 (0x10258 bytes) 000361 1787605191.222057 write 0x0092ba thru 0x00fa7a (0x67c1 bytes) 000363 1787605191.227557 trunc from 0x02ed65 to 0x000a4c (0x2e318 bytes) 000364 1787605191.255720 mapwrite 0x008464 thru 0x012ea7 (0xaa44 bytes) 000366 1787605191.350915 mapwrite 0x034df4 thru 0x03ffff (0xb20c bytes) 000370 1787605191.492332 trunc from 0x040000 to 0x02290f (0x1d6f0 bytes) 000371 1787605191.526787 MIRROR_RESYNC 2 mirrors 000374 1787605192.387717 mapwrite 0x020e7d thru 0x021695 (0x819 bytes) 000375 1787605192.596906 read 0x018e93 thru 0x01c0ad (0x321b bytes) 000377 1787605192.604799 mapwrite 0x033786 thru 0x037b6b (0x43e6 bytes) 000380 1787605192.701871 read 0x024af8 thru 0x033393 (0xe89c bytes) 000382 1787605192.716973 write 0x004091 thru 0x008388 (0x42f8 bytes) 000383 1787605192.718452 read 0x0227ef thru 0x031ba2 (0xf3b4 bytes) 000384 1787605192.719895 read 0x00b0db thru 0x01b084 (0xffaa bytes) 000387 1787605192.722309 write 0x01fed5 thru 0x02439f (0x44cb bytes) 000391 1787605192.731101 mapwrite 0x029138 thru 0x0309d6 (0x789f bytes) 000392 1787605192.779753 mapread 0x014021 thru 0x01ef9e (0xaf7e bytes) 000393 1787605192.780933 mapwrite 0x03758f thru 0x03ffff (0x8a71 bytes) 000394 1787605192.863766 MIRROR_RESYNC 2 mirrors 000395 1787605193.588335 trunc from 0x040000 to 0x02b6c1 (0x1493e bytes) 000396 1787605193.715560 MIRROR_RESYNC 2 mirrors 000398 1787605195.102384 write 0x004fc0 thru 0x010e6c (0xbead bytes) 000399 1787605195.192647 trunc from 0x02b6c2 to 0x03e139 (0x12a78 bytes) 000401 1787605195.230816 trunc from 0x03e13a to 0x016ffa (0x2713f bytes) 000403 1787605195.253885 trunc from 0x016ffb to 0x03883f (0x21845 bytes) 000405 1787605195.284209 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000406 1787605195.493741 mapread 0x00656c thru 0x00e3cd (0x7e62 bytes) 000407 1787605195.520452 mapread 0x01efd7 thru 0x022813 (0x383d bytes) 000415 1787605195.535423 mapread 0x032ea6 thru 0x033055 (0x1b0 bytes) 000418 1787605195.553759 trunc from 0x038840 to 0x03732e (0x1511 bytes) 000420 1787605195.574908 trunc from 0x03732f to 0x01f45d (0x17ed1 bytes) 000423 1787605195.598359 write 0x02d7fd thru 0x037abd (0xa2c1 bytes) 000424 1787605195.600164 trunc from 0x037abe to 0x0248e5 (0x131d8 bytes) 000426 1787605195.624633 write 0x038e02 thru 0x03dc56 (0x4e55 bytes) 000427 1787605195.626667 mapwrite 0x01aedd thru 0x021cb0 (0x6dd4 bytes) 000429 1787605195.687160 mapread 0x021ff5 thru 0x028ef9 (0x6f05 bytes) 000430 1787605195.689000 write 0x032a7c thru 0x03c2cb (0x9850 bytes) 000431 1787605195.696126 write 0x004a9e thru 0x00c94a (0x7ead bytes) 000432 1787605195.697883 write 0x02a5c6 thru 0x0330d5 (0x8b10 bytes) 000433 1787605195.706864 read 0x03cc59 thru 0x03dc56 (0xffe bytes) 000434 1787605195.707711 mapwrite 0x019c6b thru 0x023512 (0x98a8 bytes) 000435 1787605195.756039 mapwrite 0x0166cd thru 0x017f4a (0x187e bytes) 000437 1787605195.813533 read 0x036c8f thru 0x03d665 (0x69d7 bytes) 000438 1787605195.814727 mapwrite 0x011a1e thru 0x019856 (0x7e39 bytes) 000439 1787605195.888127 mapwrite 0x0356a0 thru 0x03f9f4 (0xa355 bytes) 000441 1787605195.965885 trunc from 0x03f9f5 to 0x03395c (0xc098 bytes) 000442 1787605195.987739 trunc from 0x03395d to 0x036577 (0x2c1b bytes) 000444 1787605196.006364 mapread 0x01a0de thru 0x01bd44 (0x1c67 bytes) 000446 1787605196.010260 read 0x019e78 thru 0x01ea7e (0x4c07 bytes) 000447 1787605196.011621 trunc from 0x036578 to 0x0314f8 (0x507f bytes) 000451 1787605196.031873 read 0x0246b6 thru 0x0296a9 (0x4ff4 bytes) 000452 1787605196.032810 trunc from 0x0314f9 to 0x018e05 (0x186f3 bytes) 000454 1787605196.050758 mapread 0x0002f1 thru 0x007a73 (0x7783 bytes) 000456 1787605196.053443 mapread 0x00a4ff thru 0x00f1a9 (0x4cab bytes) 000459 1787605196.055792 write 0x006903 thru 0x00e17b (0x7879 bytes) 000461 1787605196.065660 mapread 0x0111d8 thru 0x0154b3 (0x42dc bytes) 000463 1787605196.067935 mapwrite 0x014f7f thru 0x01883c (0x38be bytes) 000464 1787605196.140185 mapread 0x014f8a thru 0x018e05 (0x3e7c bytes) 000465 1787605196.145437 read 0x015dbc thru 0x018e05 (0x304a bytes) 000467 1787605196.154647 mapread 0x00e530 thru 0x018e05 (0xa8d6 bytes) 000470 1787605196.158920 trunc from 0x018e06 to 0x002b4f (0x162b6 bytes) 000471 1787605196.181733 write 0x03c78b thru 0x03ffff (0x3875 bytes) 000472 1787605196.183867 trunc from 0x040000 to 0x022e3c (0x1d1c3 bytes) 000474 1787605196.205914 read 0x00fe36 thru 0x01ed29 (0xeef4 bytes) 000475 1787605196.217081 trunc from 0x022e3d to 0x01867f (0xa7bd bytes) 000479 1787605196.236092 write 0x02d446 thru 0x03b824 (0xe3df bytes) 000480 1787605196.238575 mapread 0x008c76 thru 0x01489a (0xbc25 bytes) 000484 1787605196.241568 read 0x033df7 thru 0x039b59 (0x5d63 bytes) 000486 1787605196.243810 mapwrite 0x039cf4 thru 0x03ffff (0x630c bytes) 000488 1787605196.326316 mapwrite 0x02e811 thru 0x03c2d8 (0xdac8 bytes) 000490 1787605196.374926 mapread 0x02d6b1 thru 0x030013 (0x2963 bytes) 000491 1787605196.377090 trunc from 0x040000 to 0x0018c9 (0x3e736 bytes) 000494 1787605196.398890 mapread 0x0004c3 thru 0x0018c9 (0x1407 bytes) 000496 1787605196.401265 read 0x000676 thru 0x0018c9 (0x1254 bytes) 000498 1787605196.402899 mapwrite 0x019290 thru 0x029149 (0xfeba bytes) 000499 1787605196.482806 mapwrite 0x0225b8 thru 0x02465c (0x20a5 bytes) 000500 1787605196.536813 mapwrite 0x03e43b thru 0x03ffff (0x1bc5 bytes) 000502 1787605196.680908 mapwrite 0x018311 thru 0x018350 (0x40 bytes) 000503 1787605196.793056 mapread 0x0026c9 thru 0x002a4f (0x387 bytes) 000505 1787605196.794620 write 0x03c1be thru 0x03ffff (0x3e42 bytes) 000508 1787605196.801829 mapread 0x02fe09 thru 0x03aa79 (0xac71 bytes) 000510 1787605196.805886 mapwrite 0x03d937 thru 0x03ffff (0x26c9 bytes) 000511 1787605196.846863 mapwrite 0x010e1d thru 0x01bea4 (0xb088 bytes) 000513 1787605196.910273 MIRROR_EXTEND to 2 mirrors 000519 1787605198.074355 mapread 0x006d17 thru 0x00b6ed (0x49d7 bytes) 000521 1787605198.148406 mapwrite 0x02c50c thru 0x03090c (0x4401 bytes) 000523 1787605198.471902 MIRROR_RESYNC 2 mirrors 000524 1787605199.361220 trunc from 0x040000 to 0x005856 (0x3a7a9 bytes) 000528 1787605199.490699 write 0x004c80 thru 0x007a7c (0x2dfd bytes) 000530 1787605199.503920 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000531 1787605199.707750 mapread 0x007361 thru 0x007a7c (0x71c bytes) 000532 1787605199.718532 trunc from 0x007a7d to 0x018001 (0x10585 bytes) 000533 1787605199.743658 mapread 0x010861 thru 0x01676a (0x5f0a bytes) 000534 1787605199.762184 write 0x02fc01 thru 0x030b6d (0xf6d bytes) 000537 1787605199.766835 trunc from 0x030b6e to 0x008e36 (0x27d37 bytes) 000539 1787605199.786655 write 0x015528 thru 0x0207cc (0xb2a5 bytes) 000541 1787605199.790221 trunc from 0x0207cd to 0x03c28e (0x1bac2 bytes) 000543 1787605199.816542 mapread 0x025f3c thru 0x027640 (0x1705 bytes) 000546 1787605199.832837 read 0x00348e thru 0x00ecfc (0xb86f bytes) 000550 1787605199.848406 trunc from 0x03c28f to 0x0168df (0x259af bytes) 000552 1787605199.877457 mapread 0x00b547 thru 0x00c4af (0xf69 bytes) 000558 1787605199.886893 mapwrite 0x008808 thru 0x0140f0 (0xb8e9 bytes) 000560 1787605199.977362 mapread 0x005802 thru 0x00b96d (0x616c bytes) 000563 1787605199.984045 trunc from 0x0168e0 to 0x00af9e (0xb941 bytes) 000566 1787605200.012581 read 0x006c4e thru 0x00af9e (0x4351 bytes) 000567 1787605200.018846 MIRROR_EXTEND to 2 mirrors 000568 1787605200.898639 write 0x03071c thru 0x03f5e7 (0xeecc bytes) 000570 1787605201.045007 trunc from 0x03f5e8 to 0x0247dd (0x1ae0a bytes) 000571 1787605201.129839 mapwrite 0x027559 thru 0x02a75d (0x3205 bytes) 000572 1787605201.333795 mapread 0x028d88 thru 0x02a75d (0x19d6 bytes) 000574 1787605201.337765 trunc from 0x02a75e to 0x037d4d (0xd5f0 bytes) 000576 1787605201.370794 mapread 0x0043c8 thru 0x00a121 (0x5d5a bytes) 000579 1787605201.401679 trunc from 0x037d4e to 0x03a7b5 (0x2a68 bytes) 000580 1787605201.421634 mapwrite 0x028a47 thru 0x02aa3b (0x1ff5 bytes) 000581 1787605201.489141 trunc from 0x03a7b6 to 0x039fe1 (0x7d4 bytes) 000582 1787605201.524886 mapwrite 0x00b3e0 thru 0x00f199 (0x3dba bytes) 000583 1787605201.608636 trunc from 0x039fe2 to 0x03d152 (0x3171 bytes) 000585 1787605201.643077 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000587 1787605201.976160 read 0x0064bd thru 0x008b60 (0x26a4 bytes) 000588 1787605201.984441 mapwrite 0x012053 thru 0x015889 (0x3837 bytes) 000589 1787605202.131231 mapread 0x005a91 thru 0x011ae6 (0xc056 bytes) 000596 1787605202.151213 trunc from 0x03d153 to 0x027a5f (0x156f3 bytes) 000598 1787605202.176814 trunc from 0x027a60 to 0x01971d (0xe342 bytes) 000599 1787605202.205806 trunc from 0x01971e to 0x01d033 (0x3916 bytes) 000600 1787605202.235957 trunc from 0x01d034 to 0x00bbfa (0x11439 bytes) 000601 1787605202.272702 write 0x01a7e3 thru 0x0217c5 (0x6fe3 bytes) 000602 1787605202.274087 read 0x0109b0 thru 0x019538 (0x8b89 bytes) 000604 1787605202.285092 trunc from 0x0217c6 to 0x0285b6 (0x6df1 bytes) 000605 1787605202.306305 mapwrite 0x021812 thru 0x02f60c (0xddfb bytes) 000606 1787605202.416532 mapwrite 0x00bb20 thru 0x017fd3 (0xc4b4 bytes) 000610 1787605202.481726 mapwrite 0x01d482 thru 0x02483f (0x73be bytes) 000611 1787605202.568072 write 0x0201a0 thru 0x0267b4 (0x6615 bytes) 000612 1787605202.569099 write 0x00fbc0 thru 0x01a054 (0xa495 bytes) 000613 1787605202.573117 mapwrite 0x006b14 thru 0x00dbde (0x70cb bytes) 000618 1787605202.694588 write 0x000951 thru 0x00f888 (0xef38 bytes) 000620 1787605202.712013 MIRROR_EXTEND to 2 mirrors 000621 1787605204.014879 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000622 1787605204.226765 mapread 0x01545d thru 0x01553b (0xdf bytes) 000623 1787605204.240917 trunc from 0x02f60d to 0x00f7c9 (0x1fe43 bytes) 000624 1787605204.294475 write 0x008cae thru 0x00b308 (0x265b bytes) 000628 1787605204.331811 mapwrite 0x004862 thru 0x0099df (0x517e bytes) 000631 1787605204.446737 read 0x005232 thru 0x00bb96 (0x6965 bytes) 000632 1787605204.447923 trunc from 0x00f7ca to 0x021d73 (0x125aa bytes) 000633 1787605204.479876 read 0x009bd2 thru 0x00a71c (0xb4b bytes) 000636 1787605204.486880 mapwrite 0x003760 thru 0x00781c (0x40bd bytes) 000637 1787605204.548857 mapwrite 0x021840 thru 0x02dd91 (0xc552 bytes) 000640 1787605204.680428 mapwrite 0x0048ae thru 0x011127 (0xc87a bytes) 000641 1787605204.740646 write 0x006fdc thru 0x00829d (0x12c2 bytes) 000642 1787605204.743098 trunc from 0x02dd92 to 0x030535 (0x27a4 bytes) 000643 1787605204.767210 MIRROR_EXTEND to 2 mirrors 000644 1787605205.929273 trunc from 0x030536 to 0x00335e (0x2d1d7 bytes) 000645 1787605206.208473 read 0x00289f thru 0x00335e (0xac0 bytes) 000650 1787605206.226451 write 0x0000d0 thru 0x00afd2 (0xaf03 bytes) 000651 1787605206.234361 write 0x0147cd thru 0x01ea82 (0xa2b6 bytes) 000652 1787605206.240862 trunc from 0x01ea83 to 0x016582 (0x8500 bytes) 000653 1787605206.297436 mapread 0x00c93c thru 0x016582 (0x9c47 bytes) 000654 1787605206.329670 write 0x03b6fd thru 0x03ffff (0x4903 bytes) 000655 1787605206.332466 trunc from 0x040000 to 0x03d768 (0x2897 bytes) 000657 1787605206.356173 write 0x002c7a thru 0x0030e4 (0x46b bytes) 000660 1787605206.370405 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d769): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d769): Mirror 2: 0x71d5c610 (stale) CRC-32 checksum value for chunk [0, 0x3d769): Mirror 1: 0x6299b01c Mirror 3: 0x6299b01c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d769: skipped 000661 1787605207.639896 MIRROR_RESYNC 3 mirrors 000662 1787605208.527132 read 0x014442 thru 0x023654 (0xf213 bytes) 000665 1787605208.538212 mapread 0x0095cf thru 0x00a656 (0x1088 bytes) 000666 1787605208.550743 mapread 0x018806 thru 0x02285b (0xa056 bytes) 000668 1787605208.552131 read 0x00788e thru 0x011cb1 (0xa424 bytes) 000673 1787605208.579606 MIRROR_RESYNC 3 mirrors 000674 1787605208.991670 MIRROR_EXTEND to 4 mirrors 000677 1787605210.203276 write 0x03b9d7 thru 0x03ffff (0x4629 bytes) 000679 1787605210.272328 trunc from 0x040000 to 0x02008c (0x1ff73 bytes) 000680 1787605210.307888 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2008d): Mirror 2: 0xbe5bdadd (stale) Mirror 4: 0xbe5bdadd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2008d): Mirror 2: 0xbe5bdadd (stale) Mirror 4: 0xbe5bdadd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2008d: skipped 000681 1787605210.695150 mapread 0x012afd thru 0x02008c (0xd590 bytes) 000683 1787605210.767963 write 0x03d209 thru 0x03ffff (0x2df7 bytes) 000684 1787605210.776730 trunc from 0x040000 to 0x017aee (0x28511 bytes) 000686 1787605210.833812 mapwrite 0x035b91 thru 0x03ffff (0xa46f bytes) 000687 1787605211.003585 read 0x01f0b3 thru 0x025bc7 (0x6b15 bytes) 000688 1787605211.050587 MIRROR_RESYNC 3 mirrors 000689 1787605212.374809 mapread 0x0123b8 thru 0x01a6ca (0x8313 bytes) 000690 1787605212.423538 mapwrite 0x009d78 thru 0x00ed1e (0x4fa7 bytes) 000692 1787605212.653774 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x352fcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x352fcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000695 1787605213.126746 read 0x0017ff thru 0x009372 (0x7b74 bytes) 000696 1787605213.143113 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000697 1787605213.297628 mapread 0x01a587 thru 0x01abc1 (0x63b bytes) 000698 1787605213.310595 mapwrite 0x00170c thru 0x008431 (0x6d26 bytes) 000699 1787605213.438104 mapwrite 0x03c4e2 thru 0x03ffff (0x3b1e bytes) 000700 1787605213.504605 read 0x03dfbd thru 0x03ffff (0x2043 bytes) 000702 1787605213.508775 mapwrite 0x00b6da thru 0x00cbc0 (0x14e7 bytes) 000703 1787605213.653324 mapread 0x019fdc thru 0x021b00 (0x7b25 bytes) 000704 1787605213.685581 mapread 0x004978 thru 0x0095c6 (0x4c4f bytes) 000705 1787605213.699121 write 0x03e424 thru 0x03ffff (0x1bdc bytes) 000708 1787605213.732058 mapwrite 0x02f00a thru 0x039a79 (0xaa70 bytes) 000709 1787605213.908226 mapread 0x03a300 thru 0x03f95b (0x565c bytes) 000711 1787605213.922863 read 0x00bc46 thru 0x013e1b (0x81d6 bytes) 000712 1787605213.941317 write 0x021503 thru 0x02cc8f (0xb78d bytes) 000713 1787605213.943247 mapread 0x00af4b thru 0x0133d6 (0x848c bytes) 000714 1787605213.944740 mapread 0x01cbf5 thru 0x023596 (0x69a2 bytes) 000717 1787605213.978762 MIRROR_EXTEND to 2 mirrors 000718 1787605215.023973 MIRROR_RESYNC 2 mirrors 000721 1787605215.443180 mapread 0x00ef76 thru 0x01c096 (0xd121 bytes) 000723 1787605215.499316 mapread 0x013a0c thru 0x01acb5 (0x72aa bytes) 000724 1787605215.516971 trunc from 0x040000 to 0x01e757 (0x218a8 bytes) 000725 1787605215.601650 mapwrite 0x010952 thru 0x01e4de (0xdb8d bytes) 000726 1787605215.748724 mapread 0x0026df thru 0x00c94a (0xa26c bytes) 000727 1787605215.772551 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000729 1787605216.011702 mapwrite 0x012d34 thru 0x01c5c9 (0x9896 bytes) 000731 1787605216.124620 trunc from 0x01e758 to 0x000c1b (0x1db3c bytes) 000735 1787605216.153715 read 0x000313 thru 0x000c1b (0x909 bytes) 000740 1787605216.163100 write 0x03d690 thru 0x03ffff (0x2970 bytes) 000741 1787605216.165109 mapread 0x0206e3 thru 0x02ec73 (0xe591 bytes) 000743 1787605216.183007 read 0x005d18 thru 0x011b34 (0xbe1d bytes) 000744 1787605216.196051 mapwrite 0x02380d thru 0x028d28 (0x551c bytes) 000745 1787605216.255100 write 0x00af95 thru 0x010b66 (0x5bd2 bytes) 000746 1787605216.257319 write 0x0296be thru 0x0344e7 (0xae2a bytes) 000747 1787605216.261457 write 0x01273a thru 0x021d7a (0xf641 bytes) 000749 1787605216.267311 read 0x00df3d thru 0x017db9 (0x9e7d bytes) 000752 1787605216.271683 mapwrite 0x02dc49 thru 0x03a953 (0xcd0b bytes) 000753 1787605216.376391 mapread 0x0185ab thru 0x01a7b8 (0x220e bytes) 000755 1787605216.380087 trunc from 0x040000 to 0x00c4f8 (0x33b07 bytes) 000757 1787605216.405363 mapwrite 0x029675 thru 0x038fbe (0xf94a bytes) 000758 1787605216.482436 mapwrite 0x01c2d9 thru 0x0288b9 (0xc5e1 bytes) 000759 1787605216.534914 read 0x031495 thru 0x038fbe (0x7b2a bytes) 000760 1787605216.535595 write 0x027cff thru 0x02bad3 (0x3dd5 bytes) 000761 1787605216.536638 write 0x011961 thru 0x01d3ee (0xba8e bytes) 000764 1787605216.541442 read 0x01e15c thru 0x02a214 (0xc0b9 bytes) 000765 1787605216.542662 mapread 0x038755 thru 0x038fbe (0x86a bytes) 000766 1787605216.545085 write 0x03ccda thru 0x03ffff (0x3326 bytes) 000767 1787605216.547101 trunc from 0x040000 to 0x031e12 (0xe1ed bytes) 000768 1787605216.568226 mapread 0x0136e0 thru 0x01ff9b (0xc8bc bytes) 000772 1787605216.582009 MIRROR_EXTEND to 2 mirrors 000773 1787605217.261402 trunc from 0x031e13 to 0x013ef1 (0x1df21 bytes) 000775 1787605217.386576 read 0x00fc2d thru 0x013ef1 (0x42c5 bytes) 000777 1787605217.413291 read 0x000766 thru 0x00bcb7 (0xb552 bytes) 000779 1787605217.461919 trunc from 0x013ef2 to 0x0372c4 (0x233d3 bytes) 000780 1787605217.528714 trunc from 0x0372c5 to 0x01198f (0x25935 bytes) 000787 1787605217.565296 mapwrite 0x00badd thru 0x013110 (0x7634 bytes) 000788 1787605217.729056 mapwrite 0x009c93 thru 0x00e6ba (0x4a28 bytes) 000790 1787605217.794385 mapread 0x01117f thru 0x013110 (0x1f92 bytes) 000791 1787605217.799973 mapread 0x00c61c thru 0x013110 (0x6af5 bytes) 000793 1787605217.803237 mapwrite 0x028c74 thru 0x0298ff (0xc8c bytes) 000796 1787605217.949275 write 0x00f4ef thru 0x013470 (0x3f82 bytes) 000797 1787605217.977260 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000800 1787605218.191382 read 0x00ce3c thru 0x0120d2 (0x5297 bytes) 000805 1787605218.212644 trunc from 0x029900 to 0x034b62 (0xb263 bytes) 000806 1787605218.230407 mapread 0x00b7f1 thru 0x01951e (0xdd2e bytes) 000810 1787605218.258032 mapwrite 0x03c0a1 thru 0x03ffff (0x3f5f bytes) 000813 1787605218.336035 read 0x03b06f thru 0x03ffff (0x4f91 bytes) 000815 1787605218.339276 mapread 0x018009 thru 0x019446 (0x143e bytes) 000816 1787605218.340926 read 0x03d179 thru 0x03ffff (0x2e87 bytes) 000817 1787605218.342365 mapread 0x011c9f thru 0x0199b3 (0x7d15 bytes) 000820 1787605218.345455 trunc from 0x040000 to 0x021436 (0x1ebc9 bytes) 000821 1787605218.363750 mapread 0x015da1 thru 0x01f34c (0x95ac bytes) 000822 1787605218.375600 trunc from 0x021437 to 0x03562a (0x141f4 bytes) 000824 1787605218.395820 read 0x002085 thru 0x01031b (0xe297 bytes) 000825 1787605218.396628 mapread 0x0004fe thru 0x0090fd (0x8c00 bytes) 000827 1787605218.400231 mapwrite 0x0222a9 thru 0x027adf (0x5837 bytes) 000828 1787605218.455194 write 0x025349 thru 0x03312a (0xdde2 bytes) 000829 1787605218.456697 read 0x00b6c3 thru 0x00f3fc (0x3d3a bytes) 000830 1787605218.457299 mapwrite 0x0345a8 thru 0x03b284 (0x6cdd bytes) 000831 1787605218.539368 mapwrite 0x01a8fe thru 0x0252d8 (0xa9db bytes) 000832 1787605218.602009 read 0x002973 thru 0x008ad1 (0x615f bytes) 000833 1787605218.604061 mapread 0x03610f thru 0x036e9a (0xd8c bytes) 000835 1787605218.612283 read 0x00a0c8 thru 0x015280 (0xb1b9 bytes) 000840 1787605218.628352 mapread 0x039d69 thru 0x03b284 (0x151c bytes) 000842 1787605218.638249 read 0x018f75 thru 0x0252ee (0xc37a bytes) 000843 1787605218.641337 trunc from 0x03b285 to 0x017b38 (0x2374c bytes) 000844 1787605218.682828 write 0x00e6fb thru 0x01a20f (0xbb15 bytes) 000845 1787605218.684427 write 0x03d789 thru 0x03ffff (0x2877 bytes) 000847 1787605218.688282 write 0x032424 thru 0x03ea08 (0xc5e5 bytes) 000848 1787605218.699018 write 0x01e932 thru 0x02ccd0 (0xe39f bytes) 000850 1787605218.703754 write 0x0107f4 thru 0x0119e9 (0x11f6 bytes) 000853 1787605218.706739 mapwrite 0x031156 thru 0x037aaf (0x695a bytes) 000854 1787605218.774796 write 0x00dc5b thru 0x010029 (0x23cf bytes) 000856 1787605218.778186 write 0x03af45 thru 0x03ffff (0x50bb bytes) 000859 1787605218.781418 write 0x039c84 thru 0x03ffff (0x637c bytes) 000860 1787605218.784003 mapread 0x03f1e6 thru 0x03ffff (0xe1a bytes) 000861 1787605218.785949 write 0x029e85 thru 0x037953 (0xdacf bytes) 000862 1787605218.787972 mapread 0x037189 thru 0x03cb37 (0x59af bytes) 000863 1787605218.790169 trunc from 0x040000 to 0x007db7 (0x38248 bytes) 000866 1787605218.814798 write 0x030c0a thru 0x03a608 (0x99ff bytes) 000867 1787605218.819286 write 0x01e4fc thru 0x025866 (0x736b bytes) 000869 1787605218.836821 write 0x019eb0 thru 0x0272f1 (0xd442 bytes) 000870 1787605218.854943 mapwrite 0x030e50 thru 0x03ffff (0xf1b0 bytes) 000873 1787605218.981979 read 0x029b4e thru 0x032e88 (0x933b bytes) 000874 1787605218.996242 trunc from 0x040000 to 0x003e4e (0x3c1b1 bytes) 000875 1787605219.028299 write 0x02b8d0 thru 0x03a6eb (0xee1c bytes) 000876 1787605219.030565 read 0x01651b thru 0x018c31 (0x2717 bytes) 000878 1787605219.043755 read 0x017d8f thru 0x020e8b (0x90fd bytes) 000879 1787605219.044911 trunc from 0x03a6ec to 0x001dfd (0x388ee bytes) 000880 1787605219.064896 mapread 0x001cde thru 0x001dfd (0x120 bytes) 000881 1787605219.067430 write 0x03571d thru 0x0368e2 (0x11c6 bytes) 000885 1787605219.075009 mapwrite 0x034d44 thru 0x03ea3a (0x9cf7 bytes) 000887 1787605219.192347 mapread 0x01338b thru 0x01de69 (0xaadf bytes) 000888 1787605219.194357 read 0x034c22 thru 0x03dd34 (0x9113 bytes) 000889 1787605219.195959 trunc from 0x03ea3b to 0x02439a (0x1a6a0 bytes) 000891 1787605219.220342 read 0x0009ec thru 0x00fc30 (0xf245 bytes) 000893 1787605219.221905 write 0x01dd0f thru 0x024f0b (0x71fd bytes) 000894 1787605219.224544 mapwrite 0x00e43d thru 0x015fa6 (0x7b6a bytes) 000897 1787605219.327168 trunc from 0x024f0c to 0x00f1bd (0x15d4e bytes) 000898 1787605219.374969 mapwrite 0x02e073 thru 0x033a1d (0x59ab bytes) 000899 1787605219.510040 mapread 0x020f50 thru 0x02456f (0x3620 bytes) 000901 1787605219.518260 read 0x02db58 thru 0x033a1d (0x5ec6 bytes) 000903 1787605219.524582 mapwrite 0x02d4c6 thru 0x036f03 (0x9a3e bytes) 000904 1787605219.622708 mapwrite 0x002324 thru 0x006344 (0x4021 bytes) 000906 1787605219.686270 MIRROR_EXTEND to 2 mirrors 000909 1787605221.032709 write 0x004c60 thru 0x0096c1 (0x4a62 bytes) 000911 1787605221.170168 MIRROR_RESYNC 2 mirrors 000913 1787605222.321454 write 0x02b750 thru 0x02d3a5 (0x1c56 bytes) 000915 1787605222.370512 trunc from 0x036f04 to 0x0356ab (0x1858 bytes) 000917 1787605222.392258 mapwrite 0x018af2 thru 0x01c668 (0x3b77 bytes) 000919 1787605222.455467 mapread 0x02f797 thru 0x0356ab (0x5f15 bytes) 000920 1787605222.468429 read 0x034c26 thru 0x0356ab (0xa86 bytes) 000927 1787605222.478205 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) 000928 1787605222.617385 read 0x001a2d thru 0x01164d (0xfc21 bytes) 000929 1787605222.629221 mapwrite 0x00f757 thru 0x01d782 (0xe02c bytes) 000931 1787605222.702235 mapwrite 0x03d500 thru 0x03e805 (0x1306 bytes) 000934 1787605222.781843 write 0x002ac2 thru 0x012368 (0xf8a7 bytes) 000936 1787605222.792386 mapwrite 0x024cdd thru 0x029933 (0x4c57 bytes) 000938 1787605222.885313 mapwrite 0x00b8ed thru 0x012297 (0x69ab bytes) 000941 1787605222.950752 write 0x009d36 thru 0x00d505 (0x37d0 bytes) 000942 1787605222.958832 MIRROR_EXTEND to 2 mirrors 000944 1787605224.153395 mapread 0x015be4 thru 0x01da50 (0x7e6d bytes) 000948 1787605224.225979 trunc from 0x03e806 to 0x00236b (0x3c49a bytes) 000950 1787605224.311426 trunc from 0x00236c to 0x016c93 (0x14928 bytes) 000952 1787605224.340248 mapwrite 0x02b20f thru 0x02e517 (0x3309 bytes) 000954 1787605224.415220 trunc from 0x02e518 to 0x0106be (0x1de59 bytes) 000958 1787605224.441009 write 0x00ffad thru 0x016f04 (0x6f58 bytes) 000959 1787605224.449358 write 0x036795 thru 0x03ffff (0x986b bytes) 000961 1787605224.452552 write 0x0384c7 thru 0x03ffff (0x7b39 bytes) 000962 1787605224.454348 read 0x002eb3 thru 0x0031e3 (0x331 bytes) 000966 1787605224.469234 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xef9cff74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38b6fa6b Mirror 3: 0x38b6fa6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000967 1787605225.413344 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xef9cff74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38b6fa6b Mirror 3: 0x38b6fa6b Mirror 4: 0x38b6fa6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1787605226.416143 mapread 0x038358 thru 0x03ffff (0x7ca8 bytes) 000969 1787605226.454599 trunc from 0x040000 to 0x035094 (0xaf6b bytes) 000971 1787605226.548191 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: 0xc5221b6a (stale) Mirror 3: 0x41e1155b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: 0xc5221b6a (stale) Mirror 3: 0x41e1155b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35095: skipped 000974 1787605226.889421 mapwrite 0x00cbf9 thru 0x00fb49 (0x2f51 bytes) 000980 1787605226.969332 write 0x015674 thru 0x01d176 (0x7b03 bytes) 000982 1787605227.000607 MIRROR_RESYNC 3 mirrors 000983 1787605228.292754 read 0x01bbac thru 0x026a45 (0xae9a bytes) 000984 1787605228.305353 write 0x00f265 thru 0x018a9d (0x9839 bytes) 000985 1787605228.407830 mapwrite 0x0109e8 thru 0x01364b (0x2c64 bytes) 000986 1787605228.466882 mapread 0x005228 thru 0x00de26 (0x8bff bytes) 000987 1787605228.495219 mapread 0x011ebd thru 0x01b49a (0x95de bytes) 000988 1787605228.509815 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: 0xb1e90bd0 (stale) Mirror 3: 0xb1e90bd0 (stale) CRC-32 checksum value for chunk [0, 0x35095): Mirror 1: 0x939ae91b Mirror 4: 0x939ae91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35095: skipped 000989 1787605229.545078 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: 0xb1e90bd0 (stale) Mirror 3: 0xb1e90bd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35095): Mirror 2: 0xb1e90bd0 (stale) Mirror 3: 0xb1e90bd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35095: skipped 000990 1787605229.949685 read 0x014071 thru 0x01f316 (0xb2a6 bytes) 000991 1787605229.963840 mapwrite 0x016563 thru 0x021b3a (0xb5d8 bytes) 000992 1787605230.097592 mapwrite 0x02f59c thru 0x031bcb (0x2630 bytes) 000993 1787605230.156382 trunc from 0x035095 to 0x00e501 (0x26b93 bytes) truncating to largest ever: 0x3fb82 000995 1787605230.216379 trunc from 0x00e502 to 0x03fb81 (0x31680 bytes) 000996 1787605230.235912 trunc from 0x03fb82 to 0x01cfcf (0x22bb2 bytes) 000997 1787605230.251376 mapread 0x00af01 thru 0x0193e3 (0xe4e3 bytes) 000998 1787605230.278974 trunc from 0x01cfd0 to 0x01f0e8 (0x2119 bytes) 000999 1787605230.308098 MIRROR_RESYNC 3 mirrors 001000 1787605231.401712 write 0x03f254 thru 0x03ffff (0xdac bytes) All operations completed A-OK! PASS 70a (81s) == sanity-flr test 71: check mirror extend parent fid ==== 17:00:37 (1787605237) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 17:00:41 (1787605241) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x244:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259662 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x262:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x244:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229041 s, 4.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x262:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x245:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x244:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x262:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x245:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x244:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x263:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 17:00:51 (1787605251) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4759488 .. Extending file size to 7703520 .. failed Extending file size to 8341824 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8562208 .. Extending file size to 8801024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7880928 .. Extending file size to 8649408 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9377504 .. Extending file size to 8757216 .. Extending file size to 8783104 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59334 59336 59337 59339 59341 failed 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (95s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:02:27 (1787605347) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1207776 .. Extending file size to 7912704 .. done Extending file size to 1493760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9051744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5218848 .. Extending file size to 5767872 .. Extending file size to 9096064 .. Extending file size to 9232384 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60821 60822 60825 60826 60827 done 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:03:54 (1787605434) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2304 2206336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2304 2206336 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194912 s, 10.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.504009 s, 4.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3a3df391 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:04:18 (1787605458) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12483 s, 16.8 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 17:04:27 (1787605467) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x171:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x278:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:04:35 (1787605475) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207012 s, 10.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x40:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x43:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:04:44 (1787605484) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140355 s, 14.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x279:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x279:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27a:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:04:52 (1787605492) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164859 s, 12.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x25f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27c:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:05:28 (1787605528) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100188 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x41:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x45:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x41:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x45:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x42:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:06:34 (1787605594) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33684 s, 6.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x260:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x260:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (69s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:07:43 (1787605663) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg214-server: Pool lustre.test_204f created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.391189 s, 5.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x261:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x27f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x261:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg214-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:09:12 (1787605752) PASS 205a (16s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:09:28 (1787605768) PASS 205b (12s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:09:40 (1787605780) oleg214-server: Pool lustre.test_206 created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg214-server: Pool lustre.test_206 destroyed PASS 206 (34s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:10:14 (1787605814) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109459 s, 9.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373611 s, 28.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe64debd7 Mirror 3: 0xe64debd7 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:10:27 (1787605827) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:10:31 (1787605831) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:10:36 (1787605836) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.165173 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1930000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2010000 Read this (4 bytes): BBBB 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:11:17 (1787605877) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0318764 s, 257 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:11:25 (1787605885) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557526 s, 18.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x192 lmm_fid: [0x200000405:0x192:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x192 lmm_fid: [0x200000405:0x192:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x26b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611538 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 17:11:27.000000000 -0400 Modify: 2026-08-24 17:11:27.000000000 -0400 Change: 2026-08-24 17:11:28.000000000 -0400 Birth: 2026-08-24 17:11:27.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:11:33 (1787605893) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:11:36 (1787605896) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.604273 s, 67.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0480242 s, 85.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 17:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x194:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 17:11 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:11:45 (1787605905) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:11:54 (1787605914) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1961 sec =========== 17:12:05 (1787605925) === sanity-flr: start cleanup 17:12:07 (1787605927) === === sanity-flr: finish cleanup 17:12:17 (1787605937) ===