-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 06:37:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg140-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:37:57 (1788863877) === oleg140-client.virtnet: executing check_config_client /mnt/lustre oleg140-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg140-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ec684eb8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ec684eb8000.idle_timeout=debug disable quota as required oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:38:35 (1788863915) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7603] == sanity-flr test 0a: lfs mirror create with -N option == 06:38:37 (1788863917) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:38:52 (1788863932) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:38:56 (1788863936) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:38:59 (1788863939) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:39:19 (1788863959) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:39:22 (1788863962) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:39:25 (1788863965) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ec684eb8000.stats=clear osc.lustre-OST0001-osc-ffff8ec684eb8000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:39:33 (1788863973) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:39:43 (1788863983) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:39:50 (1788863990) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:39:53 (1788863993) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:40:00 (1788864000) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 06:40:08 (1788864008) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:40:15 (1788864015) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:40:18 (1788864018) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:40:24 (1788864024) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:40:33 (1788864033) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:40:40 (1788864040) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.65931 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144818 s, 7.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:40:50 (1788864050) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.065197 s, 32.2 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:40:56 (1788864056) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:41:06 (1788864066) Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:42:11 (1788864131) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.482663 s, 21.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:42:56 (1788864176) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:44:14 (1788864254) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.65713 s, 11.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 06:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.029s user 0m0.000s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.475s user 0m0.000s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.108s user 0m0.000s sys 0m0.171s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg140-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ec684eb8000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:45:57 (1788864357) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:46:00 (1788864360) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:46:03 (1788864363) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:46:06 (1788864366) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:46:13 (1788864373) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.557722 s, 22.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:46:24 (1788864384) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330711 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292337 s, 35.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:46:31 (1788864391) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:46:38 (1788864398) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000797319 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000722176 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129336 s, 792 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000834991 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:46:48 (1788864408) create test files with size 15732373 .. 46a9908248b62e414906554892be85ed - fa6ce48c5dc52b7b6e0dd93add2b9a33 - 8c7bd27f59d7a167b412a6cbecad0d2b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:47:10 (1788864430) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:47:25 (1788864445) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:47:32 (1788864452) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517173 s, 20.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149022 s, 70.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 06:47:40 (1788864460) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990644 s, 21.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.447959 s, 9.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15359 s, 13.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 06:47:56 (1788864476) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:48:00 (1788864480) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:48:03 (1788864483) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317483 s, 33.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 06:48:11 (1788864491) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:48:15 (1788864495) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:48:26 (1788864506) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.223194 s, 47.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:48:39 (1788864519) llite.lustre-ffff8ec6912c3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.313286 s, 33.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ec6a042f800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:48:53 (1788864533) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:49:38 (1788864578) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:50:22 (1788864622) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:50:25 (1788864625) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.328365 s, 9.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138507 s, 22.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5298022a (nosync,stale) Mirror 2: 0x5298022a (nosync,stale) ** verify mirror 2 doesn't change original checksum: d3e2ca6c91c90740b76d4df2bf23a2eb - mirror 1 checksum: 2963bfa9b70cdafcc8eaf1ab6579289c - mirror 2 checksum: d3e2ca6c91c90740b76d4df2bf23a2eb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T06:50:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T06:50:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d3e2ca6c91c90740b76d4df2bf23a2eb - mirror 1 checksum: 2963bfa9b70cdafcc8eaf1ab6579289c - mirror 2 checksum: 2963bfa9b70cdafcc8eaf1ab6579289c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T06:50:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T06:50:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:50:38 (1788864638) Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 06:50:45 (1788864645) shut down facet: mds1 facet_host: oleg140-server facet_failover_host: oleg140-server Failover mds1 to oleg140-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 06:51:07 (1788864667) targets are mounted 06:51:07 (1788864667) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015917 s, 65.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:51:23 (1788864683) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:51:26 (1788864686) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:51:29 (1788864689) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00468442 s, 3.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:51:37 (1788864697) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:51:39 (1788864699) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00391907 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0623197 s, 65.7 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:51:47 (1788864707) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:52:10 (1788864730) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:52:39 (1788864759) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:53:03 (1788864783) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:53:26 (1788864806) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:53:34 (1788864814) mirror create pid 54295 mirror split pid 54297 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54295 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54297 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 06:54:40 (1788864880) Seed set to 4882 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1788864882.320027 write 0x034389 thru 0x035994 (0x160c bytes) 000006 1788864882.326740 read 0x012f19 thru 0x013c67 (0xd4f bytes) truncating to largest ever: 0x15748 000009 1788864882.332271 trunc from 0x035995 to 0x015747 (0x2024d bytes) 000011 1788864882.346744 read 0x0011c7 thru 0x00d029 (0xbe63 bytes) 000013 1788864882.356335 mapread 0x00763c thru 0x00ba81 (0x4446 bytes) 000018 1788864882.363034 write 0x00981f thru 0x010846 (0x7028 bytes) 000021 1788864882.366496 mapwrite 0x032062 thru 0x03c0e3 (0xa082 bytes) 000024 1788864882.449528 trunc from 0x03c0e4 to 0x005cc5 (0x3641e bytes) 000026 1788864882.468017 trunc from 0x005cc6 to 0x011114 (0xb44f bytes) 000027 1788864882.482791 punch from 0x0027d1 to 0x00d75b (0xaf8b bytes) 000029 1788864882.490120 mapread 0x00a956 thru 0x011114 (0x67bf bytes) 000030 1788864882.521491 trunc from 0x011115 to 0x00e8a6 (0x286e bytes) 000033 1788864882.541767 MIRROR_EXTEND to 2 mirrors 000034 1788864883.072144 mapwrite 0x012935 thru 0x016e8e (0x455a bytes) truncating to largest ever: 0x38f04 000035 1788864883.226802 trunc from 0x016e8f to 0x038f03 (0x22075 bytes) 000037 1788864883.249153 MIRROR_RESYNC 2 mirrors 000038 1788864883.823884 read 0x01bbfe thru 0x021926 (0x5d29 bytes) 000039 1788864883.832303 write 0x037ddc thru 0x038721 (0x946 bytes) 000040 1788864883.976340 mapread 0x017264 thru 0x02317a (0xbf17 bytes) 000041 1788864884.011454 mapwrite 0x0089c0 thru 0x014805 (0xbe46 bytes) 000043 1788864884.071974 write 0x008f98 thru 0x015307 (0xc370 bytes) 000044 1788864884.073898 write 0x007577 thru 0x00a623 (0x30ad bytes) 000045 1788864884.085239 MIRROR_RESYNC 2 mirrors 000046 1788864884.602064 trunc from 0x038f04 to 0x006563 (0x329a0 bytes) 000047 1788864884.688269 mapread 0x004402 thru 0x006563 (0x2162 bytes) 000048 1788864884.706626 mapwrite 0x033e9b thru 0x03d2c2 (0x9428 bytes) 000052 1788864884.777746 mapwrite 0x03ee85 thru 0x03ffff (0x117b bytes) 000054 1788864884.878880 trunc from 0x040000 to 0x00a558 (0x35aa7 bytes) 000055 1788864884.910484 trunc from 0x00a559 to 0x021ff0 (0x17a98 bytes) 000057 1788864884.946845 MIRROR_RESYNC 2 mirrors 000061 1788864885.623058 trunc from 0x021ff1 to 0x018591 (0x9a5f bytes) 000062 1788864885.688383 trunc from 0x018592 to 0x0293e0 (0x10e4f bytes) 000063 1788864885.705039 mapread 0x020689 thru 0x0293e0 (0x8d58 bytes) 000064 1788864885.720019 punch from 0x023576 to 0x0293e0 (0x5e6b bytes) 000067 1788864885.725050 read 0x025620 thru 0x0287f6 (0x31d7 bytes) 000069 1788864885.729987 mapwrite 0x018a6d thru 0x01c83c (0x3dd0 bytes) 000070 1788864885.783424 MIRROR_RESYNC 2 mirrors 000071 1788864886.381408 trunc from 0x0293e1 to 0x033aec (0xa70c bytes) 000074 1788864886.448414 read 0x01fce1 thru 0x02d583 (0xd8a3 bytes) 000075 1788864886.455439 read 0x01aa89 thru 0x023070 (0x85e8 bytes) 000076 1788864886.467614 write 0x0143c8 thru 0x01e9bb (0xa5f4 bytes) 000079 1788864886.469786 trunc from 0x033aed to 0x01b0bf (0x18a2d bytes) 000080 1788864886.494849 trunc from 0x01b0c0 to 0x01eaa2 (0x39e3 bytes) 000082 1788864886.524727 trunc from 0x01eaa3 to 0x0033bd (0x1b6e5 bytes) 000084 1788864886.540339 mapread 0x0001ba thru 0x0033bd (0x3204 bytes) 000085 1788864886.541315 punch from 0x0018e9 to 0x0033bd (0x1ad5 bytes) 000087 1788864886.549074 MIRROR_RESYNC 2 mirrors 000088 1788864886.979066 punch from 0x001b71 to 0x0033bd (0x184d bytes) 000089 1788864887.045427 punch from 0x00139d to 0x0033bd (0x2021 bytes) 000090 1788864887.053223 trunc from 0x0033be to 0x024bb3 (0x217f6 bytes) 000091 1788864887.092996 mapread 0x02286d thru 0x024bb3 (0x2347 bytes) 000092 1788864887.110420 read 0x00bbdb thru 0x01441f (0x8845 bytes) 000093 1788864887.119092 read 0x017377 thru 0x022d92 (0xba1c bytes) 000094 1788864887.127417 mapwrite 0x00a237 thru 0x0174ee (0xd2b8 bytes) 000095 1788864887.174438 read 0x0083cf thru 0x00d526 (0x5158 bytes) 000097 1788864887.175576 read 0x011478 thru 0x015916 (0x449f bytes) 000098 1788864887.176102 trunc from 0x024bb4 to 0x02f059 (0xa4a6 bytes) 000099 1788864887.192906 mapread 0x013828 thru 0x016eb4 (0x368d bytes) 000101 1788864887.194409 trunc from 0x02f05a to 0x01abc4 (0x14495 bytes) truncating to largest ever: 0x3d7d1 000102 1788864887.210932 trunc from 0x01abc5 to 0x03d7d0 (0x22c0c bytes) 000103 1788864887.230930 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000104 1788864887.380990 write 0x01c7a9 thru 0x026130 (0x9988 bytes) 000105 1788864887.392117 read 0x03a840 thru 0x03d7d0 (0x2f91 bytes) 000106 1788864887.398321 mapread 0x02ed2e thru 0x0383b5 (0x9688 bytes) 000107 1788864887.416335 read 0x01cc8c thru 0x0264a4 (0x9819 bytes) 000109 1788864887.418531 read 0x0269db thru 0x034ae2 (0xe108 bytes) 000111 1788864887.429395 punch from 0x036423 to 0x03979c (0x337a bytes) 000112 1788864887.439954 write 0x01959e thru 0x01bf8a (0x29ed bytes) 000113 1788864887.442035 mapwrite 0x017802 thru 0x01f91a (0x8119 bytes) 000115 1788864887.500662 read 0x02fa76 thru 0x03ac57 (0xb1e2 bytes) 000116 1788864887.508074 read 0x0209ac thru 0x0263fe (0x5a53 bytes) 000118 1788864887.510159 mapread 0x03478f thru 0x03d7d0 (0x9042 bytes) 000119 1788864887.511637 punch from 0x025450 to 0x02a022 (0x4bd3 bytes) 000121 1788864887.516884 write 0x036bc6 thru 0x03ffff (0x943a bytes) 000123 1788864887.519722 write 0x00f448 thru 0x01441f (0x4fd8 bytes) 000124 1788864887.521435 write 0x02ebf1 thru 0x03bc3a (0xd04a bytes) 000126 1788864887.523866 read 0x032cf6 thru 0x035370 (0x267b bytes) 000127 1788864887.524943 read 0x02aa36 thru 0x03107c (0x6647 bytes) 000129 1788864887.536082 write 0x02a0a3 thru 0x036b53 (0xcab1 bytes) 000131 1788864887.538718 mapwrite 0x03f11b thru 0x03ffff (0xee5 bytes) 000132 1788864887.594687 mapread 0x00b00f thru 0x018813 (0xd805 bytes) 000135 1788864887.596426 punch from 0x01edaa to 0x024d3b (0x5f92 bytes) 000137 1788864887.608684 MIRROR_EXTEND to 2 mirrors 000139 1788864888.335186 mapwrite 0x029739 thru 0x02f0e0 (0x59a8 bytes) 000141 1788864888.475281 trunc from 0x040000 to 0x007d5e (0x382a1 bytes) 000142 1788864888.497641 punch from 0x00441d to 0x007d5e (0x3942 bytes) 000147 1788864888.508642 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7d5f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7d5f): Mirror 2: 0xd8d07219 (stale) CRC-32 checksum value for chunk [0, 0x7d5f): Mirror 1: 0xd8d07219 Mirror 3: 0xd8d07219 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7d5f: skipped 000148 1788864889.154141 MIRROR_RESYNC 3 mirrors 000149 1788864889.687062 punch from 0x001981 to 0x0078e4 (0x5f64 bytes) 000150 1788864889.795796 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7d5f): Mirror 2: 0xd8d07219 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7d5f): Mirror 2: 0xd8d07219 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7d5f: skipped 000154 1788864890.048229 mapwrite 0x01096b thru 0x01e21a (0xd8b0 bytes) 000155 1788864890.136937 read 0x004906 thru 0x0105ea (0xbce5 bytes) 000156 1788864890.150611 mapread 0x002722 thru 0x00c16d (0x9a4c bytes) 000158 1788864890.162951 read 0x019c02 thru 0x01e21a (0x4619 bytes) 000159 1788864890.164480 read 0x00e4c5 thru 0x01bfcf (0xdb0b bytes) 000160 1788864890.166710 mapread 0x005dfb thru 0x00917b (0x3381 bytes) 000162 1788864890.170355 trunc from 0x01e21b to 0x0291e1 (0xafc7 bytes) 000166 1788864890.191576 read 0x00c9ad thru 0x0116fe (0x4d52 bytes) 000168 1788864890.192642 write 0x0275d7 thru 0x02e13d (0x6b67 bytes) 000169 1788864890.227193 mapread 0x00990c thru 0x015e4c (0xc541 bytes) 000171 1788864890.230385 mapwrite 0x000b63 thru 0x00a488 (0x9926 bytes) 000174 1788864890.408041 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e13e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e13e): Mirror 2: 0x6af2b97e (stale) CRC-32 checksum value for chunk [0, 0x2e13e): Mirror 1: 0x95af6316 Mirror 3: 0x95af6316 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e13e: skipped 000175 1788864891.221337 trunc from 0x02e13e to 0x034e64 (0x6d27 bytes) 000176 1788864891.316352 mapwrite 0x00e298 thru 0x0125a2 (0x430b bytes) 000179 1788864891.380031 mapread 0x02a93e thru 0x02d158 (0x281b bytes) 000182 1788864891.390672 trunc from 0x034e65 to 0x0013e3 (0x33a81 bytes) 000183 1788864891.417567 mapwrite 0x037453 thru 0x0392d3 (0x1e81 bytes) 000184 1788864891.487525 punch from 0x03086a to 0x0392d3 (0x8a6a bytes) 000185 1788864891.492515 punch from 0x01dbc7 to 0x02a43e (0xc878 bytes) 000186 1788864891.496321 mapread 0x012dcb thru 0x01ea81 (0xbcb7 bytes) 000187 1788864891.533208 mapread 0x013be9 thru 0x01ec3c (0xb054 bytes) 000190 1788864891.536049 mapread 0x01af2b thru 0x01ce4a (0x1f20 bytes) 000191 1788864891.536982 write 0x02a337 thru 0x02e204 (0x3ece bytes) 000192 1788864891.545784 punch from 0x0192f7 to 0x01d770 (0x447a bytes) 000193 1788864891.556202 trunc from 0x0392d4 to 0x02b919 (0xd9ba bytes) 000194 1788864891.569893 trunc from 0x02b91a to 0x00d1f6 (0x1e723 bytes) 000195 1788864891.587156 read 0x0018f4 thru 0x00d1f6 (0xb903 bytes) 000196 1788864891.601049 read 0x009a7a thru 0x009f69 (0x4f0 bytes) 000198 1788864891.602461 read 0x006822 thru 0x00d1f6 (0x69d5 bytes) 000200 1788864891.604179 write 0x027902 thru 0x0324bb (0xabba bytes) 000203 1788864891.608522 read 0x02cf81 thru 0x0324bb (0x553b bytes) 000204 1788864891.609226 read 0x02f346 thru 0x0324bb (0x3176 bytes) 000205 1788864891.610208 mapwrite 0x023a23 thru 0x02f4d2 (0xbab0 bytes) 000206 1788864891.660965 write 0x01a48f thru 0x028f2d (0xea9f bytes) 000207 1788864891.662238 punch from 0x02e93a to 0x0324bb (0x3b82 bytes) 000210 1788864891.674266 write 0x02a83b thru 0x02d54c (0x2d12 bytes) 000211 1788864891.675942 trunc from 0x0324bc to 0x01ffc5 (0x124f6 bytes) 000212 1788864891.689709 write 0x0190c0 thru 0x0272d0 (0xe211 bytes) 000214 1788864891.693039 mapread 0x02382b thru 0x0272d0 (0x3aa6 bytes) 000215 1788864891.694187 read 0x000ff6 thru 0x003bbf (0x2bca bytes) 000216 1788864891.698853 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x272d1): Mirror 2: 0x8ec5cd9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x272d1): Mirror 2: 0x8ec5cd9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x272d1: skipped 000217 1788864891.969384 write 0x02bdb6 thru 0x03a06e (0xe2b9 bytes) 000219 1788864891.987156 mapwrite 0x006e20 thru 0x00f372 (0x8553 bytes) 000220 1788864892.138345 punch from 0x038cc5 to 0x03a06e (0x13aa bytes) 000221 1788864892.153925 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a06f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a06f): Mirror 2: 0x5fa423fb (stale) CRC-32 checksum value for chunk [0, 0x3a06f): Mirror 1: 0xe987ceba Mirror 3: 0xe987ceba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a06f: skipped 000223 1788864893.058570 MIRROR_RESYNC 3 mirrors 000224 1788864893.734303 write 0x017d34 thru 0x01f06f (0x733c bytes) 000226 1788864893.801085 write 0x004101 thru 0x012cff (0xebff bytes) 000227 1788864893.811035 mapwrite 0x016e97 thru 0x023891 (0xc9fb bytes) 000228 1788864893.868842 write 0x018b89 thru 0x020ed2 (0x834a bytes) 000231 1788864893.871890 mapread 0x025579 thru 0x02bbd2 (0x665a bytes) 000232 1788864893.884113 read 0x021c0f thru 0x027503 (0x58f5 bytes) 000233 1788864893.884871 read 0x02f9bf thru 0x03a06e (0xa6b0 bytes) 000238 1788864893.897894 trunc from 0x03a06f to 0x038f23 (0x114b bytes) 000239 1788864893.921634 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f24): Mirror 2: 0xeed9b5d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f24): Mirror 2: 0xeed9b5d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f24: skipped 000242 1788864894.220137 write 0x035ce9 thru 0x03ffff (0xa317 bytes) 000244 1788864894.225656 read 0x021a5a thru 0x02de89 (0xc430 bytes) 000245 1788864894.232842 read 0x005ee3 thru 0x00ae58 (0x4f76 bytes) 000246 1788864894.254138 trunc from 0x040000 to 0x00d1a5 (0x32e5a bytes) 000248 1788864894.269038 write 0x01cbeb thru 0x02042c (0x3842 bytes) 000250 1788864894.271214 mapread 0x01570e thru 0x018594 (0x2e87 bytes) 000251 1788864894.288270 write 0x02401d thru 0x025fd0 (0x1fb4 bytes) 000252 1788864894.290770 punch from 0x02332a to 0x025fd0 (0x2ca7 bytes) 000253 1788864894.303788 read 0x00f8cf thru 0x016901 (0x7033 bytes) 000254 1788864894.312329 mapwrite 0x00e66a thru 0x014dc6 (0x675d bytes) 000255 1788864894.387301 mapread 0x01ef8e thru 0x022821 (0x3894 bytes) 000256 1788864894.388952 read 0x00f31b thru 0x018bec (0x98d2 bytes) 000260 1788864894.394648 punch from 0x0107d9 to 0x01d02a (0xc852 bytes) 000261 1788864894.399876 trunc from 0x025fd1 to 0x00334d (0x22c83 bytes) 000263 1788864894.425003 punch from 0x0006c7 to 0x00334d (0x2c87 bytes) 000264 1788864894.430248 trunc from 0x00334e to 0x001361 (0x1fec bytes) 000265 1788864894.464737 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1362): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1362): Mirror 2: 0x67c7de30 (stale) CRC-32 checksum value for chunk [0, 0x1362): Mirror 1: 0x42791657 Mirror 3: 0x42791657 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1362: skipped 000266 1788864895.258864 read 0x0000d4 thru 0x000525 (0x452 bytes) 000267 1788864895.283791 mapread 0x000906 thru 0x001361 (0xa5c bytes) 000268 1788864895.312068 mapread 0x000ce7 thru 0x001361 (0x67b bytes) 000269 1788864895.337969 MIRROR_RESYNC 3 mirrors 000270 1788864896.253914 mapread 0x0008e7 thru 0x001361 (0xa7b bytes) 000272 1788864896.295840 MIRROR_RESYNC 3 mirrors 000273 1788864896.655698 mapread 0x000b3f thru 0x001361 (0x823 bytes) 000275 1788864896.657421 mapread 0x000b22 thru 0x001361 (0x840 bytes) 000276 1788864896.658388 punch from 0x000fea to 0x001361 (0x378 bytes) 000277 1788864896.779907 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1362): Mirror 3: 0x42791657 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1362): Mirror 3: 0x42791657 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1362: skipped 000278 1788864897.115078 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000279 1788864897.285700 write 0x037c60 thru 0x03eaa6 (0x6e47 bytes) 000280 1788864897.286999 punch from 0x014ddb to 0x017596 (0x27bc bytes) 000281 1788864897.315216 read 0x021d27 thru 0x027740 (0x5a1a bytes) 000282 1788864897.325390 punch from 0x024ad9 to 0x030619 (0xbb41 bytes) 000283 1788864897.333319 trunc from 0x03eaa7 to 0x00e368 (0x3073e bytes) 000284 1788864897.396487 trunc from 0x00e369 to 0x035887 (0x2751f bytes) 000285 1788864897.414999 mapread 0x02f716 thru 0x0300cd (0x9b8 bytes) 000287 1788864897.430234 punch from 0x026019 to 0x0315f6 (0xb5de bytes) 000288 1788864897.438365 trunc from 0x035888 to 0x000a75 (0x34e12 bytes) 000289 1788864897.455337 trunc from 0x000a76 to 0x02f7d3 (0x2ed5e bytes) 000292 1788864897.470852 trunc from 0x02f7d4 to 0x02e620 (0x11b3 bytes) 000297 1788864897.490125 read 0x02578c thru 0x02e620 (0x8e95 bytes) 000300 1788864897.514227 trunc from 0x02e621 to 0x00f794 (0x1ee8c bytes) truncating to largest ever: 0x3fade 000301 1788864897.532869 trunc from 0x00f795 to 0x03fadd (0x30349 bytes) 000302 1788864897.546054 read 0x00db84 thru 0x01bbd3 (0xe050 bytes) 000304 1788864897.561938 read 0x031841 thru 0x03b39e (0x9b5e bytes) 000306 1788864897.563778 read 0x00dd3c thru 0x017df4 (0xa0b9 bytes) 000307 1788864897.564932 punch from 0x012947 to 0x01494c (0x2006 bytes) 000308 1788864897.569452 punch from 0x00d62e to 0x018878 (0xb24b bytes) 000311 1788864897.575401 read 0x030383 thru 0x03c5c4 (0xc242 bytes) 000312 1788864897.576175 trunc from 0x03fade to 0x03efda (0xb03 bytes) 000314 1788864897.594104 punch from 0x01ee59 to 0x0244fc (0x56a4 bytes) 000315 1788864897.598240 mapread 0x034a58 thru 0x03efda (0xa583 bytes) 000317 1788864897.602856 read 0x03c5f6 thru 0x03efda (0x29e5 bytes) 000318 1788864897.603512 read 0x02009d thru 0x02e14e (0xe0b2 bytes) 000319 1788864897.611857 mapwrite 0x0231c8 thru 0x031f1f (0xed58 bytes) 000322 1788864897.658447 mapread 0x03dc0c thru 0x03efda (0x13cf bytes) 000323 1788864897.660815 write 0x027ba9 thru 0x0299fd (0x1e55 bytes) 000324 1788864897.662233 mapwrite 0x032198 thru 0x035479 (0x32e2 bytes) 000325 1788864897.701039 write 0x02bb66 thru 0x030496 (0x4931 bytes) 000326 1788864897.702498 mapread 0x0068e8 thru 0x008f70 (0x2689 bytes) 000327 1788864897.703944 trunc from 0x03efdb to 0x03393b (0xb69f bytes) 000330 1788864897.725824 write 0x010d78 thru 0x018a98 (0x7d21 bytes) 000331 1788864897.727857 trunc from 0x03393c to 0x023846 (0x100f5 bytes) 000333 1788864897.743457 read 0x000baa thru 0x00f9d1 (0xee28 bytes) 000335 1788864897.744919 trunc from 0x023847 to 0x0376c7 (0x13e81 bytes) 000338 1788864897.768361 read 0x01986b thru 0x028941 (0xf0d7 bytes) 000340 1788864897.776713 punch from 0x029160 to 0x030a13 (0x78b4 bytes) 000342 1788864897.789943 read 0x02a56f thru 0x02d370 (0x2e02 bytes) 000345 1788864897.800824 mapwrite 0x029fcd thru 0x02a7ce (0x802 bytes) 000346 1788864897.833805 write 0x00e043 thru 0x013bdb (0x5b99 bytes) 000348 1788864897.836149 mapread 0x012fa5 thru 0x016b2c (0x3b88 bytes) 000349 1788864897.837246 read 0x01c45d thru 0x020f28 (0x4acc bytes) 000350 1788864897.838263 mapread 0x005860 thru 0x00682d (0xfce bytes) 000354 1788864897.844462 mapread 0x006118 thru 0x00dac7 (0x79b0 bytes) 000355 1788864897.845468 punch from 0x0161b5 to 0x01a8ac (0x46f8 bytes) 000357 1788864897.860048 trunc from 0x0376c8 to 0x01cf31 (0x1a796 bytes) 000358 1788864897.879585 MIRROR_EXTEND to 2 mirrors 000359 1788864898.436819 mapwrite 0x0215e5 thru 0x025363 (0x3d7f bytes) 000360 1788864898.588425 punch from 0x0033b2 to 0x005e9f (0x2aee bytes) 000361 1788864898.594428 punch from 0x00cce8 to 0x0104ab (0x37c4 bytes) 000362 1788864898.598999 trunc from 0x025364 to 0x0290c0 (0x3d5d bytes) 000363 1788864898.640003 mapwrite 0x017f4b thru 0x020dbc (0x8e72 bytes) 000364 1788864898.696134 trunc from 0x0290c1 to 0x005fed (0x230d3 bytes) 000365 1788864898.725698 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000366 1788864898.851856 write 0x03376c thru 0x03769f (0x3f34 bytes) 000367 1788864898.853469 write 0x022a0c thru 0x024cf4 (0x22e9 bytes) 000368 1788864898.862941 trunc from 0x0376a0 to 0x01965d (0x1e042 bytes) 000369 1788864898.877098 read 0x0012f0 thru 0x009f2e (0x8c3f bytes) 000370 1788864898.886801 read 0x00f358 thru 0x0123fb (0x30a4 bytes) 000371 1788864898.893639 read 0x00ec91 thru 0x0163d6 (0x7746 bytes) 000372 1788864898.894674 mapread 0x001df8 thru 0x009259 (0x7462 bytes) 000374 1788864898.896953 mapwrite 0x01011f thru 0x015ddc (0x5cbe bytes) 000375 1788864898.936758 punch from 0x00df5e to 0x01965d (0xb700 bytes) 000376 1788864898.941024 read 0x00c414 thru 0x016c30 (0xa81d bytes) 000387 1788864898.958132 mapread 0x017a26 thru 0x01965d (0x1c38 bytes) 000389 1788864898.959500 mapwrite 0x03dee9 thru 0x03ffff (0x2117 bytes) 000392 1788864899.021998 mapwrite 0x017677 thru 0x026bd6 (0xf560 bytes) 000393 1788864899.057655 trunc from 0x040000 to 0x0343c4 (0xbc3b bytes) 000396 1788864899.076454 punch from 0x0018a8 to 0x00582a (0x3f83 bytes) 000397 1788864899.081069 punch from 0x0095e3 to 0x00df7c (0x499a bytes) 000398 1788864899.085455 punch from 0x022810 to 0x0300b8 (0xd8a9 bytes) 000402 1788864899.093831 write 0x035c36 thru 0x03d1c9 (0x7594 bytes) 000403 1788864899.095739 mapread 0x0320ff thru 0x03d1c9 (0xb0cb bytes) 000404 1788864899.096928 punch from 0x01dcf6 to 0x0220c2 (0x43cd bytes) 000409 1788864899.110162 mapread 0x014920 thru 0x015abd (0x119e bytes) 000411 1788864899.111891 punch from 0x019232 to 0x01f613 (0x63e2 bytes) 000412 1788864899.116101 read 0x0196ff thru 0x0232bd (0x9bbf bytes) 000416 1788864899.128864 trunc from 0x03d1ca to 0x026d2a (0x1649f bytes) 000418 1788864899.144037 mapwrite 0x007bcb thru 0x014305 (0xc73b bytes) 000421 1788864899.185934 MIRROR_EXTEND to 2 mirrors 000422 1788864899.649519 punch from 0x00960d to 0x014a64 (0xb458 bytes) 000423 1788864899.717754 mapread 0x0131be thru 0x01d930 (0xa773 bytes) 000424 1788864899.738387 write 0x02cb93 thru 0x03b9b1 (0xee1f bytes) 000425 1788864899.739968 mapread 0x0004a7 thru 0x00240d (0x1f67 bytes) 000427 1788864899.748631 mapwrite 0x008d4b thru 0x00f1b7 (0x646d bytes) 000428 1788864899.794070 read 0x020545 thru 0x021939 (0x13f5 bytes) 000430 1788864899.799222 trunc from 0x03b9b2 to 0x03af4f (0xa62 bytes) 000432 1788864899.839668 mapwrite 0x02d781 thru 0x0316c1 (0x3f41 bytes) 000433 1788864899.878884 mapwrite 0x03031c thru 0x03311d (0x2e02 bytes) 000435 1788864899.909936 mapread 0x01e54c thru 0x02e3a2 (0xfe57 bytes) 000436 1788864899.923422 punch from 0x0288a8 to 0x031bfe (0x9357 bytes) 000437 1788864899.926558 trunc from 0x03af50 to 0x0228ac (0x186a3 bytes) 000438 1788864899.941380 read 0x01e45b thru 0x01e91b (0x4c1 bytes) 000439 1788864899.942033 read 0x0152ca thru 0x0228ac (0xd5e3 bytes) 000440 1788864899.943097 mapread 0x009f2e thru 0x011fbc (0x808f bytes) 000441 1788864899.944037 mapwrite 0x036fe2 thru 0x03c3d0 (0x53ef bytes) 000442 1788864899.998104 trunc from 0x03c3d1 to 0x01dda2 (0x1e62e bytes) 000444 1788864900.020170 trunc from 0x01dda3 to 0x00aaa1 (0x13301 bytes) 000445 1788864900.044013 mapwrite 0x01b196 thru 0x02239f (0x720a bytes) 000446 1788864900.149038 punch from 0x00f39c to 0x01e776 (0xf3db bytes) 000447 1788864900.155115 write 0x009915 thru 0x0183be (0xeaaa bytes) 000449 1788864900.160983 read 0x014d38 thru 0x020d54 (0xc01d bytes) 000450 1788864900.175741 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000451 1788864900.314001 mapread 0x0150b9 thru 0x019242 (0x418a bytes) 000454 1788864900.338443 mapwrite 0x02a76e thru 0x0372d9 (0xcb6c bytes) 000455 1788864900.424768 mapread 0x016300 thru 0x01714e (0xe4f bytes) 000458 1788864900.433322 MIRROR_EXTEND to 2 mirrors 000459 1788864900.949865 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000462 1788864901.155364 mapread 0x030b54 thru 0x0372d9 (0x6786 bytes) 000463 1788864901.177444 mapwrite 0x03f27d thru 0x03ffff (0xd83 bytes) 000465 1788864901.280951 punch from 0x01c318 to 0x01f70a (0x33f3 bytes) 000468 1788864901.286944 mapread 0x00662c thru 0x011307 (0xacdc bytes) 000470 1788864901.308055 write 0x016666 thru 0x021284 (0xac1f bytes) 000471 1788864901.319003 write 0x03a475 thru 0x03ffff (0x5b8b bytes) 000472 1788864901.321392 write 0x005248 thru 0x0125a4 (0xd35d bytes) 000474 1788864901.324485 mapread 0x01f016 thru 0x02b88e (0xc879 bytes) 000475 1788864901.326140 trunc from 0x040000 to 0x00b836 (0x347c9 bytes) 000477 1788864901.350115 write 0x02b2cd thru 0x031d01 (0x6a35 bytes) 000479 1788864901.354933 mapwrite 0x020f49 thru 0x027331 (0x63e9 bytes) 000481 1788864901.418853 read 0x021f14 thru 0x0264f0 (0x45dd bytes) 000482 1788864901.419888 read 0x0310ae thru 0x031d01 (0xc54 bytes) 000484 1788864901.422984 mapread 0x01b501 thru 0x021792 (0x6292 bytes) 000486 1788864901.424786 mapwrite 0x0368e8 thru 0x03ffff (0x9718 bytes) 000489 1788864901.502506 mapwrite 0x030f80 thru 0x03d4aa (0xc52b bytes) 000490 1788864901.555444 mapwrite 0x020a0f thru 0x027ac1 (0x70b3 bytes) 000491 1788864901.602141 mapwrite 0x0114c3 thru 0x014911 (0x344f bytes) 000492 1788864901.648120 punch from 0x030540 to 0x03f953 (0xf414 bytes) 000495 1788864901.658710 MIRROR_EXTEND to 2 mirrors 000496 1788864902.440722 mapread 0x028916 thru 0x02ce70 (0x455b bytes) 000497 1788864902.460072 punch from 0x0370a6 to 0x03ffff (0x8f5a bytes) 000500 1788864902.543178 mapwrite 0x02b977 thru 0x02eea8 (0x3532 bytes) 000501 1788864902.597435 mapwrite 0x01bb7e thru 0x023ffe (0x8481 bytes) 000503 1788864902.669679 trunc from 0x040000 to 0x023b47 (0x1c4b8 bytes) 000504 1788864902.701132 mapwrite 0x02b2c7 thru 0x02f49a (0x41d4 bytes) 000505 1788864902.789895 write 0x007ff0 thru 0x00b7c3 (0x37d4 bytes) 000507 1788864902.809318 MIRROR_RESYNC 2 mirrors 000508 1788864903.385921 punch from 0x00c9f2 to 0x00f581 (0x2b90 bytes) 000509 1788864903.429440 punch from 0x01979e to 0x01d89f (0x4102 bytes) 000511 1788864903.439734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f49b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f49b): Mirror 2: 0x7b51638e (stale) CRC-32 checksum value for chunk [0, 0x2f49b): Mirror 1: 0xac205ee0 Mirror 3: 0xac205ee0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f49b: skipped 000513 1788864904.152804 read 0x02534b thru 0x02b036 (0x5cec bytes) 000514 1788864904.165064 mapread 0x011c92 thru 0x01bb82 (0x9ef1 bytes) 000515 1788864904.200809 MIRROR_RESYNC 3 mirrors 000517 1788864904.892432 trunc from 0x02f49b to 0x019ef6 (0x155a4 bytes) 000518 1788864904.982345 read 0x007fca thru 0x0168dc (0xe913 bytes) 000519 1788864904.999089 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19ef7): Mirror 3: 0x97be895 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19ef7): Mirror 3: 0x97be895 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19ef7: skipped 000520 1788864905.280365 mapwrite 0x03c9dc thru 0x03ffff (0x3624 bytes) 000521 1788864905.345988 mapwrite 0x01440e thru 0x023c76 (0xf869 bytes) 000525 1788864905.422397 read 0x01ec31 thru 0x024e2c (0x61fc bytes) 000527 1788864905.428461 MIRROR_RESYNC 2 mirrors 000528 1788864906.355275 mapwrite 0x001f40 thru 0x008ce1 (0x6da2 bytes) 000529 1788864906.551891 read 0x00b466 thru 0x01a31d (0xeeb8 bytes) 000530 1788864906.565598 punch from 0x0205c5 to 0x027efa (0x7936 bytes) 000533 1788864906.576576 MIRROR_RESYNC 2 mirrors 000534 1788864907.248324 MIRROR_RESYNC 2 mirrors 000536 1788864907.599227 punch from 0x020e4a to 0x02229f (0x1456 bytes) 000539 1788864907.680994 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x39526a49 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39526a49 Mirror 4: 0x39526a49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1788864909.084668 punch from 0x00c85e to 0x00d650 (0xdf3 bytes) 000541 1788864909.169921 read 0x016e8a thru 0x01bced (0x4e64 bytes) 000543 1788864909.181157 read 0x004fbc thru 0x006fa7 (0x1fec bytes) 000546 1788864909.195935 mapread 0x024136 thru 0x025fd4 (0x1e9f bytes) 000547 1788864909.197341 write 0x012054 thru 0x018ebf (0x6e6c bytes) 000549 1788864909.200251 write 0x011073 thru 0x01d134 (0xc0c2 bytes) 000551 1788864909.202780 read 0x02f497 thru 0x03574b (0x62b5 bytes) 000553 1788864909.205004 punch from 0x00f2fe to 0x012d16 (0x3a19 bytes) 000555 1788864909.224015 trunc from 0x040000 to 0x0125cd (0x2da32 bytes) 000556 1788864909.244978 trunc from 0x0125ce to 0x027792 (0x151c5 bytes) 000557 1788864909.268269 mapread 0x01e3de thru 0x027792 (0x93b5 bytes) 000558 1788864909.281271 write 0x02b62e thru 0x02be4b (0x81e bytes) 000561 1788864909.283028 mapwrite 0x018277 thru 0x024336 (0xc0c0 bytes) 000563 1788864909.344797 read 0x028a46 thru 0x02be4b (0x3406 bytes) 000565 1788864909.357349 MIRROR_RESYNC 3 mirrors 000566 1788864910.223643 punch from 0x01a714 to 0x01b55c (0xe49 bytes) 000568 1788864910.368119 punch from 0x0157dd to 0x020a44 (0xb268 bytes) 000569 1788864910.383750 mapread 0x0293c4 thru 0x02be4b (0x2a88 bytes) 000572 1788864910.424260 trunc from 0x02be4c to 0x032825 (0x69da bytes) 000573 1788864910.509045 punch from 0x007727 to 0x0110b7 (0x9991 bytes) 000575 1788864910.521336 write 0x03cec7 thru 0x03ffff (0x3139 bytes) 000576 1788864910.522734 trunc from 0x040000 to 0x03df9d (0x2062 bytes) 000578 1788864910.571411 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df9e): Mirror 4: 0x148d4834 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df9e): Mirror 4: 0x148d4834 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df9e: skipped 000579 1788864911.166792 mapwrite 0x01b780 thru 0x023ba2 (0x8423 bytes) 000580 1788864911.434000 write 0x030d25 thru 0x038534 (0x7810 bytes) 000581 1788864911.450810 trunc from 0x03df9e to 0x002a42 (0x3b55b bytes) 000582 1788864911.488898 mapwrite 0x011290 thru 0x01f42e (0xe19f bytes) 000583 1788864911.681593 read 0x0117e7 thru 0x01dc60 (0xc47a bytes) 000585 1788864911.682620 mapread 0x01a796 thru 0x01e0c0 (0x392b bytes) 000588 1788864911.700702 MIRROR_RESYNC 2 mirrors 000589 1788864912.447489 MIRROR_RESYNC 2 mirrors 000590 1788864912.730222 trunc from 0x01f42f to 0x02ebfa (0xf7cc bytes) 000591 1788864912.831305 mapwrite 0x011e94 thru 0x01f61b (0xd788 bytes) 000592 1788864912.889728 trunc from 0x02ebfb to 0x03834c (0x9752 bytes) 000593 1788864912.905418 write 0x00d388 thru 0x0167ee (0x9467 bytes) 000594 1788864912.906639 read 0x033e7c thru 0x03834c (0x44d1 bytes) 000598 1788864912.918301 mapwrite 0x02570c thru 0x032398 (0xcc8d bytes) 000599 1788864912.984581 punch from 0x021f4e to 0x02999b (0x7a4e bytes) 000600 1788864912.992873 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3834d): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3834d): Mirror 4: 0x86e6a464 (stale) CRC-32 checksum value for chunk [0, 0x3834d): Mirror 1: 0x15f44f15 Mirror 5: 0x15f44f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3834d: skipped 000601 1788864913.780804 write 0x0237eb thru 0x027f26 (0x473c bytes) 000602 1788864913.848580 punch from 0x02cd18 to 0x03834c (0xb635 bytes) 000603 1788864913.861967 mapwrite 0x0048b4 thru 0x00cf50 (0x869d bytes) 000604 1788864913.920239 punch from 0x005333 to 0x00c095 (0x6d63 bytes) 000605 1788864913.923821 read 0x0178be thru 0x025c55 (0xe398 bytes) 000606 1788864913.931294 mapwrite 0x03b70d thru 0x03ffff (0x48f3 bytes) 000608 1788864913.993418 write 0x02f4e3 thru 0x03ea00 (0xf51e bytes) 000610 1788864913.997983 read 0x0259e0 thru 0x034d05 (0xf326 bytes) 000612 1788864914.015439 MIRROR_RESYNC 3 mirrors 000613 1788864914.670143 punch from 0x01f9a9 to 0x028647 (0x8c9f bytes) 000614 1788864914.724226 read 0x010664 thru 0x01c9d9 (0xc376 bytes) 000615 1788864914.736756 trunc from 0x040000 to 0x002292 (0x3dd6d bytes) 000617 1788864914.774116 MIRROR_RESYNC 3 mirrors 000618 1788864915.428131 mapread 0x000d6b thru 0x002292 (0x1528 bytes) 000619 1788864915.445408 read 0x001300 thru 0x002292 (0xf93 bytes) 000620 1788864915.445860 write 0x014335 thru 0x0204fa (0xc1c6 bytes) 000621 1788864915.579208 read 0x00b90c thru 0x014c9a (0x938f bytes) 000622 1788864915.593649 punch from 0x00a12f to 0x01584a (0xb71c bytes) 000623 1788864915.605437 trunc from 0x0204fb to 0x038d71 (0x18877 bytes) 000624 1788864915.620150 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38d72): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d72): Mirror 4: 0x646d4029 (stale) Mirror 5: 0x646d4029 (stale) CRC-32 checksum value for chunk [0, 0x38d72): Mirror 1: 0xe799d112 Mirror 6: 0xe799d112 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d72: skipped 000625 1788864916.661254 trunc from 0x038d72 to 0x03fad8 (0x6d67 bytes) 000626 1788864916.810624 mapread 0x032c27 thru 0x03cec7 (0xa2a1 bytes) 000627 1788864916.845395 punch from 0x00cb0c to 0x0199ef (0xcee4 bytes) 000629 1788864916.851218 mapwrite 0x021f9d thru 0x0234e5 (0x1549 bytes) 000630 1788864916.913994 trunc from 0x03fad9 to 0x02491d (0x1b1bb bytes) 000631 1788864916.931380 write 0x02ced6 thru 0x039ac8 (0xcbf3 bytes) 000632 1788864916.934164 mapwrite 0x011b22 thru 0x01c6c9 (0xaba8 bytes) 000633 1788864916.995096 mapwrite 0x005e38 thru 0x00c5dc (0x67a5 bytes) 000634 1788864917.047630 mapwrite 0x0135b3 thru 0x01e175 (0xabc3 bytes) 000636 1788864917.086206 write 0x014dc2 thru 0x024cfd (0xff3c bytes) 000638 1788864917.087933 write 0x025fec thru 0x02f177 (0x918c bytes) 000639 1788864917.097516 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ac9): Mirror 5: 0x4a81c119 (stale) Mirror 6: 0xcbf7d6e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ac9): Mirror 5: 0x4a81c119 (stale) Mirror 6: 0xcbf7d6e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ac9: skipped 000641 1788864917.488633 punch from 0x01dbb2 to 0x023396 (0x57e5 bytes) 000642 1788864917.492130 mapread 0x0149d5 thru 0x022d9d (0xe3c9 bytes) 000644 1788864917.515064 write 0x039108 thru 0x03ffff (0x6ef8 bytes) 000645 1788864917.519582 read 0x00c0c5 thru 0x014e5e (0x8d9a bytes) 000647 1788864917.529870 MIRROR_RESYNC 3 mirrors 000648 1788864918.189210 write 0x034432 thru 0x03ffff (0xbbce bytes) 000650 1788864918.255749 punch from 0x007097 to 0x0120e5 (0xb04f bytes) 000652 1788864918.268505 punch from 0x00a76c to 0x00de57 (0x36ec bytes) 000653 1788864918.273051 mapwrite 0x0045cb thru 0x00d536 (0x8f6c bytes) 000654 1788864918.333614 write 0x0267b8 thru 0x02a9a4 (0x41ed bytes) 000655 1788864918.343330 write 0x037dbd thru 0x03ffff (0x8243 bytes) 000656 1788864918.344742 read 0x028558 thru 0x03008c (0x7b35 bytes) 000657 1788864918.352093 mapread 0x009b16 thru 0x013e83 (0xa36e bytes) 000658 1788864918.358849 mapwrite 0x00d46f thru 0x014f04 (0x7a96 bytes) 000659 1788864918.408569 write 0x03b170 thru 0x03ffff (0x4e90 bytes) 000661 1788864918.412731 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbafe72f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbafe72f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000662 1788864918.627136 write 0x01e920 thru 0x02b3c6 (0xcaa7 bytes) 000663 1788864918.647111 mapwrite 0x000784 thru 0x001952 (0x11cf bytes) 000664 1788864918.716426 write 0x034686 thru 0x03469a (0x15 bytes) 000665 1788864918.727288 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbafe72f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa8c50c05 Mirror 7: 0xa8c50c05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000666 1788864919.333817 mapread 0x03353f thru 0x03ffff (0xcac1 bytes) 000667 1788864919.362037 punch from 0x00528f to 0x0114fd (0xc26f bytes) 000668 1788864919.432488 read 0x01ec4f thru 0x02df14 (0xf2c6 bytes) 000669 1788864919.448628 mapwrite 0x03a271 thru 0x03ffff (0x5d8f bytes) 000670 1788864919.499890 read 0x02afe9 thru 0x02bf82 (0xf9a bytes) 000672 1788864919.503736 MIRROR_RESYNC 3 mirrors 000673 1788864920.212519 punch from 0x03330d to 0x036e76 (0x3b6a bytes) 000674 1788864920.277837 MIRROR_RESYNC 3 mirrors 000675 1788864921.043477 read 0x0159fe thru 0x01dd58 (0x835b bytes) 000676 1788864921.056520 write 0x02c426 thru 0x03bcbf (0xf89a bytes) 000677 1788864921.114149 mapread 0x00898b thru 0x00dc93 (0x5309 bytes) 000678 1788864921.134692 mapwrite 0x01517c thru 0x01fe64 (0xace9 bytes) 000679 1788864921.191698 trunc from 0x040000 to 0x02f24b (0x10db4 bytes) 000683 1788864921.217853 write 0x01bd77 thru 0x01d05d (0x12e7 bytes) 000684 1788864921.219640 read 0x002c61 thru 0x012215 (0xf5b5 bytes) 000685 1788864921.229079 read 0x008b49 thru 0x00a39f (0x1857 bytes) 000686 1788864921.230750 punch from 0x023e80 to 0x02c115 (0x8296 bytes) 000688 1788864921.240176 read 0x00a151 thru 0x00edcc (0x4c7c bytes) 000690 1788864921.241527 trunc from 0x02f24c to 0x0241c1 (0xb08a bytes) 000693 1788864921.263873 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x241c2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c2): Mirror 6: 0x1c448e12 (stale) Mirror 7: 0x1c448e12 (stale) CRC-32 checksum value for chunk [0, 0x241c2): Mirror 1: 0x98b9103e Mirror 8: 0x98b9103e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c2: skipped 000696 1788864922.016814 write 0x000ed7 thru 0x004d1e (0x3e48 bytes) 000697 1788864922.084726 mapwrite 0x020ca6 thru 0x0223fa (0x1755 bytes) 000698 1788864922.136632 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x241c2): Mirror 7: 0x1c448e12 (stale) Mirror 8: 0x98b9103e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241c2): Mirror 7: 0x1c448e12 (stale) Mirror 8: 0x98b9103e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241c2: skipped 000699 1788864922.381371 punch from 0x01ac90 to 0x0241c1 (0x9532 bytes) 000700 1788864922.385825 read 0x01225a thru 0x01d0fc (0xaea3 bytes) 000701 1788864922.398708 punch from 0x010cf8 to 0x011ca3 (0xfac bytes) 000702 1788864922.403987 write 0x0385dc thru 0x03ffff (0x7a24 bytes) 000703 1788864922.405284 trunc from 0x040000 to 0x00894b (0x376b4 bytes) 000705 1788864922.444016 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x894c): Mirror 7: 0x1c13edf3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x894c): Mirror 7: 0x1c13edf3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x894c: skipped 000706 1788864922.660772 trunc from 0x00894c to 0x005230 (0x371b bytes) 000707 1788864922.674136 mapread 0x002bc6 thru 0x005230 (0x266b bytes) 000708 1788864922.689397 trunc from 0x005231 to 0x013ea3 (0xec73 bytes) 000709 1788864922.704727 mapread 0x00d1aa thru 0x00e8b4 (0x170b bytes) 000710 1788864922.714962 mapread 0x00d794 thru 0x013ea3 (0x6710 bytes) 000711 1788864922.715998 mapread 0x008d0d thru 0x0124db (0x97cf bytes) 000712 1788864922.718056 mapread 0x00665b thru 0x0074e3 (0xe89 bytes) 000714 1788864922.719685 punch from 0x00949d to 0x00c0b5 (0x2c19 bytes) 000716 1788864922.724825 write 0x0016b1 thru 0x0061a1 (0x4af1 bytes) 000719 1788864922.727492 write 0x00668d thru 0x013d7a (0xd6ee bytes) 000721 1788864922.730143 trunc from 0x013ea4 to 0x01893b (0x4a98 bytes) 000722 1788864922.749677 trunc from 0x01893c to 0x00164f (0x172ec bytes) 000723 1788864922.769605 write 0x032dd1 thru 0x03ffff (0xd22f bytes) 000725 1788864922.771660 punch from 0x02d1dc to 0x03d018 (0xfe3d bytes) 000727 1788864922.785844 trunc from 0x040000 to 0x00af53 (0x350ac bytes) 000728 1788864922.799231 trunc from 0x00af54 to 0x01fd44 (0x14df1 bytes) 000729 1788864922.813716 trunc from 0x01fd45 to 0x01d92c (0x2418 bytes) 000731 1788864922.838620 read 0x010f21 thru 0x0171c0 (0x62a0 bytes) 000732 1788864922.846318 punch from 0x00ffdd to 0x018616 (0x863a bytes) 000734 1788864922.850516 trunc from 0x01d92d to 0x03d354 (0x1fa28 bytes) 000736 1788864922.868927 punch from 0x026df9 to 0x02f401 (0x8609 bytes) 000737 1788864922.881000 MIRROR_RESYNC 2 mirrors 000738 1788864923.613260 punch from 0x015c62 to 0x023d42 (0xe0e1 bytes) 000740 1788864923.702688 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000741 1788864923.832911 read 0x00aa1e thru 0x01960c (0xebef bytes) 000742 1788864923.852251 read 0x033b26 thru 0x036f2f (0x340a bytes) 000743 1788864923.860871 mapread 0x00f449 thru 0x01cc3b (0xd7f3 bytes) 000744 1788864923.862254 mapwrite 0x0268da thru 0x028c52 (0x2379 bytes) 000745 1788864923.900666 mapread 0x0289da thru 0x02c409 (0x3a30 bytes) 000747 1788864923.901981 punch from 0x027a48 to 0x02c80a (0x4dc3 bytes) 000748 1788864923.906943 mapwrite 0x01645f thru 0x01cfe2 (0x6b84 bytes) 000750 1788864923.948382 trunc from 0x03d355 to 0x03413a (0x921a bytes) 000752 1788864923.964039 punch from 0x00e5e4 to 0x016345 (0x7d62 bytes) 000753 1788864923.969674 mapread 0x014ea7 thru 0x01d7f7 (0x8951 bytes) 000755 1788864923.983693 read 0x0063e9 thru 0x00b4c8 (0x50e0 bytes) 000757 1788864923.985772 write 0x0120b6 thru 0x01aef8 (0x8e43 bytes) 000762 1788864923.999767 read 0x00b4c5 thru 0x012dcd (0x7909 bytes) 000763 1788864924.000669 read 0x00173c thru 0x008325 (0x6bea bytes) 000764 1788864924.001279 read 0x006770 thru 0x00d9b5 (0x7246 bytes) 000766 1788864924.003296 trunc from 0x03413b to 0x00832e (0x2be0c bytes) 000767 1788864924.019111 write 0x034edb thru 0x03c115 (0x723b bytes) 000769 1788864924.022387 read 0x0103e1 thru 0x0116a9 (0x12c9 bytes) 000770 1788864924.031276 read 0x02598d thru 0x02c7f5 (0x6e69 bytes) 000772 1788864924.033398 punch from 0x035fc2 to 0x039bf8 (0x3c37 bytes) 000775 1788864924.052804 MIRROR_EXTEND to 2 mirrors 000776 1788864924.859000 read 0x0022d6 thru 0x00b2b0 (0x8fdb bytes) 000777 1788864924.886378 MIRROR_RESYNC 2 mirrors 000780 1788864925.206302 write 0x0159c9 thru 0x023a74 (0xe0ac bytes) 000781 1788864925.284240 mapread 0x00cba5 thru 0x0168b2 (0x9d0e bytes) 000782 1788864925.304060 write 0x02c294 thru 0x02ff9a (0x3d07 bytes) 000783 1788864925.320521 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: 0x2f93ecd4 (stale) CRC-32 checksum value for chunk [0, 0x3c116): Mirror 1: 0xa1a7dfff Mirror 3: 0xa1a7dfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c116: skipped 000784 1788864926.177387 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: 0x2f93ecd4 (stale) CRC-32 checksum value for chunk [0, 0x3c116): Mirror 1: 0xa1a7dfff Mirror 3: 0xa1a7dfff Mirror 4: 0xa1a7dfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c116: skipped 000786 1788864926.993511 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: 0x2f93ecd4 (stale) CRC-32 checksum value for chunk [0, 0x3c116): Mirror 1: 0xa1a7dfff Mirror 3: 0xa1a7dfff Mirror 4: 0xa1a7dfff Mirror 5: 0xa1a7dfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c116: skipped 000788 1788864928.026659 mapwrite 0x010cff thru 0x016dbf (0x60c1 bytes) 000790 1788864928.184707 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: 0x2f93ecd4 (stale) Mirror 3: 0xa1a7dfff (stale) Mirror 4: 0xa1a7dfff (stale) Mirror 5: 0xa1a7dfff (stale) CRC-32 checksum value for chunk [0, 0x3c116): Mirror 1: 0x6eaf9f30 Mirror 6: 0x6eaf9f30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c116: skipped 000791 1788864929.112389 write 0x013a10 thru 0x01c855 (0x8e46 bytes) 000794 1788864929.263324 MIRROR_RESYNC 6 mirrors 000796 1788864930.602251 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000798 1788864931.038227 read 0x01223f thru 0x013b81 (0x1943 bytes) 000799 1788864931.048010 read 0x037462 thru 0x03c115 (0x4cb4 bytes) 000801 1788864931.065418 mapwrite 0x00a096 thru 0x00b3b3 (0x131e bytes) 000802 1788864931.207102 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c116): Mirror 2: 0x928974fd (stale) Mirror 3: 0x928974fd (stale) Mirror 4: 0x928974fd (stale) Mirror 5: 0x928974fd (stale) CRC-32 checksum value for chunk [0, 0x3c116): Mirror 1: 0x6d5bf32d Mirror 6: 0x6d5bf32d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c116: skipped 000803 1788864932.227478 mapread 0x03b4e7 thru 0x03c115 (0xc2f bytes) 000805 1788864932.298144 mapwrite 0x02e6d0 thru 0x036786 (0x80b7 bytes) 000807 1788864932.548302 punch from 0x011fc3 to 0x021708 (0xf746 bytes) 000808 1788864932.553122 punch from 0x00f8c5 to 0x018e1a (0x9556 bytes) 000809 1788864932.557861 trunc from 0x03c116 to 0x03c24c (0x137 bytes) 000812 1788864932.580969 read 0x02c5a8 thru 0x030142 (0x3b9b bytes) 000813 1788864932.582151 write 0x02793e thru 0x02d517 (0x5bda bytes) 000815 1788864932.585204 punch from 0x02d63f to 0x039d8b (0xc74d bytes) 000817 1788864932.608456 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c24d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c24d): Mirror 2: 0xd07f2902 (stale) Mirror 3: 0xd07f2902 (stale) Mirror 4: 0xd07f2902 (stale) Mirror 5: 0xd07f2902 (stale) Mirror 6: 0x44c561fd (stale) CRC-32 checksum value for chunk [0, 0x3c24d): Mirror 1: 0x3b530742 Mirror 7: 0x3b530742 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c24d: skipped 000818 1788864933.752162 mapwrite 0x010696 thru 0x01378d (0x30f8 bytes) 000819 1788864933.970436 trunc from 0x03c24d to 0x027720 (0x14b2c bytes) 000820 1788864933.992527 mapread 0x02329a thru 0x027720 (0x4487 bytes) 000821 1788864934.005822 trunc from 0x027721 to 0x0306cc (0x8fac bytes) 000823 1788864934.020242 mapwrite 0x037711 thru 0x03ffff (0x88ef bytes) 000824 1788864934.097579 read 0x00add5 thru 0x00c11d (0x1349 bytes) 000825 1788864934.099545 punch from 0x020cda to 0x02ac7d (0x9fa4 bytes) 000826 1788864934.105068 write 0x024199 thru 0x02a15e (0x5fc6 bytes) 000827 1788864934.114380 write 0x034ea0 thru 0x036d15 (0x1e76 bytes) 000829 1788864934.120728 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6fc0367c (stale) Mirror 3: 0x6fc0367c (stale) Mirror 4: 0x6fc0367c (stale) Mirror 5: 0x6fc0367c (stale) Mirror 6: 0x304fd628 (stale) Mirror 7: 0xaad66d88 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb4012ff Mirror 8: 0xdb4012ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1788864934.852926 mapwrite 0x03bae9 thru 0x03df35 (0x244d bytes) 000834 1788864935.019030 mapread 0x02b637 thru 0x030bc8 (0x5592 bytes) 000835 1788864935.030425 read 0x006dc7 thru 0x010553 (0x978d bytes) 000836 1788864935.040963 read 0x01b8cc thru 0x023361 (0x7a96 bytes) 000837 1788864935.055775 mapread 0x014d4b thru 0x01f400 (0xa6b6 bytes) 000838 1788864935.066781 MIRROR_RESYNC 8 mirrors 000840 1788864936.104318 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000841 1788864936.392227 read 0x006ea6 thru 0x01409d (0xd1f8 bytes) 000842 1788864936.407701 punch from 0x03ca4a to 0x03ffff (0x35b6 bytes) 000844 1788864936.473707 punch from 0x0336d5 to 0x039438 (0x5d64 bytes) 000845 1788864936.483057 punch from 0x016545 to 0x01bb47 (0x5603 bytes) 000846 1788864936.487811 read 0x006c04 thru 0x007e8b (0x1288 bytes) 000847 1788864936.500341 MIRROR_RESYNC 7 mirrors 000848 1788864937.463161 mapwrite 0x016772 thru 0x01b4b3 (0x4d42 bytes) 000849 1788864937.595937 trunc from 0x040000 to 0x037e50 (0x81af bytes) 000850 1788864937.615995 punch from 0x030574 to 0x037e50 (0x78dd bytes) 000851 1788864937.620112 read 0x02481c thru 0x0315eb (0xcdd0 bytes) 000852 1788864937.630177 write 0x01ef0c thru 0x01fda2 (0xe97 bytes) 000857 1788864937.632361 mapread 0x02785b thru 0x0292b9 (0x1a5f bytes) 000858 1788864937.633332 punch from 0x001d62 to 0x00d1a5 (0xb444 bytes) 000859 1788864937.641703 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37e51): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e51): Mirror 2: 0xd5330a9 (stale) Mirror 3: 0xd5330a9 (stale) Mirror 5: 0xd5330a9 (stale) Mirror 6: 0xd5330a9 (stale) Mirror 7: 0xd5330a9 (stale) Mirror 8: 0xd5330a9 (stale) CRC-32 checksum value for chunk [0, 0x37e51): Mirror 1: 0x32a2ea40 Mirror 9: 0x32a2ea40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e51: skipped 000860 1788864938.421640 mapwrite 0x0215f7 thru 0x021cab (0x6b5 bytes) 000862 1788864938.549775 read 0x02792e thru 0x029825 (0x1ef8 bytes) 000863 1788864938.555143 read 0x02f390 thru 0x030e66 (0x1ad7 bytes) 000865 1788864938.564470 mapwrite 0x012fdf thru 0x020173 (0xd195 bytes) 000867 1788864938.601860 read 0x0224bb thru 0x0261a0 (0x3ce6 bytes) 000869 1788864938.607688 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37e51): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e51): Mirror 2: 0xd5330a9 (stale) Mirror 3: 0xd5330a9 (stale) Mirror 5: 0xd5330a9 (stale) Mirror 6: 0xd5330a9 (stale) Mirror 7: 0xd5330a9 (stale) Mirror 8: 0xd5330a9 (stale) Mirror 9: 0x32a2ea40 (stale) CRC-32 checksum value for chunk [0, 0x37e51): Mirror 1: 0x9f237f40 Mirror 10: 0x9f237f40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e51: skipped 000871 1788864939.266493 read 0x02017c thru 0x02aa87 (0xa90c bytes) 000874 1788864939.283205 read 0x010d51 thru 0x0173b7 (0x6667 bytes) 000875 1788864939.294502 read 0x029336 thru 0x034019 (0xace4 bytes) 000876 1788864939.297957 read 0x000dd6 thru 0x0015b3 (0x7de bytes) 000877 1788864939.301734 write 0x00734d thru 0x007973 (0x627 bytes) 000878 1788864939.383784 trunc from 0x037e51 to 0x0150d8 (0x22d78 bytes) 000881 1788864939.424732 mapwrite 0x00c912 thru 0x01a945 (0xe034 bytes) 000882 1788864939.499402 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a946): Mirror 2: 0xc64eeda0 (stale) Mirror 3: 0xc64eeda0 (stale) Mirror 5: 0xc64eeda0 (stale) Mirror 6: 0xc64eeda0 (stale) Mirror 7: 0xc64eeda0 (stale) Mirror 8: 0xc64eeda0 (stale) Mirror 10: 0xededd0af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a946): Mirror 2: 0xc64eeda0 (stale) Mirror 3: 0xc64eeda0 (stale) Mirror 5: 0xc64eeda0 (stale) Mirror 6: 0xc64eeda0 (stale) Mirror 7: 0xc64eeda0 (stale) Mirror 8: 0xc64eeda0 (stale) Mirror 10: 0xededd0af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a946: skipped 000884 1788864939.992207 mapread 0x00e2e0 thru 0x013ef9 (0x5c1a bytes) 000885 1788864940.010147 trunc from 0x01a946 to 0x010ffa (0x994b bytes) 000886 1788864940.023954 mapwrite 0x00d045 thru 0x0171b9 (0xa175 bytes) 000887 1788864940.095178 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x171ba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x171ba): Mirror 2: 0x10d99d0f (stale) Mirror 3: 0x10d99d0f (stale) Mirror 5: 0x10d99d0f (stale) Mirror 6: 0x10d99d0f (stale) Mirror 7: 0x10d99d0f (stale) Mirror 8: 0x10d99d0f (stale) Mirror 10: 0x3328b04c (stale) CRC-32 checksum value for chunk [0, 0x171ba): Mirror 1: 0x552ce6d Mirror 11: 0x552ce6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x171ba: skipped 000889 1788864940.880932 write 0x00215a thru 0x008408 (0x62af bytes) 000891 1788864941.002299 mapwrite 0x034619 thru 0x0374f4 (0x2edc bytes) 000892 1788864941.121470 read 0x02ff5a thru 0x0374f4 (0x759b bytes) 000893 1788864941.123113 punch from 0x00ca07 to 0x01722b (0xa825 bytes) 000894 1788864941.128722 read 0x03407e thru 0x034242 (0x1c5 bytes) 000895 1788864941.129269 read 0x01e116 thru 0x02215b (0x4046 bytes) 000896 1788864941.153788 punch from 0x03693c to 0x0374f4 (0xbb9 bytes) 000898 1788864941.159291 read 0x002382 thru 0x008b72 (0x67f1 bytes) 000899 1788864941.160172 trunc from 0x0374f5 to 0x02a054 (0xd4a0 bytes) 000901 1788864941.177819 write 0x038e09 thru 0x03c591 (0x3789 bytes) 000902 1788864941.178977 trunc from 0x03c592 to 0x01d96e (0x1ec23 bytes) 000905 1788864941.199461 read 0x019fd4 thru 0x01bb46 (0x1b73 bytes) 000906 1788864941.206284 MIRROR_RESYNC 9 mirrors 000908 1788864942.673219 read 0x014490 thru 0x01d96e (0x94df bytes) 000909 1788864942.680142 read 0x015aaa thru 0x01be86 (0x63dd bytes) 000910 1788864942.680463 mapread 0x01617f thru 0x01d96e (0x77f0 bytes) 000911 1788864942.681922 write 0x0181c4 thru 0x0254f6 (0xd333 bytes) 000912 1788864942.757196 punch from 0x0039e2 to 0x0040d4 (0x6f3 bytes) 000915 1788864942.773806 mapwrite 0x001bfb thru 0x011175 (0xf57b bytes) 000916 1788864942.855791 mapwrite 0x021e78 thru 0x02e282 (0xc40b bytes) 000917 1788864942.938799 trunc from 0x02e283 to 0x01bacc (0x127b6 bytes) 000918 1788864942.956255 mapwrite 0x018334 thru 0x01ec7d (0x694a bytes) 000920 1788864943.034066 trunc from 0x01ec7e to 0x03b4dc (0x1c85f bytes) 000921 1788864943.052866 punch from 0x001083 to 0x002adc (0x1a5a bytes) 000922 1788864943.056045 mapwrite 0x020b35 thru 0x02d9f1 (0xcebd bytes) 000923 1788864943.184908 mapwrite 0x0349d2 thru 0x03a14a (0x5779 bytes) 000924 1788864943.232764 mapwrite 0x02b802 thru 0x033a9c (0x829b bytes) 000926 1788864943.270611 punch from 0x00b30c to 0x012179 (0x6e6e bytes) 000927 1788864943.274145 read 0x02f771 thru 0x03b4dc (0xbd6c bytes) 000928 1788864943.274594 read 0x02eee2 thru 0x0318cc (0x29eb bytes) 000930 1788864943.275199 punch from 0x02b7a2 to 0x035705 (0x9f64 bytes) 000931 1788864943.279145 write 0x02cd3c thru 0x03a931 (0xdbf6 bytes) 000932 1788864943.287026 trunc from 0x03b4dd to 0x02d902 (0xdbda bytes) 000933 1788864943.295957 read 0x01822b thru 0x0182cd (0xa3 bytes) 000934 1788864943.296408 read 0x00953e thru 0x00dc29 (0x46ec bytes) 000937 1788864943.298339 write 0x03d6cd thru 0x03ffff (0x2933 bytes) 000938 1788864943.299414 write 0x02580f thru 0x02e4f6 (0x8ce8 bytes) 000939 1788864943.305744 read 0x02325a thru 0x02c0bc (0x8e63 bytes) 000940 1788864943.306334 punch from 0x018cb3 to 0x02368f (0xa9dd bytes) 000943 1788864943.320054 write 0x02f49b thru 0x037116 (0x7c7c bytes) 000944 1788864943.321342 read 0x02adb2 thru 0x03059c (0x57eb bytes) 000945 1788864943.330132 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56221661 (stale) Mirror 3: 0x56221661 (stale) Mirror 5: 0x56221661 (stale) Mirror 6: 0x56221661 (stale) Mirror 7: 0x56221661 (stale) Mirror 8: 0x56221661 (stale) Mirror 10: 0x56221661 (stale) Mirror 11: 0x56221661 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35474b15 Mirror 12: 0x35474b15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000946 1788864944.207350 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56221661 (stale) Mirror 3: 0x56221661 (stale) Mirror 5: 0x56221661 (stale) Mirror 6: 0x56221661 (stale) Mirror 8: 0x56221661 (stale) Mirror 10: 0x56221661 (stale) Mirror 11: 0x56221661 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56221661 (stale) Mirror 3: 0x56221661 (stale) Mirror 5: 0x56221661 (stale) Mirror 6: 0x56221661 (stale) Mirror 8: 0x56221661 (stale) Mirror 10: 0x56221661 (stale) Mirror 11: 0x56221661 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35474b15 Mirror 12: 0x35474b15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1788864944.885095 mapwrite 0x0126f4 thru 0x01afca (0x88d7 bytes) 000948 1788864945.108018 MIRROR_RESYNC 9 mirrors 000949 1788864946.734670 MIRROR_RESYNC 9 mirrors 000951 1788864947.215790 mapwrite 0x0147ef thru 0x01c9d7 (0x81e9 bytes) 000952 1788864947.455813 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc0def563 (stale) Mirror 3: 0xc0def563 (stale) Mirror 5: 0xc0def563 (stale) Mirror 6: 0xc0def563 (stale) Mirror 8: 0xc0def563 (stale) Mirror 10: 0xc0def563 (stale) Mirror 11: 0xc0def563 (stale) Mirror 12: 0xc0def563 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6a1b3e37 Mirror 13: 0x6a1b3e37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1788864948.748265 trunc from 0x040000 to 0x030871 (0xf78e bytes) 000957 1788864948.866864 mapwrite 0x01946f thru 0x025022 (0xbbb4 bytes) 000958 1788864948.953577 MIRROR_RESYNC 10 mirrors 000959 1788864950.280934 trunc from 0x030872 to 0x037ad1 (0x7260 bytes) 000960 1788864950.355256 write 0x005d59 thru 0x014ece (0xf176 bytes) 000961 1788864950.374269 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ad2): Mirror 2: 0x5eec933f (stale) Mirror 3: 0x5eec933f (stale) Mirror 5: 0x5eec933f (stale) Mirror 6: 0x5eec933f (stale) Mirror 8: 0x5eec933f (stale) Mirror 10: 0x5eec933f (stale) Mirror 11: 0x5eec933f (stale) Mirror 13: 0x5eec933f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ad2): Mirror 2: 0x5eec933f (stale) Mirror 3: 0x5eec933f (stale) Mirror 5: 0x5eec933f (stale) Mirror 6: 0x5eec933f (stale) Mirror 8: 0x5eec933f (stale) Mirror 10: 0x5eec933f (stale) Mirror 11: 0x5eec933f (stale) Mirror 13: 0x5eec933f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ad2: skipped 000962 1788864950.851324 MIRROR_RESYNC 9 mirrors 000963 1788864952.040782 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) 000964 1788864952.377202 trunc from 0x037ad2 to 0x017ab1 (0x20020 bytes) 000965 1788864952.458111 trunc from 0x017ab2 to 0x0225e5 (0xab34 bytes) 000968 1788864952.487127 read 0x0075e9 thru 0x012e07 (0xb81f bytes) 000970 1788864952.499126 mapwrite 0x010e01 thru 0x012d3e (0x1f3e bytes) 000971 1788864952.555903 MIRROR_RESYNC 8 mirrors 000974 1788864953.812914 write 0x01c19c thru 0x01d733 (0x1598 bytes) 000975 1788864953.888174 read 0x02131d thru 0x0225e5 (0x12c9 bytes) 000976 1788864953.893251 trunc from 0x0225e6 to 0x035e2e (0x13849 bytes) 000979 1788864953.926118 mapread 0x02f737 thru 0x034160 (0x4a2a bytes) 000980 1788864953.946557 mapwrite 0x0309fb thru 0x03564c (0x4c52 bytes) 000983 1788864954.010562 mapwrite 0x03c8fe thru 0x03ffff (0x3702 bytes) 000984 1788864954.099722 punch from 0x01e160 to 0x01e7ab (0x64c bytes) 000985 1788864954.109587 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91d4b674 (stale) Mirror 3: 0x91d4b674 (stale) Mirror 5: 0x91d4b674 (stale) Mirror 6: 0x91d4b674 (stale) Mirror 8: 0x91d4b674 (stale) Mirror 10: 0x91d4b674 (stale) Mirror 11: 0x91d4b674 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59d633b7 Mirror 12: 0x59d633b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1788864955.065521 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91d4b674 (stale) Mirror 5: 0x91d4b674 (stale) Mirror 6: 0x91d4b674 (stale) Mirror 8: 0x91d4b674 (stale) Mirror 10: 0x91d4b674 (stale) Mirror 11: 0x91d4b674 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91d4b674 (stale) Mirror 5: 0x91d4b674 (stale) Mirror 6: 0x91d4b674 (stale) Mirror 8: 0x91d4b674 (stale) Mirror 10: 0x91d4b674 (stale) Mirror 11: 0x91d4b674 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59d633b7 Mirror 12: 0x59d633b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000988 1788864955.865130 trunc from 0x040000 to 0x011936 (0x2e6c9 bytes) 000989 1788864956.000732 mapread 0x00fc85 thru 0x011936 (0x1cb2 bytes) 000991 1788864956.020444 read 0x008357 thru 0x011936 (0x95e0 bytes) 000992 1788864956.027147 MIRROR_RESYNC 8 mirrors 000993 1788864957.111030 punch from 0x0076f1 to 0x00a88b (0x319b bytes) 000994 1788864957.171184 punch from 0x003521 to 0x00ec49 (0xb729 bytes) 000996 1788864957.175824 trunc from 0x011937 to 0x033e54 (0x2251e bytes) 000998 1788864957.196022 mapwrite 0x01e0bc thru 0x021867 (0x37ac bytes) 000999 1788864957.264251 read 0x0191a5 thru 0x025cd7 (0xcb33 bytes) 001000 1788864957.264936 mapwrite 0x03d060 thru 0x03ffff (0x2fa0 bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 06:56:02 (1788864962) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 06:56:06 (1788864966) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225225 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223799 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x320:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 06:56:15 (1788864975) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3855200 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 2870720 .. Extending file size to 7736896 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 4094848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6248320 .. failed Extending file size to 8694208 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8274720 .. Extending file size to 8878944 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8844320 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9328192 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9007776 .. Waiting 57494 57496 57497 57498 57500 done 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:57:37 (1788865057) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6384352 .. Extending file size to 3452864 .. Extending file size to 8286848 .. Extending file size to 7702848 .. Extending file size to 9390144 .. Extending file size to 8584224 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 59139 59141 59142 59144 59145 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:58:53 (1788865133) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17078 s, 12.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.350527 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa469c4e7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:59:13 (1788865153) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113546 s, 18.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 06:59:20 (1788865160) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x338:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x353:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:59:27 (1788865167) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145865 s, 14.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x355:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:59:35 (1788865175) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238928 s, 8.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x356:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x356:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x357:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:59:44 (1788865184) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272571 s, 7.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x358:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x358:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x359:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:00:19 (1788865219) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176216 s, 6.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33e:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:01:19 (1788865279) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.364612 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:02:17 (1788865337) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. Creating new pool test_204f oleg140-server: Pool lustre.test_204f created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353127 s, 5.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg140-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:03:39 (1788865419) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:03:48 (1788865428) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:03:56 (1788865436) oleg140-server: Pool lustre.test_206 created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg140-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:04:20 (1788865460) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894564 s, 11.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051741 s, 20.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9965f142 Mirror 3: 0x9965f142 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:04:30 (1788865470) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:04:33 (1788865473) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:04:36 (1788865476) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.118288 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2310000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1600000 Read this (4 bytes): BBBB 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:05:10 (1788865510) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0180335 s, 454 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:05:17 (1788865517) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873897 s, 12.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x263 lmm_fid: [0x200000404:0x263:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x263 lmm_fid: [0x200000404:0x263:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834531 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 07:05:18.000000000 -0400 Modify: 2026-09-08 07:05:19.000000000 -0400 Change: 2026-09-08 07:05:20.000000000 -0400 Birth: 2026-09-08 07:05:18.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:05:25 (1788865525) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:05:29 (1788865529) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.540017 s, 75.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0402391 s, 102 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 07:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x265:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 07:05 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:05:38 (1788865538) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:05:45 (1788865545) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1697 sec =========== 07:05:55 (1788865555) === sanity-flr: start cleanup 07:05:57 (1788865557) === === sanity-flr: finish cleanup 07:06:03 (1788865563) ===