-----============= acceptance-small: sanity-flr ============----- Fri Apr 3 01:32:04 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg109-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:32:26 (1775194346) === oleg109-client.virtnet: executing check_config_client /mnt/lustre oleg109-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg109-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b46c3fba800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b46c3fba800.idle_timeout=debug disable quota as required oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:33:03 (1775194383) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7569] == sanity-flr test 0a: lfs mirror create with -N option == 01:33:06 (1775194386) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:33:27 (1775194407) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:33:31 (1775194411) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:33:36 (1775194416) PASS 0d (37s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:34:12 (1775194452) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:34:16 (1775194456) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:34:20 (1775194460) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b46c3fba800.stats=clear osc.lustre-OST0001-osc-ffff8b46c3fba800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:34:30 (1775194470) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:34:44 (1775194484) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:34:55 (1775194495) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:34:59 (1775194499) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 01:35:10 (1775194510) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:35:19 (1775194519) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:35:23 (1775194523) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:35:31 (1775194531) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:35:41 (1775194541) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:35:48 (1775194548) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.3953 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114078 s, 9.2 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:36:00 (1775194560) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.606707 s, 34.6 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:36:10 (1775194570) Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:37:30 (1775194650) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.800828 s, 14.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:38:21 (1775194701) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:39:49 (1775194789) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.69967 s, 6.8 MB/s -rw-r--r-- 1 root root 52428800 Apr 3 01:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.323s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.558s user 0m0.001s sys 0m0.200s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.241s user 0m0.000s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b46c3fba800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:41:35 (1775194895) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:41:38 (1775194898) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:41:41 (1775194901) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 01:41:45 (1775194905) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 01:41:55 (1775194915) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.886588 s, 14.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (17s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:42:12 (1775194932) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335189 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415056 s, 25.3 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:42:21 (1775194941) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:42:30 (1775194950) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967533 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134934 s, 759 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0018394 s, 557 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00154503 s, 663 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:42:42 (1775194962) create test files with size 15750194 .. e35d59f8fe3db52723972d1816da7b4a - a8ea4d061962ee2846a0f0dfc616477d - 6f73fe9286278fd6ebc9a8d367dc1a72 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre PASS 37 (34s) == sanity-flr test 38: resync ============================ 01:43:16 (1775194996) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:43:35 (1775195015) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:43:43 (1775195023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185204 s, 5.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238001 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 01:43:54 (1775195034) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132743 s, 15.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.362027 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276123 s, 7.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 01:44:16 (1775195056) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:44:21 (1775195061) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:44:25 (1775195065) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175936 s, 59.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 01:44:35 (1775195075) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:44:40 (1775195080) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:44:53 (1775195093) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.389643 s, 26.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:45:09 (1775195109) llite.lustre-ffff8b46d098a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.344635 s, 30.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 4102 flags: 1 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 4102 flags: 1 llite.lustre-ffff8b46c50b4800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:45:27 (1775195127) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:46:15 (1775195175) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:47:02 (1775195222) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 01:47:07 (1775195227) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.354766 s, 8.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.154895 s, 20.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbdadced (stale) Mirror 2: 0xbdadced (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: c9f41872e58bb49c35f8302b2be298ae - mirror 1 checksum: 88cac19d522483df0fb786f8e17e7021 - mirror 2 checksum: c9f41872e58bb49c35f8302b2be298ae - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 3 01:47:11 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 3 01:47:11 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c9f41872e58bb49c35f8302b2be298ae - mirror 1 checksum: 88cac19d522483df0fb786f8e17e7021 - mirror 2 checksum: 88cac19d522483df0fb786f8e17e7021 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 3 01:47:11 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 3 01:47:11 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:47:22 (1775195242) Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 01:47:30 (1775195250) shut down facet: mds1 facet_host: oleg109-server facet_failover_host: oleg109-server Failover mds1 to oleg109-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 01:47:54 (1775195274) targets are mounted 01:47:54 (1775195274) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301923 s, 34.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:48:11 (1775195291) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:48:15 (1775195295) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:48:19 (1775195299) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00837196 s, 2.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:48:27 (1775195307) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:48:31 (1775195311) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:48:35 (1775195315) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:49:05 (1775195345) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:49:36 (1775195376) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:50:02 (1775195402) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:50:29 (1775195429) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 01:50:38 (1775195438) mirror create pid 52355 mirror split pid 52356 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52355 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52356 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 01:51:49 (1775195509) Seed set to 5511 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x1703f 000011 1775195511.791459 trunc from 00000000 to 0x01703e (0x1703f bytes) 000012 1775195511.828263 write 0x00bd8c thru 0x00d5b0 (0x1825 bytes) 000013 1775195511.855507 write 0x01d765 thru 0x0220ca (0x4966 bytes) truncating to largest ever: 0x2e34d 000016 1775195511.868311 trunc from 0x0220cb to 0x02e34c (0xc282 bytes) 000019 1775195511.927153 trunc from 0x02e34d to 0x0140e9 (0x1a263 bytes) 000022 1775195511.982853 mapread 0x006d98 thru 0x007adb (0xd44 bytes) 000023 1775195512.004587 read 0x006ed9 thru 0x00bc37 (0x4d5f bytes) 000026 1775195512.011455 mapread 0x005a24 thru 0x00b04b (0x5628 bytes) 000027 1775195512.013344 mapread 0x00d803 thru 0x012e9b (0x5699 bytes) 000028 1775195512.030596 trunc from 0x0140ea to 0x02966b (0x15582 bytes) 000029 1775195512.075305 trunc from 0x02966c to 0x01915c (0x1050f bytes) 000030 1775195512.112078 mapread 0x0085d6 thru 0x010c4f (0x867a bytes) 000032 1775195512.117901 write 0x01e89a thru 0x0269b1 (0x8118 bytes) 000033 1775195512.119619 mapwrite 0x00e72f thru 0x01cf85 (0xe857 bytes) 000034 1775195512.367999 write 0x03d948 thru 0x03f26e (0x1927 bytes) 000035 1775195512.369110 write 0x00f765 thru 0x01d6a8 (0xdf44 bytes) 000036 1775195512.371776 write 0x0035a6 thru 0x0047b2 (0x120d bytes) 000038 1775195512.374463 mapwrite 0x0236f4 thru 0x023846 (0x153 bytes) 000039 1775195512.436872 write 0x0243aa thru 0x031b20 (0xd777 bytes) 000041 1775195512.447575 read 0x025a36 thru 0x02c5b8 (0x6b83 bytes) 000043 1775195512.453174 mapwrite 0x01f49d thru 0x021afe (0x2662 bytes) 000044 1775195512.561187 write 0x012a58 thru 0x01d7bb (0xad64 bytes) 000045 1775195512.576490 read 0x01e993 thru 0x027e85 (0x94f3 bytes) 000046 1775195512.580837 mapread 0x00f611 thru 0x010ae3 (0x14d3 bytes) 000047 1775195512.586666 mapread 0x021f12 thru 0x02d0d6 (0xb1c5 bytes) 000048 1775195512.592464 read 0x01d4c8 thru 0x029f69 (0xcaa2 bytes) 000051 1775195512.600015 trunc from 0x03f26f to 0x024d4d (0x1a521 bytes) 000053 1775195512.635497 write 0x03dd7c thru 0x03ffff (0x2284 bytes) 000056 1775195512.642359 trunc from 0x040000 to 0x01fb8b (0x20474 bytes) 000058 1775195512.693166 mapwrite 0x01fccf thru 0x02146b (0x179d bytes) 000060 1775195512.905066 mapread 0x00d194 thru 0x011f29 (0x4d96 bytes) 000064 1775195512.926195 mapwrite 0x010685 thru 0x014b15 (0x4491 bytes) 000065 1775195513.091835 mapread 0x01e5cd thru 0x020bc8 (0x25fc bytes) truncating to largest ever: 0x3bc39 000068 1775195513.108078 trunc from 0x02146c to 0x03bc38 (0x1a7cd bytes) 000069 1775195513.147875 read 0x0316f0 thru 0x038ae2 (0x73f3 bytes) 000071 1775195513.165829 mapwrite 0x009164 thru 0x017324 (0xe1c1 bytes) 000073 1775195513.260581 trunc from 0x03bc39 to 0x01b25d (0x209db bytes) 000075 1775195513.300371 mapread 0x013b97 thru 0x01b25d (0x76c7 bytes) 000077 1775195513.316850 MIRROR_EXTEND to 2 mirrors 000078 1775195514.480439 mapwrite 0x0033c8 thru 0x006344 (0x2f7d bytes) 000079 1775195514.758079 write 0x01ee9c thru 0x02d1bc (0xe321 bytes) 000080 1775195514.760228 write 0x0272dd thru 0x032d55 (0xba79 bytes) 000082 1775195514.763457 mapread 0x00663f thru 0x008bb4 (0x2576 bytes) 000083 1775195514.771015 MIRROR_RESYNC 2 mirrors 000084 1775195515.604509 write 0x012781 thru 0x01af10 (0x8790 bytes) 000085 1775195515.679346 mapwrite 0x03544f thru 0x03e0b1 (0x8c63 bytes) 000086 1775195515.830128 trunc from 0x03e0b2 to 0x018e60 (0x25251 bytes) 000088 1775195515.867683 read 0x016640 thru 0x018e60 (0x2821 bytes) 000089 1775195515.869082 write 0x028502 thru 0x037c0f (0xf70e bytes) 000090 1775195515.877296 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000091 1775195516.096270 write 0x0314d2 thru 0x03cf02 (0xba31 bytes) 000093 1775195516.111049 read 0x014ce1 thru 0x014fcf (0x2ef bytes) 000095 1775195516.121574 read 0x001993 thru 0x00bb23 (0xa191 bytes) 000098 1775195516.147173 write 0x00a9e6 thru 0x011e7b (0x7496 bytes) 000100 1775195516.150699 mapwrite 0x00b573 thru 0x015fb6 (0xaa44 bytes) 000101 1775195516.226129 write 0x02100e thru 0x02a93b (0x992e bytes) 000103 1775195516.228123 read 0x03b3df thru 0x03cf02 (0x1b24 bytes) 000104 1775195516.230416 write 0x036a09 thru 0x03ffff (0x95f7 bytes) 000105 1775195516.235082 trunc from 0x040000 to 0x039ef2 (0x610d bytes) 000106 1775195516.265276 mapwrite 0x02a725 thru 0x036388 (0xbc64 bytes) 000107 1775195516.332350 mapwrite 0x02b3a6 thru 0x033ad7 (0x8732 bytes) 000111 1775195516.404417 mapwrite 0x01fd0a thru 0x02e6a1 (0xe998 bytes) 000113 1775195516.486176 trunc from 0x039ef3 to 0x01c1f5 (0x1dcfd bytes) 000115 1775195516.518106 write 0x010774 thru 0x0161c7 (0x5a54 bytes) 000116 1775195516.521048 trunc from 0x01c1f6 to 0x01db9f (0x19aa bytes) 000117 1775195516.554911 MIRROR_EXTEND to 2 mirrors 000118 1775195517.383978 read 0x0030ca thru 0x0077af (0x46e6 bytes) 000120 1775195517.408240 MIRROR_EXTEND to 3 mirrors 000121 1775195518.168214 read 0x01987b thru 0x01db9f (0x4325 bytes) 000122 1775195518.179695 write 0x03b831 thru 0x03ffff (0x47cf bytes) 000126 1775195518.261826 trunc from 0x040000 to 0x01ed50 (0x212af bytes) 000127 1775195518.282238 mapwrite 0x002237 thru 0x00d687 (0xb451 bytes) 000128 1775195518.344200 mapread 0x00f501 thru 0x01d0a1 (0xdba1 bytes) 000129 1775195518.366266 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ed51): Mirror 2: 0x2ea2df6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed51): Mirror 2: 0x2ea2df6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed51: skipped 000131 1775195518.636215 trunc from 0x01ed51 to 0x00ab0d (0x14243 bytes) 000133 1775195518.666812 MIRROR_RESYNC 2 mirrors 000136 1775195519.328843 mapread 0x0092be thru 0x00ab0d (0x1850 bytes) 000137 1775195519.343221 write 0x00b1a7 thru 0x017120 (0xbf7a bytes) 000139 1775195519.430501 mapread 0x00e2ac thru 0x017120 (0x8e75 bytes) 000140 1775195519.433537 mapread 0x0116f3 thru 0x014ffd (0x390b bytes) 000142 1775195519.437261 read 0x0015df thru 0x00cfb5 (0xb9d7 bytes) 000143 1775195519.456033 mapread 0x006315 thru 0x00765c (0x1348 bytes) 000144 1775195519.472494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000145 1775195519.668538 trunc from 0x017121 to 0x00bf40 (0xb1e0 bytes) 000146 1775195519.691205 mapread 0x005c28 thru 0x00bf40 (0x6319 bytes) 000147 1775195519.720483 trunc from 0x00bf41 to 0x022cc0 (0x16d80 bytes) 000149 1775195519.745887 read 0x00d83a thru 0x015c59 (0x8420 bytes) 000150 1775195519.759594 read 0x021e20 thru 0x022cc0 (0xea1 bytes) 000152 1775195519.777692 read 0x01ae02 thru 0x01c96a (0x1b69 bytes) 000154 1775195519.780805 mapread 0x00559f thru 0x00789a (0x22fc bytes) 000157 1775195519.783126 write 0x03321a thru 0x03365f (0x446 bytes) 000158 1775195519.785826 mapread 0x0152b5 thru 0x015548 (0x294 bytes) 000160 1775195519.788526 trunc from 0x033660 to 0x028d61 (0xa8fe bytes) 000162 1775195519.808591 trunc from 0x028d62 to 0x00cea2 (0x1bebf bytes) 000163 1775195519.832358 write 0x0088b0 thru 0x00df2c (0x567d bytes) 000165 1775195519.835852 write 0x02e61a thru 0x0390f8 (0xaadf bytes) 000167 1775195519.838066 trunc from 0x0390f9 to 0x0243e9 (0x14d0f bytes) 000168 1775195519.863788 write 0x00a1f7 thru 0x00fcf1 (0x5afb bytes) 000169 1775195519.877245 mapwrite 0x025f03 thru 0x0275dc (0x16da bytes) 000171 1775195519.984087 mapwrite 0x01d33b thru 0x01f0fb (0x1dc1 bytes) 000172 1775195520.147547 read 0x010c2c thru 0x01c5ac (0xb981 bytes) 000174 1775195520.158099 write 0x0227fe thru 0x026a7c (0x427f bytes) 000175 1775195520.164511 mapwrite 0x0226c0 thru 0x022aa1 (0x3e2 bytes) 000176 1775195520.276369 write 0x029fe6 thru 0x02f7f1 (0x580c bytes) 000177 1775195520.284166 read 0x009c46 thru 0x0142d8 (0xa693 bytes) 000179 1775195520.287342 read 0x01a02a thru 0x01e964 (0x493b bytes) 000180 1775195520.289418 read 0x022686 thru 0x02dfb8 (0xb933 bytes) 000182 1775195520.303308 mapwrite 0x01d5b6 thru 0x02ada0 (0xd7eb bytes) 000184 1775195520.427053 MIRROR_EXTEND to 2 mirrors 000185 1775195521.211675 mapread 0x00bcd0 thru 0x017fc0 (0xc2f1 bytes) 000187 1775195521.270629 trunc from 0x02f7f2 to 0x025a4a (0x9da7 bytes) 000188 1775195521.451795 trunc from 0x025a4b to 0x037fc4 (0x1257a bytes) 000191 1775195521.512229 write 0x004804 thru 0x0097f3 (0x4ff0 bytes) 000192 1775195521.542437 mapwrite 0x00400f thru 0x0132d4 (0xf2c6 bytes) 000193 1775195521.829072 write 0x030a33 thru 0x03a71d (0x9ceb bytes) 000194 1775195521.851802 mapread 0x016910 thru 0x024bd7 (0xe2c8 bytes) 000197 1775195521.895292 read 0x0137e4 thru 0x01de49 (0xa666 bytes) 000198 1775195521.896879 mapread 0x004729 thru 0x00c3e7 (0x7cbf bytes) 000200 1775195521.906123 mapwrite 0x001267 thru 0x00e618 (0xd3b2 bytes) 000201 1775195521.984339 read 0x0151d9 thru 0x017523 (0x234b bytes) 000202 1775195521.990294 mapwrite 0x024a2f thru 0x025a9a (0x106c bytes) 000203 1775195522.058208 write 0x0028f7 thru 0x009178 (0x6882 bytes) 000205 1775195522.063157 read 0x0347b4 thru 0x03a71d (0x5f6a bytes) 000208 1775195522.073147 write 0x0240df thru 0x02e0ae (0x9fd0 bytes) 000209 1775195522.083573 write 0x00ba8b thru 0x01b1d3 (0xf749 bytes) 000210 1775195522.085299 trunc from 0x03a71e to 0x01162d (0x290f0 bytes) 000212 1775195522.128614 mapread 0x010fdb thru 0x01162d (0x653 bytes) 000213 1775195522.129811 trunc from 0x01162e to 0x01bb1a (0xa4ed bytes) 000216 1775195522.175280 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bb1b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb1b): Mirror 2: 0x7e2a242f (stale) CRC-32 checksum value for chunk [0, 0x1bb1b): Mirror 1: 0x81771ab0 Mirror 3: 0x81771ab0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb1b: skipped 000217 1775195523.588115 mapwrite 0x02a1f5 thru 0x02d3d1 (0x31dd bytes) 000219 1775195523.898660 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d3d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d3d2): Mirror 2: 0x1047fe87 (stale) Mirror 3: 0xb613b466 (stale) CRC-32 checksum value for chunk [0, 0x2d3d2): Mirror 1: 0x8cb4b600 Mirror 4: 0x8cb4b600 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d3d2: skipped 000220 1775195525.413135 write 0x035407 thru 0x03ffff (0xabf9 bytes) 000224 1775195525.606719 write 0x014ec5 thru 0x016d28 (0x1e64 bytes) 000225 1775195525.623674 read 0x03aacd thru 0x03ffff (0x5533 bytes) 000226 1775195525.624474 read 0x021114 thru 0x02edf3 (0xdce0 bytes) 000227 1775195525.646202 mapread 0x01748a thru 0x01baea (0x4661 bytes) 000229 1775195525.671763 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x16a92ffd (stale) Mirror 3: 0xd6abbf74 (stale) Mirror 4: 0x19f55954 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd02a69bc Mirror 5: 0xd02a69bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1775195527.433775 MIRROR_RESYNC 5 mirrors 000232 1775195528.544914 trunc from 0x040000 to 0x02b2e3 (0x14d1c bytes) 000235 1775195528.627739 trunc from 0x02b2e4 to 0x0246e5 (0x6bfe bytes) 000237 1775195528.663620 trunc from 0x0246e6 to 0x029408 (0x4d23 bytes) 000239 1775195528.681091 mapwrite 0x02be86 thru 0x03be79 (0xfff4 bytes) 000241 1775195528.789525 mapread 0x00e858 thru 0x01179c (0x2f45 bytes) 000244 1775195528.809736 trunc from 0x03be7a to 0x00f28d (0x2cbec bytes) 000246 1775195528.844772 read 0x000527 thru 0x00c920 (0xc3fa bytes) 000248 1775195528.858848 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf28e): Mirror 2: 0x52405a49 (stale) Mirror 3: 0x52405a49 (stale) Mirror 5: 0x52405a49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf28e): Mirror 2: 0x52405a49 (stale) Mirror 3: 0x52405a49 (stale) Mirror 5: 0x52405a49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf28e: skipped 000249 1775195529.258359 mapwrite 0x01b4c4 thru 0x01cd89 (0x18c6 bytes) 000252 1775195529.406081 mapread 0x016a2c thru 0x01cd89 (0x635e bytes) 000253 1775195529.424631 write 0x039c7e thru 0x03ffff (0x6382 bytes) 000254 1775195529.431226 mapread 0x00b4b4 thru 0x00f875 (0x43c2 bytes) 000256 1775195529.450043 trunc from 0x040000 to 0x00847d (0x37b82 bytes) 000257 1775195529.506114 MIRROR_RESYNC 4 mirrors 000261 1775195530.949445 mapread 0x001042 thru 0x001237 (0x1f6 bytes) 000263 1775195530.970182 mapwrite 0x001c6b thru 0x007e2a (0x61c0 bytes) 000264 1775195531.217078 mapwrite 0x01a160 thru 0x021ea5 (0x7d46 bytes) 000265 1775195531.351835 mapread 0x010a7a thru 0x01b0cd (0xa654 bytes) 000266 1775195531.363866 trunc from 0x021ea6 to 0x025be0 (0x3d3b bytes) 000267 1775195531.387886 mapread 0x00e8b0 thru 0x01a636 (0xbd87 bytes) 000269 1775195531.392530 write 0x03e5c3 thru 0x03ffff (0x1a3d bytes) 000270 1775195531.396097 read 0x0297b8 thru 0x02e549 (0x4d92 bytes) 000271 1775195531.416187 write 0x039d28 thru 0x03ffff (0x62d8 bytes) 000272 1775195531.435082 trunc from 0x040000 to 0x037114 (0x8eeb bytes) 000274 1775195531.486481 read 0x020acd thru 0x020e2c (0x360 bytes) 000275 1775195531.487260 read 0x023298 thru 0x023d5a (0xac3 bytes) 000276 1775195531.514645 mapwrite 0x015eda thru 0x01c358 (0x647f bytes) 000277 1775195531.628848 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37115): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37115): Mirror 2: 0xe927bace (stale) Mirror 3: 0xe927bace (stale) Mirror 5: 0xe927bace (stale) CRC-32 checksum value for chunk [0, 0x37115): Mirror 1: 0xf797fe99 Mirror 6: 0xf797fe99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37115: skipped 000281 1775195532.668120 read 0x009ffd thru 0x00aa2d (0xa31 bytes) 000282 1775195532.682299 write 0x0171e5 thru 0x025911 (0xe72d bytes) 000283 1775195532.804412 mapread 0x020bd8 thru 0x021658 (0xa81 bytes) 000284 1775195532.805446 mapwrite 0x03be47 thru 0x03e1e3 (0x239d bytes) 000285 1775195532.926818 write 0x01ead9 thru 0x02b43b (0xc963 bytes) 000286 1775195532.938687 mapread 0x020a89 thru 0x020d26 (0x29e bytes) 000290 1775195532.942809 mapread 0x02d008 thru 0x036fd3 (0x9fcc bytes) 000291 1775195532.963468 mapwrite 0x01605a thru 0x01efbf (0x8f66 bytes) 000294 1775195533.031129 mapread 0x01db0f thru 0x026218 (0x870a bytes) 000295 1775195533.034761 mapwrite 0x00a80b thru 0x01a058 (0xf84e bytes) 000297 1775195533.121237 trunc from 0x03e1e4 to 0x00a59d (0x33c46 bytes) 000300 1775195533.165430 trunc from 0x00a59e to 0x011059 (0x6abc bytes) truncating to largest ever: 0x3ceaf 000301 1775195533.196317 trunc from 0x01105a to 0x03ceae (0x2be55 bytes) 000304 1775195533.219837 MIRROR_RESYNC 5 mirrors 000305 1775195534.498386 MIRROR_EXTEND to 6 mirrors 000306 1775195535.360660 trunc from 0x03ceaf to 0x022617 (0x1a897 bytes) 000309 1775195535.430339 mapread 0x015b2e thru 0x022339 (0xc80c bytes) 000310 1775195535.470751 mapwrite 0x0181bf thru 0x0199d6 (0x1818 bytes) 000312 1775195535.532572 read 0x01632e thru 0x022617 (0xc2ea bytes) 000315 1775195535.537485 trunc from 0x022618 to 0x00c55e (0x160b9 bytes) 000316 1775195535.572679 read 0x00b3a1 thru 0x00c55e (0x11be bytes) 000317 1775195535.590200 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xc55f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc55f): Mirror 2: 0x32837314 (stale) Mirror 3: 0x32837314 (stale) Mirror 5: 0x32837314 (stale) Mirror 6: 0x32837314 (stale) Mirror 7: 0x32837314 (stale) CRC-32 checksum value for chunk [0, 0xc55f): Mirror 1: 0x32837314 Mirror 8: 0x32837314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc55f: skipped 000318 1775195536.955187 mapread 0x009072 thru 0x00c55e (0x34ed bytes) 000319 1775195536.998103 trunc from 0x00c55f to 0x02547f (0x18f21 bytes) 000320 1775195537.177877 mapwrite 0x01124d thru 0x01759c (0x6350 bytes) 000323 1775195537.338860 trunc from 0x025480 to 0x02b899 (0x641a bytes) 000325 1775195537.405098 trunc from 0x02b89a to 0x02ede4 (0x354b bytes) 000327 1775195537.460956 mapread 0x000f22 thru 0x001dae (0xe8d bytes) 000330 1775195537.479531 trunc from 0x02ede5 to 0x01b7bc (0x13628 bytes) 000331 1775195537.503066 mapwrite 0x01a951 thru 0x027de7 (0xd497 bytes) 000333 1775195537.627537 MIRROR_RESYNC 7 mirrors 000334 1775195539.397477 mapread 0x001603 thru 0x00edf2 (0xd7f0 bytes) 000336 1775195539.427726 write 0x0108b6 thru 0x011ca8 (0x13f3 bytes) 000337 1775195539.495001 trunc from 0x027de8 to 0x00aed9 (0x1cf0e bytes) 000338 1775195539.524352 read 0x006cbc thru 0x00aed9 (0x421e bytes) 000342 1775195539.540112 MIRROR_RESYNC 7 mirrors 000344 1775195541.011249 trunc from 0x00aeda to 0x0079aa (0x352f bytes) 000350 1775195541.110855 mapwrite 0x03d2f1 thru 0x03ffff (0x2d0f bytes) 000351 1775195541.235547 trunc from 0x040000 to 0x01bd77 (0x24288 bytes) 000352 1775195541.265880 read 0x0159c7 thru 0x01aede (0x5518 bytes) 000355 1775195541.284868 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1bd78): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd78): Mirror 2: 0x772ef779 (stale) Mirror 3: 0x772ef779 (stale) Mirror 5: 0x772ef779 (stale) Mirror 6: 0x772ef779 (stale) Mirror 7: 0x772ef779 (stale) Mirror 8: 0x772ef779 (stale) CRC-32 checksum value for chunk [0, 0x1bd78): Mirror 1: 0xd4e5a623 Mirror 9: 0xd4e5a623 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd78: skipped 000357 1775195542.512802 read 0x01a960 thru 0x01bd77 (0x1418 bytes) 000358 1775195542.522831 mapwrite 0x02171f thru 0x02ad00 (0x95e2 bytes) 000360 1775195542.728707 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad01): Mirror 2: 0xa1d2d553 (stale) Mirror 3: 0xa1d2d553 (stale) Mirror 5: 0xa1d2d553 (stale) Mirror 6: 0xa1d2d553 (stale) Mirror 7: 0xa1d2d553 (stale) Mirror 8: 0xa1d2d553 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad01): Mirror 2: 0xa1d2d553 (stale) Mirror 3: 0xa1d2d553 (stale) Mirror 5: 0xa1d2d553 (stale) Mirror 6: 0xa1d2d553 (stale) Mirror 7: 0xa1d2d553 (stale) Mirror 8: 0xa1d2d553 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad01: skipped 000361 1775195543.216233 mapwrite 0x022117 thru 0x026506 (0x43f0 bytes) 000362 1775195543.305455 mapwrite 0x03c88a thru 0x03ffff (0x3776 bytes) 000365 1775195543.390344 mapread 0x03fad9 thru 0x03ffff (0x527 bytes) 000366 1775195543.392639 mapwrite 0x01b48f thru 0x02132c (0x5e9e bytes) 000367 1775195543.469319 read 0x032413 thru 0x03a29d (0x7e8b bytes) 000368 1775195543.515379 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x346f1e6c (stale) Mirror 5: 0x346f1e6c (stale) Mirror 6: 0x346f1e6c (stale) Mirror 7: 0x346f1e6c (stale) Mirror 8: 0x346f1e6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x346f1e6c (stale) Mirror 5: 0x346f1e6c (stale) Mirror 6: 0x346f1e6c (stale) Mirror 7: 0x346f1e6c (stale) Mirror 8: 0x346f1e6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1775195544.196707 mapread 0x028889 thru 0x0335c8 (0xad40 bytes) 000370 1775195544.252708 read 0x031544 thru 0x03968b (0x8148 bytes) 000376 1775195544.265466 read 0x01b76e thru 0x025a70 (0xa303 bytes) 000377 1775195544.283441 write 0x02d1dc thru 0x02efab (0x1dd0 bytes) 000378 1775195544.284826 read 0x036bb9 thru 0x03ffff (0x9447 bytes) 000380 1775195544.292056 read 0x02e0a2 thru 0x031125 (0x3084 bytes) 000381 1775195544.294680 write 0x02a054 thru 0x032db6 (0x8d63 bytes) 000382 1775195544.297984 write 0x02ba70 thru 0x02bdb3 (0x344 bytes) 000383 1775195544.300514 read 0x003f67 thru 0x00d075 (0x910f bytes) 000385 1775195544.303644 mapwrite 0x00ef7d thru 0x01d7e6 (0xe86a bytes) 000388 1775195544.382311 mapwrite 0x020c38 thru 0x02b3dc (0xa7a5 bytes) 000390 1775195544.442315 read 0x0194c8 thru 0x01a8b5 (0x13ee bytes) 000391 1775195544.443370 mapread 0x00af26 thru 0x01a4a4 (0xf57f bytes) 000395 1775195544.448630 trunc from 0x040000 to 0x023781 (0x1c87e bytes) 000396 1775195544.471548 mapread 0x01d238 thru 0x023781 (0x654a bytes) 000397 1775195544.473738 write 0x00bd3b thru 0x00c0bb (0x381 bytes) 000400 1775195544.483353 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23782): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23782): Mirror 3: 0x4480dbc3 (stale) Mirror 5: 0x4480dbc3 (stale) Mirror 6: 0x4480dbc3 (stale) Mirror 7: 0x4480dbc3 (stale) Mirror 8: 0x4480dbc3 (stale) CRC-32 checksum value for chunk [0, 0x23782): Mirror 1: 0x96866d45 Mirror 9: 0x96866d45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23782: skipped 000401 1775195545.715133 write 0x03abf8 thru 0x03b46d (0x876 bytes) 000402 1775195545.850353 mapwrite 0x021a80 thru 0x02c35f (0xa8e0 bytes) 000404 1775195546.138870 mapwrite 0x0372c3 thru 0x03ffff (0x8d3d bytes) 000405 1775195546.347076 read 0x03a4b3 thru 0x03ffff (0x5b4d bytes) 000406 1775195546.347759 mapread 0x03f67f thru 0x03ffff (0x981 bytes) 000408 1775195546.357325 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x346f1e6c (stale) Mirror 6: 0x346f1e6c (stale) Mirror 7: 0x346f1e6c (stale) Mirror 8: 0x346f1e6c (stale) Mirror 9: 0xc5394f3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x346f1e6c (stale) Mirror 6: 0x346f1e6c (stale) Mirror 7: 0x346f1e6c (stale) Mirror 8: 0x346f1e6c (stale) Mirror 9: 0xc5394f3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1775195546.907312 mapread 0x02325a thru 0x02dbd2 (0xa979 bytes) 000411 1775195546.945454 trunc from 0x040000 to 0x03bca0 (0x435f bytes) 000413 1775195546.974450 read 0x039180 thru 0x03bca0 (0x2b21 bytes) 000414 1775195546.986701 read 0x038988 thru 0x03b19e (0x2817 bytes) 000415 1775195547.009510 mapread 0x007d9d thru 0x00d359 (0x55bd bytes) 000419 1775195547.020866 mapwrite 0x031feb thru 0x03ffff (0xe015 bytes) 000420 1775195547.139536 trunc from 0x040000 to 0x0038bc (0x3c743 bytes) 000421 1775195547.168497 read 0x00053f thru 0x0038bc (0x337e bytes) 000422 1775195547.170541 read 0x001ca2 thru 0x0038bc (0x1c1b bytes) 000423 1775195547.172183 trunc from 0x0038bd to 0x01be9e (0x185e2 bytes) 000425 1775195547.201873 write 0x006f41 thru 0x01679e (0xf85e bytes) 000432 1775195547.231131 read 0x002260 thru 0x00842d (0x61ce bytes) 000433 1775195547.231884 mapread 0x0159d1 thru 0x01be9e (0x64ce bytes) 000434 1775195547.240823 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1be9f): Mirror 3: 0xf1d4e5b6 (stale) Mirror 6: 0xf1d4e5b6 (stale) Mirror 7: 0xf1d4e5b6 (stale) Mirror 8: 0xf1d4e5b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1be9f): Mirror 3: 0xf1d4e5b6 (stale) Mirror 6: 0xf1d4e5b6 (stale) Mirror 7: 0xf1d4e5b6 (stale) Mirror 8: 0xf1d4e5b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1be9f: skipped 000435 1775195547.747154 trunc from 0x01be9f to 0x01de2d (0x1f8f bytes) 000436 1775195547.770602 write 0x02c710 thru 0x035c3e (0x952f bytes) 000437 1775195547.772615 read 0x024e34 thru 0x02a708 (0x58d5 bytes) 000439 1775195547.782035 write 0x0081f7 thru 0x01117f (0x8f89 bytes) 000441 1775195547.799494 write 0x02df0d thru 0x02e4c5 (0x5b9 bytes) 000442 1775195547.800816 trunc from 0x035c3f to 0x00df30 (0x27d0e bytes) 000443 1775195547.841210 write 0x00e9ac thru 0x01733a (0x898f bytes) 000445 1775195547.845282 trunc from 0x01733b to 0x033449 (0x1c10f bytes) 000446 1775195547.877284 read 0x014f86 thru 0x01d20f (0x828a bytes) 000448 1775195547.899714 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3344a): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3344a): Mirror 3: 0x47a8bcd7 (stale) Mirror 6: 0x47a8bcd7 (stale) Mirror 7: 0x47a8bcd7 (stale) Mirror 8: 0x47a8bcd7 (stale) CRC-32 checksum value for chunk [0, 0x3344a): Mirror 1: 0x7adcc2cb Mirror 9: 0x7adcc2cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3344a: skipped 000451 1775195549.159015 write 0x01d346 thru 0x01f1db (0x1e96 bytes) 000452 1775195549.307520 read 0x0130c9 thru 0x01548b (0x23c3 bytes) 000453 1775195549.313847 read 0x028e00 thru 0x02d6b2 (0x48b3 bytes) 000455 1775195549.333314 mapread 0x030866 thru 0x033449 (0x2be4 bytes) 000456 1775195549.335206 trunc from 0x03344a to 0x01eb2b (0x1491e bytes) 000457 1775195549.364589 write 0x019867 thru 0x01c966 (0x3100 bytes) 000459 1775195549.368519 read 0x00d5f7 thru 0x013307 (0x5d11 bytes) 000463 1775195549.371331 trunc from 0x01eb2c to 0x01d612 (0x1519 bytes) 000464 1775195549.398121 write 0x024b45 thru 0x0338e2 (0xed9e bytes) 000466 1775195549.401591 mapwrite 0x03acb3 thru 0x03b76c (0xaba bytes) 000467 1775195549.497271 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b76d): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b76d): Mirror 3: 0x5f4d5332 (stale) Mirror 6: 0x5f4d5332 (stale) Mirror 7: 0x5f4d5332 (stale) Mirror 8: 0x5f4d5332 (stale) Mirror 9: 0x6d8b80fa (stale) CRC-32 checksum value for chunk [0, 0x3b76d): Mirror 1: 0x684e7ea9 Mirror 10: 0x684e7ea9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b76d: skipped 000469 1775195551.137422 write 0x01a6cf thru 0x028801 (0xe133 bytes) 000470 1775195551.235429 write 0x02c6fa thru 0x035798 (0x909f bytes) 000472 1775195551.254091 mapread 0x024f3d thru 0x02fc79 (0xad3d bytes) 000473 1775195551.255733 write 0x01aa05 thru 0x02954b (0xeb47 bytes) 000474 1775195551.258504 mapread 0x00a491 thru 0x00abd9 (0x749 bytes) 000475 1775195551.270314 trunc from 0x03b76d to 0x0224a7 (0x192c5 bytes) 000476 1775195551.292317 write 0x03fd8c thru 0x03ffff (0x274 bytes) 000477 1775195551.293447 trunc from 0x040000 to 0x03b991 (0x466e bytes) 000479 1775195551.310298 mapread 0x01918b thru 0x01d319 (0x418f bytes) 000480 1775195551.326211 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 8: 0xfc6008c0 (stale) Mirror 9: 0x321b9ffd (stale) Mirror 10: 0xbaf1013d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 8: 0xfc6008c0 (stale) Mirror 9: 0x321b9ffd (stale) Mirror 10: 0xbaf1013d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b992: skipped 000483 1775195551.822729 write 0x005108 thru 0x0077cb (0x26c4 bytes) 000484 1775195551.836349 write 0x01ff42 thru 0x02268b (0x274a bytes) 000485 1775195551.858684 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 8: 0xfc6008c0 (stale) Mirror 9: 0x321b9ffd (stale) Mirror 10: 0xbaf1013d (stale) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 1: 0x5c838e52 Mirror 11: 0x5c838e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b992: skipped 000486 1775195552.920759 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 8: 0xfc6008c0 (stale) Mirror 10: 0xbaf1013d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 8: 0xfc6008c0 (stale) Mirror 10: 0xbaf1013d (stale) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 1: 0x5c838e52 Mirror 11: 0x5c838e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b992: skipped 000490 1775195553.895318 mapread 0x027ef2 thru 0x030086 (0x8195 bytes) 000494 1775195554.119548 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 10: 0xbaf1013d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b992): Mirror 6: 0xfc6008c0 (stale) Mirror 7: 0xfc6008c0 (stale) Mirror 10: 0xbaf1013d (stale) CRC-32 checksum value for chunk [0, 0x3b992): Mirror 1: 0x5c838e52 Mirror 11: 0x5c838e52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b992: skipped 000497 1775195555.019261 mapread 0x00cf3d thru 0x019101 (0xc1c5 bytes) 000498 1775195555.082659 read 0x00df76 thru 0x01440e (0x6499 bytes) 000499 1775195555.086743 mapread 0x01c91b thru 0x01ce6b (0x551 bytes) 000500 1775195555.096698 trunc from 0x03b992 to 0x029446 (0x1254b bytes) 000501 1775195555.225344 trunc from 0x029447 to 0x0151a1 (0x142a5 bytes) 000506 1775195555.249709 mapwrite 0x000935 thru 0x000a52 (0x11e bytes) 000509 1775195555.362103 mapwrite 0x00f80a thru 0x01b696 (0xbe8d bytes) 000514 1775195555.472369 mapwrite 0x039f73 thru 0x03ffff (0x608d bytes) 000515 1775195555.558280 mapread 0x005d34 thru 0x008c21 (0x2eee bytes) 000516 1775195555.563613 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x346f1e6c (stale) Mirror 7: 0x346f1e6c (stale) Mirror 10: 0x247de66c (stale) Mirror 11: 0x20d77dbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x47cf87d6 Mirror 12: 0x47cf87d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000518 1775195556.636859 mapread 0x01cf4f thru 0x02cb65 (0xfc17 bytes) 000519 1775195556.720879 write 0x02476e thru 0x02d40a (0x8c9d bytes) 000520 1775195556.828702 read 0x016f02 thru 0x017a9e (0xb9d bytes) 000521 1775195556.836702 trunc from 0x040000 to 0x0153e1 (0x2ac1e bytes) 000522 1775195556.892766 mapwrite 0x004830 thru 0x00a7d8 (0x5fa9 bytes) 000523 1775195556.972989 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x153e2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153e2): Mirror 6: 0xd78a5d6d (stale) Mirror 7: 0xd78a5d6d (stale) Mirror 10: 0xe9c4ac8d (stale) Mirror 11: 0x35af7432 (stale) Mirror 12: 0x61ea4474 (stale) CRC-32 checksum value for chunk [0, 0x153e2): Mirror 1: 0x10911d13 Mirror 13: 0x10911d13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153e2: skipped 000524 1775195558.096330 write 0x028537 thru 0x0346fd (0xc1c7 bytes) 000525 1775195558.253854 mapwrite 0x01b34e thru 0x027b2c (0xc7df bytes) 000526 1775195558.524993 mapread 0x00b435 thru 0x0140ce (0x8c9a bytes) 000528 1775195558.570187 trunc from 0x0346fe to 0x0277cf (0xcf2e bytes) 000530 1775195558.633548 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x277d0): Mirror 6: 0x21007ba2 (stale) Mirror 10: 0xdaca8c72 (stale) Mirror 11: 0xb4761a30 (stale) Mirror 12: 0xdc16fc7f (stale) Mirror 13: 0x639d7c5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x277d0): Mirror 6: 0x21007ba2 (stale) Mirror 10: 0xdaca8c72 (stale) Mirror 11: 0xb4761a30 (stale) Mirror 12: 0xdc16fc7f (stale) Mirror 13: 0x639d7c5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x277d0: skipped 000532 1775195559.304662 mapread 0x001a51 thru 0x00b874 (0x9e24 bytes) 000533 1775195559.332313 trunc from 0x0277d0 to 0x0363d7 (0xec08 bytes) 000536 1775195559.359602 trunc from 0x0363d8 to 0x0095a8 (0x2ce2f bytes) 000537 1775195559.389056 trunc from 0x0095a9 to 0x015790 (0xc1e8 bytes) 000538 1775195559.420090 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: 0xa2a87ead (stale) Mirror 11: 0x455b87c7 (stale) Mirror 12: 0x997eea27 (stale) Mirror 13: 0x22be2abd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: 0xa2a87ead (stale) Mirror 11: 0x455b87c7 (stale) Mirror 12: 0x997eea27 (stale) Mirror 13: 0x22be2abd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15791: skipped 000540 1775195560.078862 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: 0xa2a87ead (stale) Mirror 11: 0x455b87c7 (stale) Mirror 12: 0x997eea27 (stale) Mirror 13: 0x22be2abd (stale) CRC-32 checksum value for chunk [0, 0x15791): Mirror 1: 0x2a0a0f8f Mirror 14: 0x2a0a0f8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15791: skipped 000541 1775195561.823668 write 0x006a6a thru 0x013285 (0xc81c bytes) 000542 1775195561.964057 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: 0xa2a87ead (stale) Mirror 11: 0x455b87c7 (stale) Mirror 13: 0x22be2abd (stale) Mirror 14: 0x2a0a0f8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15791): Mirror 10: 0xa2a87ead (stale) Mirror 11: 0x455b87c7 (stale) Mirror 13: 0x22be2abd (stale) Mirror 14: 0x2a0a0f8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15791: skipped 000544 1775195562.482185 mapwrite 0x03f0fd thru 0x03ffff (0xf03 bytes) 000547 1775195562.573657 mapread 0x03efaf thru 0x03ffff (0x1051 bytes) 000549 1775195562.583163 mapread 0x007e56 thru 0x014384 (0xc52f bytes) 000551 1775195562.611910 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x247de66c (stale) Mirror 13: 0xc8c5201a (stale) Mirror 14: 0x7db5dd0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x247de66c (stale) Mirror 13: 0xc8c5201a (stale) Mirror 14: 0x7db5dd0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000552 1775195563.072150 mapwrite 0x0007b4 thru 0x00e2c5 (0xdb12 bytes) 000557 1775195563.206511 write 0x01bd59 thru 0x01c2d9 (0x581 bytes) 000560 1775195563.221559 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x247de66c (stale) Mirror 14: 0x7db5dd0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x247de66c (stale) Mirror 14: 0x7db5dd0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000561 1775195563.760439 mapwrite 0x029d9f thru 0x038f3b (0xf19d bytes) 000562 1775195563.966519 read 0x022367 thru 0x02e749 (0xc3e3 bytes) 000563 1775195563.979136 read 0x011719 thru 0x011bd4 (0x4bc bytes) 000564 1775195564.054398 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x247de66c (stale) Mirror 14: 0x7db5dd0e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ab173cc Mirror 15: 0x1ab173cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000565 1775195565.561011 mapwrite 0x006bcd thru 0x00a942 (0x3d76 bytes) 000566 1775195565.868342 write 0x023b91 thru 0x0333e0 (0xf850 bytes) 000568 1775195565.935801 read 0x03cee0 thru 0x03ffff (0x3120 bytes) 000569 1775195565.965964 trunc from 0x040000 to 0x01f7e4 (0x2081b bytes) 000572 1775195566.011678 read 0x014327 thru 0x0195c8 (0x52a2 bytes) 000573 1775195566.036882 mapread 0x00c599 thru 0x01a953 (0xe3bb bytes) 000574 1775195566.038253 mapread 0x0151cc thru 0x01ee66 (0x9c9b bytes) 000577 1775195566.054322 trunc from 0x01f7e5 to 0x02de00 (0xe61c bytes) 000578 1775195566.095678 read 0x007ca6 thru 0x00e92d (0x6c88 bytes) 000581 1775195566.103162 mapread 0x0136f4 thru 0x0195c1 (0x5ece bytes) 000584 1775195566.149596 MIRROR_RESYNC 4 mirrors 000586 1775195567.245301 read 0x02c552 thru 0x02c783 (0x232 bytes) 000587 1775195567.265108 MIRROR_EXTEND to 5 mirrors 000588 1775195568.485244 mapread 0x00c935 thru 0x019937 (0xd003 bytes) 000589 1775195568.589064 trunc from 0x02de01 to 0x003d2a (0x2a0d6 bytes) 000592 1775195568.852881 write 0x02de9d thru 0x03b9d4 (0xdb38 bytes) 000593 1775195568.857686 mapread 0x032143 thru 0x032ec6 (0xd84 bytes) 000594 1775195568.861550 mapread 0x021179 thru 0x02eba9 (0xda31 bytes) 000595 1775195568.877348 mapread 0x037e1c thru 0x03b9d4 (0x3bb9 bytes) 000597 1775195568.886881 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b9d5): Mirror 14: 0x77cf0926 (stale) Mirror 15: 0x77cf0926 (stale) Mirror 16: 0x77cf0926 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9d5): Mirror 14: 0x77cf0926 (stale) Mirror 15: 0x77cf0926 (stale) Mirror 16: 0x77cf0926 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9d5: skipped 000598 1775195569.634667 trunc from 0x03b9d5 to 0x03a197 (0x183d bytes) 000599 1775195569.705989 read 0x017aac thru 0x01d946 (0x5e9b bytes) 000601 1775195569.720537 trunc from 0x03a198 to 0x00eaac (0x2b6eb bytes) 000604 1775195569.757569 read 0x00e7fd thru 0x00eaac (0x2b0 bytes) 000605 1775195569.774008 write 0x028add thru 0x038035 (0xf559 bytes) 000606 1775195569.778235 write 0x01aeb0 thru 0x01ff07 (0x5058 bytes) 000609 1775195569.797860 read 0x002d05 thru 0x0058ae (0x2baa bytes) 000610 1775195569.800873 mapwrite 0x022dd8 thru 0x02b216 (0x843f bytes) 000611 1775195569.927603 mapwrite 0x01e1af thru 0x02d2a0 (0xf0f2 bytes) 000612 1775195570.024839 MIRROR_RESYNC 4 mirrors 000614 1775195571.470142 read 0x01d38f thru 0x021d1c (0x498e bytes) 000616 1775195571.480138 read 0x00f59f thru 0x014237 (0x4c99 bytes) 000617 1775195571.504624 mapwrite 0x007644 thru 0x014c25 (0xd5e2 bytes) 000618 1775195571.876108 trunc from 0x038036 to 0x022d3e (0x152f7 bytes) 000621 1775195571.904997 mapread 0x00dc04 thru 0x0167c0 (0x8bbd bytes) 000622 1775195571.923738 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22d3f): Mirror 15: 0xbbea8442 (stale) Mirror 16: 0xbbea8442 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22d3f): Mirror 15: 0xbbea8442 (stale) Mirror 16: 0xbbea8442 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22d3f: skipped 000624 1775195572.411879 mapread 0x0006b3 thru 0x00d810 (0xd15e bytes) 000626 1775195572.452730 read 0x019571 thru 0x0224ea (0x8f7a bytes) 000628 1775195572.462557 mapwrite 0x00b96b thru 0x0117f6 (0x5e8c bytes) 000629 1775195572.534355 trunc from 0x022d3f to 0x031293 (0xe555 bytes) 000630 1775195572.570910 mapread 0x008314 thru 0x0105f6 (0x82e3 bytes) 000633 1775195572.578143 mapread 0x01da40 thru 0x02c26c (0xe82d bytes) 000635 1775195572.611191 read 0x012898 thru 0x01648e (0x3bf7 bytes) 000637 1775195572.613310 read 0x01cf22 thru 0x02652e (0x960d bytes) 000641 1775195572.616043 write 0x01db5d thru 0x023369 (0x580d bytes) 000644 1775195572.620769 trunc from 0x031294 to 0x032c32 (0x199f bytes) 000645 1775195572.647796 trunc from 0x032c33 to 0x03557f (0x294d bytes) 000646 1775195572.669769 write 0x03c2b1 thru 0x03e3f0 (0x2140 bytes) 000648 1775195572.678224 MIRROR_RESYNC 3 mirrors 000649 1775195573.577760 trunc from 0x03e3f1 to 0x01b1c0 (0x23230 bytes) 000653 1775195573.741368 mapread 0x00c9cc thru 0x00cd5e (0x393 bytes) 000654 1775195573.771548 MIRROR_RESYNC 3 mirrors 000655 1775195574.393366 mapwrite 0x00eb38 thru 0x01198f (0x2e58 bytes) 000657 1775195574.598764 read 0x00b992 thru 0x019699 (0xdd08 bytes) 000663 1775195574.611156 trunc from 0x01b1c1 to 0x036b25 (0x1b965 bytes) 000665 1775195574.640988 mapwrite 0x02e3e5 thru 0x039867 (0xb483 bytes) 000667 1775195574.736396 write 0x0343bd thru 0x03d323 (0x8f67 bytes) 000668 1775195574.745377 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d324): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d324): Mirror 15: 0x224a1c2f (stale) Mirror 16: 0x224a1c2f (stale) CRC-32 checksum value for chunk [0, 0x3d324): Mirror 1: 0xb73cf24c Mirror 17: 0xb73cf24c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d324: skipped 000670 1775195575.916073 trunc from 0x03d324 to 0x01bb76 (0x217ad bytes) 000671 1775195576.063413 trunc from 0x01bb77 to 0x03af98 (0x1f422 bytes) 000673 1775195576.103431 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af99): Mirror 16: 0xfe808732 (stale) Mirror 17: 0x1f5f15db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af99): Mirror 16: 0xfe808732 (stale) Mirror 17: 0x1f5f15db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af99: skipped 000674 1775195576.483515 trunc from 0x03af99 to 0x0290e7 (0x11eb1 bytes) 000678 1775195576.506223 mapwrite 0x010702 thru 0x016c20 (0x651f bytes) 000680 1775195576.577891 mapread 0x003e2a thru 0x007611 (0x37e8 bytes) 000681 1775195576.592755 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x290e8): Mirror 16: 0x18d8e69c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290e8): Mirror 16: 0x18d8e69c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290e8: skipped 000686 1775195577.036195 mapread 0x019114 thru 0x021cbe (0x8bab bytes) 000687 1775195577.082695 trunc from 0x0290e8 to 0x03bfdd (0x12ef6 bytes) 000689 1775195577.130183 read 0x01b841 thru 0x027377 (0xbb37 bytes) 000695 1775195577.156449 write 0x03a471 thru 0x03ab2a (0x6ba bytes) 000697 1775195577.228798 mapread 0x012acc thru 0x02086a (0xdd9f bytes) 000698 1775195577.269966 mapread 0x000a44 thru 0x003518 (0x2ad5 bytes) 000699 1775195577.310057 read 0x017b64 thru 0x01e711 (0x6bae bytes) 000700 1775195577.316242 trunc from 0x03bfde to 0x00d7c2 (0x2e81b bytes) 000701 1775195577.396110 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd7c3): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd7c3): Mirror 16: 0x96e976f9 (stale) CRC-32 checksum value for chunk [0, 0xd7c3): Mirror 1: 0x96e976f9 Mirror 17: 0x96e976f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd7c3: skipped 000704 1775195578.659164 write 0x00c6cf thru 0x0154d3 (0x8e05 bytes) 000709 1775195578.786184 write 0x037b17 thru 0x03ffff (0x84e9 bytes) 000711 1775195578.790445 read 0x016368 thru 0x01dde9 (0x7a82 bytes) 000713 1775195578.808017 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x3b53602d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x3b53602d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000716 1775195579.182323 read 0x03a892 thru 0x03bfe7 (0x1756 bytes) 000719 1775195579.194279 trunc from 0x040000 to 0x0140c1 (0x2bf3e bytes) 000720 1775195579.287760 mapwrite 0x039716 thru 0x03ffff (0x68ea bytes) 000721 1775195579.495222 read 0x02e15a thru 0x03de33 (0xfcda bytes) 000722 1775195579.531095 mapread 0x03570b thru 0x0375aa (0x1ea0 bytes) 000723 1775195579.542245 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000724 1775195579.805890 trunc from 0x040000 to 0x01e9db (0x21624 bytes) 000726 1775195579.841911 trunc from 0x01e9dc to 0x01f851 (0xe76 bytes) 000729 1775195579.875802 mapwrite 0x03ae95 thru 0x03ffff (0x516b bytes) 000730 1775195580.058213 mapwrite 0x01114b thru 0x01b2a4 (0xa15a bytes) 000731 1775195580.161226 mapread 0x0228cd thru 0x030751 (0xde85 bytes) 000733 1775195580.190312 write 0x02b1f2 thru 0x032208 (0x7017 bytes) 000734 1775195580.197836 read 0x00c730 thru 0x00cd4b (0x61c bytes) 000735 1775195580.200281 read 0x018f46 thru 0x022628 (0x96e3 bytes) 000736 1775195580.201921 write 0x03ca1f thru 0x03ffff (0x35e1 bytes) 000737 1775195580.203306 write 0x014760 thru 0x01bae5 (0x7386 bytes) 000738 1775195580.208914 read 0x01a066 thru 0x022527 (0x84c2 bytes) 000739 1775195580.210176 mapread 0x00cd8b thru 0x010fc8 (0x423e bytes) 000740 1775195580.216004 mapwrite 0x024086 thru 0x0338a9 (0xf824 bytes) 000742 1775195580.288043 MIRROR_EXTEND to 2 mirrors 000744 1775195580.993436 trunc from 0x040000 to 0x01acd5 (0x2532a bytes) 000746 1775195581.152077 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1acd6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1acd6): Mirror 2: 0xc211e75d (stale) CRC-32 checksum value for chunk [0, 0x1acd6): Mirror 1: 0xc211e75d Mirror 3: 0xc211e75d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1acd6: skipped 000748 1775195582.655809 read 0x0127e2 thru 0x01acd5 (0x84f4 bytes) 000749 1775195582.667867 mapwrite 0x02bd84 thru 0x031192 (0x540f bytes) 000753 1775195582.979511 mapread 0x003331 thru 0x004034 (0xd04 bytes) 000754 1775195583.002767 mapread 0x013a9a thru 0x01b8c5 (0x7e2c bytes) 000756 1775195583.015776 trunc from 0x031193 to 0x00c629 (0x24b69 bytes) 000757 1775195583.064211 mapwrite 0x028699 thru 0x028b4d (0x4b5 bytes) 000758 1775195583.256567 MIRROR_RESYNC 3 mirrors 000759 1775195584.950993 read 0x0166bc thru 0x019600 (0x2f45 bytes) 000760 1775195584.958341 mapread 0x01d19a thru 0x0213e2 (0x4249 bytes) 000761 1775195584.989042 mapread 0x01d191 thru 0x0220c7 (0x4f37 bytes) 000762 1775195584.990335 write 0x0101f9 thru 0x0109fe (0x806 bytes) 000763 1775195585.207859 trunc from 0x028b4e to 0x00162a (0x27523 bytes) 000766 1775195585.271757 trunc from 0x00162b to 0x001573 (0xb7 bytes) 000767 1775195585.306620 trunc from 0x001574 to 0x019547 (0x17fd4 bytes) 000769 1775195585.384200 trunc from 0x019548 to 0x0101e2 (0x9365 bytes) 000770 1775195585.433619 trunc from 0x0101e3 to 0x00e927 (0x18bb bytes) 000771 1775195585.492639 read 0x00cd81 thru 0x00e927 (0x1ba7 bytes) 000772 1775195585.511322 mapread 0x00aaa0 thru 0x00bb58 (0x10b9 bytes) 000773 1775195585.512528 mapwrite 0x01af79 thru 0x02a666 (0xf6ee bytes) 000774 1775195585.859213 write 0x00b1f3 thru 0x00b3cb (0x1d9 bytes) 000775 1775195585.860445 write 0x019f56 thru 0x029d6d (0xfe18 bytes) 000776 1775195585.864630 trunc from 0x02a667 to 0x014988 (0x15cde bytes) 000777 1775195585.916146 read 0x01004d thru 0x013d0d (0x3cc1 bytes) 000778 1775195585.916996 trunc from 0x014989 to 0x03a8c5 (0x25f3d bytes) 000779 1775195585.941573 trunc from 0x03a8c6 to 0x023f1e (0x169a7 bytes) 000780 1775195585.959910 read 0x007ad2 thru 0x00ca8b (0x4fba bytes) 000781 1775195585.961264 write 0x037327 thru 0x03e5b3 (0x728d bytes) 000782 1775195585.968111 MIRROR_RESYNC 3 mirrors 000784 1775195586.930600 mapwrite 0x00f654 thru 0x01945e (0x9e0b bytes) 000786 1775195587.117840 mapwrite 0x006555 thru 0x014a4c (0xe4f8 bytes) 000787 1775195587.239127 trunc from 0x03e5b4 to 0x03cacf (0x1ae4 bytes) 000788 1775195587.293910 read 0x01b38e thru 0x021080 (0x5cf3 bytes) 000790 1775195587.295434 trunc from 0x03cad0 to 0x030547 (0xc588 bytes) 000791 1775195587.318338 trunc from 0x030548 to 0x02017b (0x103cc bytes) 000792 1775195587.359092 mapwrite 0x00106e thru 0x004a96 (0x3a29 bytes) 000795 1775195587.450431 write 0x0184a9 thru 0x023544 (0xb09c bytes) 000796 1775195587.455735 mapread 0x01e301 thru 0x021be9 (0x38e9 bytes) 000798 1775195587.459016 write 0x01fc2d thru 0x02327d (0x3651 bytes) 000799 1775195587.460470 read 0x00bdce thru 0x01b409 (0xf63c bytes) 000800 1775195587.461259 trunc from 0x023545 to 0x009a49 (0x19afb bytes) 000802 1775195587.510572 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x9a4a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a4a): Mirror 2: 0x468a83a8 (stale) Mirror 3: 0x468a83a8 (stale) CRC-32 checksum value for chunk [0, 0x9a4a): Mirror 1: 0x3d304261 Mirror 4: 0x3d304261 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a4a: skipped 000806 1775195589.015820 read 0x0058b4 thru 0x009a49 (0x4196 bytes) 000808 1775195589.059235 read 0x0031f7 thru 0x007081 (0x3e8b bytes) 000811 1775195589.100475 trunc from 0x009a4a to 0x008f25 (0xb24 bytes) 000812 1775195589.186550 mapwrite 0x005089 thru 0x0134c2 (0xe43a bytes) 000813 1775195589.363286 trunc from 0x0134c3 to 0x02fcec (0x1c82a bytes) 000814 1775195589.402675 write 0x00fd22 thru 0x010cab (0xf8a bytes) 000815 1775195589.408121 write 0x018343 thru 0x01a71a (0x23d8 bytes) 000817 1775195589.437845 mapread 0x026013 thru 0x02faee (0x9adc bytes) 000818 1775195589.445893 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fced): Mirror 3: 0x43fb28ea (stale) Mirror 4: 0x7fbfce6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fced): Mirror 3: 0x43fb28ea (stale) Mirror 4: 0x7fbfce6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fced: skipped 000819 1775195589.850278 read 0x02a285 thru 0x02fcec (0x5a68 bytes) 000821 1775195589.865265 write 0x03cca6 thru 0x03ffff (0x335a bytes) 000822 1775195589.867646 read 0x02794a thru 0x02c381 (0x4a38 bytes) 000823 1775195589.905865 trunc from 0x040000 to 0x039568 (0x6a97 bytes) 000825 1775195589.929291 write 0x03b527 thru 0x03ffff (0x4ad9 bytes) 000826 1775195589.932687 read 0x00a3c9 thru 0x00e2cc (0x3f04 bytes) 000827 1775195589.933837 read 0x0362a0 thru 0x03ffff (0x9d60 bytes) 000831 1775195589.947625 write 0x03c38b thru 0x03ffff (0x3c75 bytes) 000832 1775195589.949055 write 0x0153e5 thru 0x01d521 (0x813d bytes) 000837 1775195589.957986 trunc from 0x040000 to 0x01b4ab (0x24b54 bytes) 000839 1775195589.984614 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b4ac): Mirror 4: 0xe559f64e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b4ac): Mirror 4: 0xe559f64e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b4ac: skipped 000840 1775195590.362108 mapread 0x013e91 thru 0x01b4ab (0x761b bytes) 000841 1775195590.393712 read 0x003be5 thru 0x007299 (0x36b5 bytes) 000844 1775195590.410989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000845 1775195590.590884 write 0x02023a thru 0x023212 (0x2fd9 bytes) 000846 1775195590.592992 trunc from 0x023213 to 0x01ca4e (0x67c4 bytes) 000847 1775195590.618550 mapread 0x00d294 thru 0x016d32 (0x9a9f bytes) 000849 1775195590.654284 write 0x006e21 thru 0x012ff1 (0xc1d1 bytes) 000851 1775195590.678390 mapread 0x013f06 thru 0x019ab5 (0x5bb0 bytes) 000853 1775195590.700753 write 0x0321d2 thru 0x035c4d (0x3a7c bytes) 000855 1775195590.709611 trunc from 0x035c4e to 0x01a572 (0x1b6db bytes) 000857 1775195590.734820 mapread 0x00f6d3 thru 0x01319b (0x3ac9 bytes) 000858 1775195590.737874 read 0x00cb46 thru 0x0129ba (0x5e75 bytes) 000861 1775195590.743648 mapwrite 0x03f914 thru 0x03ffff (0x6ec bytes) 000863 1775195590.882031 read 0x015fd7 thru 0x01bc6a (0x5c94 bytes) 000864 1775195590.904338 read 0x023d40 thru 0x03288d (0xeb4e bytes) 000869 1775195590.911473 read 0x016aa5 thru 0x019f84 (0x34e0 bytes) 000870 1775195590.919805 MIRROR_EXTEND to 2 mirrors 000871 1775195591.788138 MIRROR_EXTEND to 3 mirrors 000872 1775195592.857598 trunc from 0x040000 to 0x01c348 (0x23cb7 bytes) 000873 1775195592.976449 mapwrite 0x011f97 thru 0x01cf18 (0xaf82 bytes) 000875 1775195593.084430 trunc from 0x01cf19 to 0x03c57d (0x1f665 bytes) 000877 1775195593.104864 mapread 0x002cad thru 0x00843d (0x5791 bytes) 000878 1775195593.114694 write 0x0009ce thru 0x003000 (0x2633 bytes) 000880 1775195593.128053 MIRROR_RESYNC 3 mirrors 000881 1775195594.230204 mapread 0x025c83 thru 0x02934d (0x36cb bytes) 000882 1775195594.263639 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000884 1775195594.812631 read 0x007eee thru 0x00e4e3 (0x65f6 bytes) 000885 1775195594.826221 mapwrite 0x01b2ac thru 0x02aa07 (0xf75c bytes) 000886 1775195595.274480 write 0x034f3a thru 0x03ffff (0xb0c6 bytes) 000888 1775195595.294057 read 0x031198 thru 0x03ffff (0xee68 bytes) 000892 1775195595.305114 mapwrite 0x031a52 thru 0x03c2f3 (0xa8a2 bytes) 000893 1775195595.397996 read 0x0246d2 thru 0x0303bb (0xbcea bytes) 000894 1775195595.407470 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4803ae6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x428e5639 Mirror 3: 0x428e5639 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000895 1775195597.029595 mapwrite 0x01a6ed thru 0x0219ab (0x72bf bytes) 000896 1775195597.259873 write 0x03f2cb thru 0x03ffff (0xd35 bytes) 000899 1775195597.285296 read 0x025992 thru 0x0285d6 (0x2c45 bytes) 000900 1775195597.298717 mapread 0x00c328 thru 0x012d98 (0x6a71 bytes) 000905 1775195597.319397 trunc from 0x040000 to 0x006470 (0x39b8f bytes) 000907 1775195597.366695 read 0x00156d thru 0x006470 (0x4f04 bytes) 000909 1775195597.396719 trunc from 0x006471 to 0x005744 (0xd2c bytes) 000910 1775195597.445346 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5745): Mirror 3: 0x59672432 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5745): Mirror 3: 0x59672432 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5745: skipped 000911 1775195597.796602 mapwrite 0x0071ff thru 0x00fe38 (0x8c3a bytes) 000912 1775195597.893772 mapwrite 0x00aac3 thru 0x01754e (0xca8c bytes) 000913 1775195598.015833 trunc from 0x01754f to 0x010518 (0x7036 bytes) 000914 1775195598.041412 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10519): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10519): Mirror 3: 0x35eafe14 (stale) CRC-32 checksum value for chunk [0, 0x10519): Mirror 1: 0xf66daf6f Mirror 4: 0xf66daf6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10519: skipped 000915 1775195599.349850 read 0x0051dc thru 0x010518 (0xb33d bytes) 000924 1775195599.391547 read 0x0099be thru 0x00a91b (0xf5e bytes) 000926 1775195599.392481 trunc from 0x010519 to 0x001df2 (0xe726 bytes) 000927 1775195599.583778 trunc from 0x001df3 to 0x037900 (0x35b0e bytes) 000928 1775195599.631410 read 0x027b3e thru 0x0287e6 (0xca9 bytes) 000929 1775195599.664767 mapwrite 0x004719 thru 0x00bfb2 (0x789a bytes) 000930 1775195599.762077 trunc from 0x037901 to 0x001b67 (0x35d99 bytes) 000931 1775195599.798801 trunc from 0x001b68 to 0x02cfd1 (0x2b46a bytes) 000933 1775195599.858406 trunc from 0x02cfd2 to 0x039135 (0xc164 bytes) 000935 1775195599.896633 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39136): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39136): Mirror 3: 0x6f595991 (stale) Mirror 4: 0x729c1d50 (stale) CRC-32 checksum value for chunk [0, 0x39136): Mirror 1: 0x9646bbbb Mirror 5: 0x9646bbbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39136: skipped 000936 1775195601.873719 mapwrite 0x03a889 thru 0x03ffff (0x5777 bytes) 000937 1775195602.204169 write 0x024670 thru 0x024f90 (0x921 bytes) 000938 1775195602.227289 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) Mirror 5: 0x2f66194 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x744b9828 Mirror 6: 0x744b9828 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000940 1775195603.273411 mapwrite 0x03e406 thru 0x03eaec (0x6e7 bytes) 000941 1775195603.482570 mapwrite 0x012153 thru 0x01def1 (0xbd9f bytes) 000942 1775195603.570563 mapread 0x00e769 thru 0x013dfb (0x5693 bytes) 000943 1775195603.573515 write 0x028538 thru 0x02a27b (0x1d44 bytes) 000944 1775195603.588042 read 0x0002d0 thru 0x001b04 (0x1835 bytes) 000945 1775195603.595736 write 0x014f3c thru 0x023e22 (0xeee7 bytes) 000946 1775195603.598777 mapread 0x016f64 thru 0x020b5c (0x9bf9 bytes) 000950 1775195603.602201 write 0x013c00 thru 0x01c305 (0x8706 bytes) 000951 1775195603.610584 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) Mirror 5: 0x2f66194 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) Mirror 5: 0x2f66194 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1775195604.057964 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1775195604.408689 write 0x01a956 thru 0x021830 (0x6edb bytes) 000954 1775195604.424297 read 0x028128 thru 0x02a842 (0x271b bytes) 000955 1775195604.435284 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59400f4 Mirror 5: 0x59400f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1775195605.503533 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x428e5639 (stale) Mirror 4: 0x727bcaf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59400f4 Mirror 5: 0x59400f4 Mirror 6: 0x59400f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000957 1775195606.522516 read 0x03b48d thru 0x03ffff (0x4b73 bytes) 000959 1775195606.535439 mapread 0x0399ce thru 0x03ffff (0x6632 bytes) 000961 1775195606.553070 read 0x02fddd thru 0x03516e (0x5392 bytes) 000963 1775195606.566401 read 0x012116 thru 0x0130e7 (0xfd2 bytes) 000964 1775195606.567129 mapwrite 0x0137c7 thru 0x017f3b (0x4775 bytes) 000965 1775195606.745259 mapwrite 0x001287 thru 0x008c9f (0x7a19 bytes) 000966 1775195606.792530 trunc from 0x040000 to 0x00e1d5 (0x31e2a bytes) 000969 1775195606.846375 read 0x0085c4 thru 0x00b803 (0x3240 bytes) 000971 1775195606.848542 write 0x027292 thru 0x027c3e (0x9ad bytes) 000972 1775195606.849663 write 0x01857e thru 0x028557 (0xffda bytes) 000973 1775195606.859226 mapwrite 0x0381cb thru 0x03ffff (0x7e35 bytes) 000975 1775195606.942321 mapwrite 0x0393d7 thru 0x03e0f2 (0x4d1c bytes) 000978 1775195606.996175 MIRROR_RESYNC 5 mirrors 000979 1775195608.067390 mapwrite 0x03c9bc thru 0x03fca2 (0x32e7 bytes) 000980 1775195608.219575 mapread 0x030424 thru 0x03c798 (0xc375 bytes) 000981 1775195608.233335 read 0x030687 thru 0x03dedd (0xd857 bytes) 000984 1775195608.235909 read 0x0152e1 thru 0x015f03 (0xc23 bytes) 000986 1775195608.255546 write 0x01faaf thru 0x02042a (0x97c bytes) 000988 1775195608.258575 mapread 0x01cd19 thru 0x02b70a (0xe9f2 bytes) 000989 1775195608.261470 mapread 0x012ea2 thru 0x014c9a (0x1df9 bytes) 000990 1775195608.263582 mapread 0x01db2f thru 0x0273f7 (0x98c9 bytes) 000991 1775195608.265090 read 0x02532f thru 0x02c687 (0x7359 bytes) 000992 1775195608.268605 mapwrite 0x02102c thru 0x02680e (0x57e3 bytes) 000993 1775195608.323309 mapread 0x0231de thru 0x023b8c (0x9af bytes) 000994 1775195608.324966 mapwrite 0x01aa43 thru 0x01aeed (0x4ab bytes) 000995 1775195608.374593 write 0x01dea9 thru 0x02c6dd (0xe835 bytes) 000997 1775195608.381323 read 0x016eb1 thru 0x02155b (0xa6ab bytes) 000998 1775195608.382557 write 0x03b0d8 thru 0x03ffff (0x4f28 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 01:53:34 (1775195614) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 01:53:38 (1775195618) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x212:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.401283 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x212:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194165 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x213:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x212:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x213:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x212:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x218:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 01:53:49 (1775195629) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7152128 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6776256 .. Extending file size to 7582656 .. Extending file size to 7925760 .. Extending file size to 8524448 .. Extending file size to 8416512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9088608 .. Extending file size to 9287136 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55377 55379 55381 55382 55383 failed 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) 192.168.201.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:55:18 (1775195718) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7754048 .. Extending file size to 2639808 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9013568 .. Extending file size to 8564384 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8867648 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 56827 56828 56829 56830 56831 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) 192.168.201.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:56:39 (1775195799) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116651 s, 18.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266266 s, 7.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd305e440 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:57:04 (1775195824) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187673 s, 11.2 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 01:57:14 (1775195834) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x229:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x229:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x22d:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:57:23 (1775195843) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218463 s, 9.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:57:32 (1775195852) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.441247 s, 4.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x230:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x230:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x231:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:57:41 (1775195861) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18084 s, 11.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x232:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x232:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x233:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:58:15 (1775195895) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227748 s, 4.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x235:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x234:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x235:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x234:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x230:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:59:20 (1775195960) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x231:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x232:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.393387 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x231:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x232:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x236:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x231:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x232:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x236:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:00:20 (1775196020) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. Creating new pool test_204f oleg109-server: Pool lustre.test_204f created Adding targets to pool oleg109-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg109-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406713 s, 5.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x233:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x237:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x233:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x237:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg109-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg109-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg109-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:01:43 (1775196103) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:01:51 (1775196111) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:01:59 (1775196119) oleg109-server: Pool lustre.test_206 created Adding targets to pool oleg109-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg109-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg109-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg109-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg109-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:02:25 (1775196145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108539 s, 9.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268311 s, 39.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe0fbce31 Mirror 3: 0xe0fbce31 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:02:36 (1775196156) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:02:42 (1775196162) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (6s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:02:48 (1775196168) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.14596 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x830000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x830000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x830000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23c0000 Read this (4 bytes): BBBB 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:03:31 (1775196211) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0258855 s, 316 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:03:39 (1775196219) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411535 s, 25.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1da lmm_fid: [0x200000404:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1da lmm_fid: [0x200000404:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x242:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834394 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-03 02:03:41.000000000 -0400 Modify: 2026-04-03 02:03:41.000000000 -0400 Change: 2026-04-03 02:03:42.000000000 -0400 Birth: 2026-04-03 02:03:41.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:03:48 (1775196228) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:03:53 (1775196233) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.709788 s, 57.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0380588 s, 108 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 3 02:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 3 02:03 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:04:02 (1775196242) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:04:10 (1775196250) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1936 sec =========== 02:04:21 (1775196261) === sanity-flr: start cleanup 02:04:23 (1775196263) === === sanity-flr: finish cleanup 02:04:28 (1775196268) ===