-----============= acceptance-small: sanity-flr ============----- Fri Jan 9 20:24:30 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg460-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:24:40 (1768008280) === oleg460-client.virtnet: executing check_config_client /mnt/lustre oleg460-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg460-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f9789a10800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f9789a10800.idle_timeout=debug disable quota as required oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:24:58 (1768008298) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7556] == sanity-flr test 0a: lfs mirror create with -N option == 20:24:59 (1768008299) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:25:09 (1768008309) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:25:11 (1768008311) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:25:13 (1768008313) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:25:26 (1768008326) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:25:28 (1768008328) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:25:31 (1768008331) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f9789a10800.stats=clear osc.lustre-OST0001-osc-ffff9f9789a10800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:25:36 (1768008336) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:25:42 (1768008342) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:25:48 (1768008348) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:25:50 (1768008350) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 20:25:56 (1768008356) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:26:00 (1768008360) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:26:02 (1768008362) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:26:06 (1768008366) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:26:13 (1768008373) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:26:18 (1768008378) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.356749 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990652 s, 10.6 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:26:24 (1768008384) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.320955 s, 65.3 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:26:29 (1768008389) Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:27:07 (1768008427) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.301477 s, 34.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (21s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:27:28 (1768008448) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (35s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:28:03 (1768008483) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.3388 s, 22.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 9 20:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.607s user 0m0.001s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.716s user 0m0.001s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.732s user 0m0.001s sys 0m0.173s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg460-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f9789a10800.ost_server_uuid in FULL state after 0 sec PASS 33b (47s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:28:50 (1768008530) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:28:52 (1768008532) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:28:54 (1768008534) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:28:56 (1768008536) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 20:29:01 (1768008541) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.265791 s, 47.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:29:08 (1768008548) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197911 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222617 s, 47.1 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:29:12 (1768008552) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:29:17 (1768008557) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000702958 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000727529 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102749 s, 997 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00080147 s, 1.3 MB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 20:29:22 (1768008562) create test files with size 15729071 .. b2934300ebc8eccee9ae86ad7a84c665 - bbafe1c8767e45805c5759974bfe9069 - 17b1d82086d80689fdde7a4b2d69af70 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 20:29:33 (1768008573) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:29:42 (1768008582) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:29:46 (1768008586) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431288 s, 24.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171947 s, 61.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 20:29:50 (1768008590) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0710102 s, 29.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.238732 s, 17.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114383 s, 18.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 20:29:59 (1768008599) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:30:01 (1768008601) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:30:03 (1768008603) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187356 s, 56.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 20:30:07 (1768008607) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:30:08 (1768008608) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:30:13 (1768008613) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.13946 s, 75.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:30:26 (1768008626) llite.lustre-ffff9f978a301000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.114095 s, 91.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f978501c000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:30:33 (1768008633) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (17s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:30:50 (1768008650) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:31:06 (1768008666) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 20:31:08 (1768008668) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137303 s, 22.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0889273 s, 35.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x74e7d1b5 (stale) Mirror 2: 0x74e7d1b5 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 9c7d17656f484ef73d818ecd9c3c0eec - mirror 1 checksum: a04ef2904259591372b7d68795b28cfc - mirror 2 checksum: 9c7d17656f484ef73d818ecd9c3c0eec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jan 9 20:31:09 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jan 9 20:31:09 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9c7d17656f484ef73d818ecd9c3c0eec - mirror 1 checksum: a04ef2904259591372b7d68795b28cfc - mirror 2 checksum: a04ef2904259591372b7d68795b28cfc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jan 9 20:31:09 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jan 9 20:31:09 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:31:14 (1768008674) Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 20:31:17 (1768008677) shut down facet: mds1 facet_host: oleg460-server facet_failover_host: oleg460-server Failover mds1 to oleg460-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 20:31:32 (1768008692) targets are mounted 20:31:32 (1768008692) facet_failover done oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193721 s, 54.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (26s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:31:40 (1768008700) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:31:41 (1768008701) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:31:43 (1768008703) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00341606 s, 4.8 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:31:46 (1768008706) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:31:48 (1768008708) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:31:49 (1768008709) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:32:00 (1768008720) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:32:20 (1768008740) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:32:39 (1768008759) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:32:58 (1768008778) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 20:33:01 (1768008781) mirror create pid 52222 mirror split pid 52223 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52222 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52223 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 20:34:05 (1768008845) Seed set to 8846 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1768008846.614462 write 0x01aa77 thru 0x01c281 (0x180b bytes) truncating to largest ever: 0x2e6bc 000002 1768008846.623219 trunc from 0x01c282 to 0x02e6bb (0x1243a bytes) 000004 1768008846.637861 mapread 0x004819 thru 0x004d0d (0x4f5 bytes) 000007 1768008846.644180 read 0x013daf thru 0x01b833 (0x7a85 bytes) 000009 1768008846.653700 mapread 0x020616 thru 0x02e39a (0xdd85 bytes) 000010 1768008846.673908 trunc from 0x02e6bc to 0x004f5b (0x29760 bytes) 000012 1768008846.688463 read 0x004000 thru 0x004f5b (0xf5c bytes) 000013 1768008846.689239 read 0x000985 thru 0x004f5b (0x45d7 bytes) 000016 1768008846.694507 trunc from 0x004f5c to 0x0124cb (0xd570 bytes) 000017 1768008846.712971 read 0x0054ea thru 0x00d871 (0x8388 bytes) 000018 1768008846.719696 write 0x029862 thru 0x02b97e (0x211d bytes) 000019 1768008846.723352 mapwrite 0x01947b thru 0x01f40f (0x5f95 bytes) 000020 1768008846.770023 write 0x025dfc thru 0x02911c (0x3321 bytes) 000021 1768008846.771192 write 0x0147f3 thru 0x014b37 (0x345 bytes) 000024 1768008846.773977 read 0x00436c thru 0x00f33b (0xafd0 bytes) truncating to largest ever: 0x33e71 000025 1768008846.774903 trunc from 0x02b97f to 0x033e70 (0x84f2 bytes) 000028 1768008846.795666 write 0x00072c thru 0x0080eb (0x79c0 bytes) 000029 1768008846.796910 mapread 0x0078f8 thru 0x015dae (0xe4b7 bytes) 000033 1768008846.799512 read 0x021cf1 thru 0x022c8b (0xf9b bytes) 000034 1768008846.800072 write 0x02f294 thru 0x03a844 (0xb5b1 bytes) 000035 1768008846.806171 read 0x02d515 thru 0x037340 (0x9e2c bytes) 000036 1768008846.806717 read 0x01d8d8 thru 0x01dba7 (0x2d0 bytes) 000039 1768008846.808120 read 0x000e4a thru 0x009cc6 (0x8e7d bytes) 000041 1768008846.809229 mapwrite 0x005c22 thru 0x00adb6 (0x5195 bytes) 000042 1768008846.848439 mapwrite 0x014950 thru 0x01961c (0x4ccd bytes) 000043 1768008846.877638 mapwrite 0x0000fc thru 0x005350 (0x5255 bytes) 000044 1768008846.915399 trunc from 0x03a845 to 0x009ce5 (0x30b5f bytes) 000047 1768008846.939560 mapread 0x007a31 thru 0x009ce5 (0x22b5 bytes) 000051 1768008846.944264 read 0x002329 thru 0x002e86 (0xb5e bytes) 000055 1768008846.946419 mapread 0x007bd7 thru 0x009ce5 (0x210f bytes) 000057 1768008846.948961 mapwrite 0x0332da thru 0x038b1b (0x5842 bytes) 000059 1768008847.024066 mapwrite 0x00a8f0 thru 0x00e850 (0x3f61 bytes) 000065 1768008847.066042 read 0x0181a1 thru 0x019890 (0x16f0 bytes) 000067 1768008847.067428 trunc from 0x038b1c to 0x0294ea (0xf631 bytes) truncating to largest ever: 0x3e74f 000069 1768008847.082521 trunc from 0x0294eb to 0x03e74e (0x15264 bytes) 000070 1768008847.100263 read 0x03df3d thru 0x03e74e (0x812 bytes) 000072 1768008847.113340 write 0x03638c thru 0x03be23 (0x5a98 bytes) 000073 1768008847.115404 mapread 0x035040 thru 0x03e74e (0x970f bytes) 000074 1768008847.116760 write 0x03150e thru 0x03f10d (0xdc00 bytes) 000075 1768008847.118853 mapread 0x010cc2 thru 0x011629 (0x968 bytes) 000077 1768008847.120233 trunc from 0x03f10e to 0x003614 (0x3baf9 bytes) 000078 1768008847.131366 write 0x02eaaf thru 0x03b41e (0xc970 bytes) 000079 1768008847.134651 read 0x02aa58 thru 0x0355a9 (0xab52 bytes) 000080 1768008847.148307 read 0x0039fb thru 0x004847 (0xe4d bytes) 000081 1768008847.149015 mapwrite 0x015a9a thru 0x02118d (0xb6f4 bytes) 000083 1768008847.204300 write 0x02b363 thru 0x02ff52 (0x4bf0 bytes) 000084 1768008847.205006 mapwrite 0x02b113 thru 0x02e3f9 (0x32e7 bytes) 000085 1768008847.239031 MIRROR_EXTEND to 2 mirrors 000086 1768008847.593489 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) 000087 1768008847.695968 mapread 0x00f849 thru 0x01e6a1 (0xee59 bytes) 000089 1768008847.724455 mapread 0x027c2e thru 0x032a8d (0xae60 bytes) 000091 1768008847.743199 mapread 0x02099c thru 0x0259c3 (0x5028 bytes) 000092 1768008847.744175 mapwrite 0x038f44 thru 0x03ffff (0x70bc bytes) 000095 1768008847.808289 mapread 0x00444c thru 0x007ee7 (0x3a9c bytes) 000096 1768008847.814035 read 0x03e255 thru 0x03ffff (0x1dab bytes) 000097 1768008847.814665 write 0x035daf thru 0x03e94f (0x8ba1 bytes) 000099 1768008847.815856 read 0x0333ae thru 0x03ffff (0xcc52 bytes) 000100 1768008847.816504 read 0x03c479 thru 0x03ffff (0x3b87 bytes) 000101 1768008847.816930 mapread 0x01816a thru 0x0267b2 (0xe649 bytes) 000102 1768008847.825455 read 0x03a6f6 thru 0x03ffff (0x590a bytes) 000104 1768008847.826229 mapread 0x02ae10 thru 0x037b60 (0xcd51 bytes) 000105 1768008847.829534 MIRROR_EXTEND to 2 mirrors 000106 1768008848.216708 read 0x03368a thru 0x03ffff (0xc976 bytes) 000107 1768008848.222225 mapread 0x0357ef thru 0x037d19 (0x252b bytes) 000110 1768008848.223696 mapread 0x020094 thru 0x02ac55 (0xabc2 bytes) 000111 1768008848.238656 read 0x0068bf thru 0x00e734 (0x7e76 bytes) 000113 1768008848.247345 write 0x0329e4 thru 0x03ffff (0xd61c bytes) 000116 1768008848.313053 trunc from 0x040000 to 0x001f3e (0x3e0c1 bytes) 000117 1768008848.342891 MIRROR_RESYNC 2 mirrors 000118 1768008848.879710 write 0x03b782 thru 0x03ffff (0x487e bytes) 000119 1768008848.945082 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x25118ada (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8075c3a Mirror 3: 0xe8075c3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000121 1768008849.554668 write 0x014e48 thru 0x0196c4 (0x487d bytes) 000122 1768008849.633210 read 0x00ba40 thru 0x016b3e (0xb0ff bytes) 000125 1768008849.654756 MIRROR_RESYNC 3 mirrors 000126 1768008850.147390 write 0x029d65 thru 0x031de0 (0x807c bytes) 000127 1768008850.187804 write 0x033a1e thru 0x03aec6 (0x74a9 bytes) 000131 1768008850.195164 read 0x02ba02 thru 0x036c1d (0xb21c bytes) 000132 1768008850.195752 mapread 0x029d6d thru 0x0350e8 (0xb37c bytes) 000137 1768008850.198082 trunc from 0x040000 to 0x024b05 (0x1b4fa bytes) 000139 1768008850.224160 write 0x03bc96 thru 0x03f649 (0x39b4 bytes) 000140 1768008850.225207 mapwrite 0x0126ee thru 0x017f48 (0x585b bytes) 000141 1768008850.275575 trunc from 0x03f64a to 0x01cb56 (0x22af3 bytes) 000142 1768008850.294245 mapwrite 0x01e89e thru 0x0237cc (0x4f2f bytes) 000144 1768008850.337389 write 0x01b702 thru 0x02270b (0x700a bytes) 000146 1768008850.341680 MIRROR_RESYNC 3 mirrors 000148 1768008850.732335 write 0x0342de thru 0x03fdb5 (0xbad8 bytes) 000150 1768008850.773507 read 0x002590 thru 0x0123c1 (0xfe32 bytes) 000155 1768008850.785389 mapwrite 0x009eff thru 0x011396 (0x7498 bytes) 000158 1768008850.829171 write 0x03968d thru 0x03ffff (0x6973 bytes) 000159 1768008850.830223 trunc from 0x040000 to 0x00a1f5 (0x35e0a bytes) 000161 1768008850.874058 mapread 0x0042b2 thru 0x00a1f5 (0x5f44 bytes) 000163 1768008850.875547 mapread 0x007edb thru 0x00a1f5 (0x231b bytes) 000166 1768008850.878995 read 0x009831 thru 0x00a1f5 (0x9c5 bytes) 000167 1768008850.879510 mapwrite 0x013aca thru 0x0141bc (0x6f3 bytes) 000168 1768008850.943018 read 0x00d708 thru 0x0141bc (0x6ab5 bytes) 000169 1768008850.948751 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x141bd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141bd): Mirror 2: 0xcb51e126 (stale) Mirror 3: 0xcb51e126 (stale) CRC-32 checksum value for chunk [0, 0x141bd): Mirror 1: 0xe47ddd59 Mirror 4: 0xe47ddd59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141bd: skipped 000171 1768008851.408837 mapwrite 0x01fb8b thru 0x0211de (0x1654 bytes) 000172 1768008851.491168 trunc from 0x0211df to 0x019041 (0x819d bytes) 000177 1768008851.505485 read 0x00392d thru 0x009291 (0x5965 bytes) 000178 1768008851.512040 mapread 0x011cad thru 0x0126d3 (0xa27 bytes) 000179 1768008851.521214 mapread 0x01584a thru 0x019041 (0x37f8 bytes) 000180 1768008851.522132 trunc from 0x019042 to 0x02ec74 (0x15c33 bytes) 000182 1768008851.532140 mapread 0x0094d9 thru 0x00fe10 (0x6938 bytes) 000183 1768008851.532991 read 0x022548 thru 0x022f12 (0x9cb bytes) 000184 1768008851.545604 write 0x02a861 thru 0x02b1d6 (0x976 bytes) 000185 1768008851.546596 trunc from 0x02ec75 to 0x03a76e (0xbafa bytes) 000189 1768008851.559803 read 0x001362 thru 0x007bc6 (0x6865 bytes) 000190 1768008851.560493 mapwrite 0x01972a thru 0x01f98a (0x6261 bytes) 000191 1768008851.585608 mapread 0x031e66 thru 0x039cb9 (0x7e54 bytes) 000192 1768008851.591381 write 0x0014ed thru 0x00ca14 (0xb528 bytes) 000193 1768008851.598366 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a76f): Mirror 3: 0x77e3ee4d (stale) Mirror 4: 0xc3688798 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a76f): Mirror 3: 0x77e3ee4d (stale) Mirror 4: 0xc3688798 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a76f: skipped 000194 1768008851.820472 mapwrite 0x025569 thru 0x03354e (0xdfe6 bytes) 000195 1768008851.864596 mapwrite 0x030992 thru 0x030a55 (0xc4 bytes) 000196 1768008851.895707 read 0x0128f2 thru 0x01954b (0x6c5a bytes) 000198 1768008851.904450 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a76f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a76f): Mirror 3: 0x77e3ee4d (stale) Mirror 4: 0xc3688798 (stale) CRC-32 checksum value for chunk [0, 0x3a76f): Mirror 1: 0x7530b510 Mirror 5: 0x7530b510 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a76f: skipped 000199 1768008852.478613 trunc from 0x03a76f to 0x005877 (0x34ef7 bytes) 000202 1768008852.541841 mapread 0x0002b6 thru 0x005877 (0x55c2 bytes) 000203 1768008852.562634 read 0x0037fd thru 0x005877 (0x207b bytes) 000204 1768008852.563254 write 0x02e7d5 thru 0x03bdf6 (0xd622 bytes) 000205 1768008852.565021 mapwrite 0x00b05c thru 0x017ee0 (0xce85 bytes) 000206 1768008852.614911 mapwrite 0x0105c6 thru 0x01425b (0x3c96 bytes) 000207 1768008852.639266 write 0x01875e thru 0x01cc2e (0x44d1 bytes) 000210 1768008852.641397 read 0x02528d thru 0x029e3b (0x4baf bytes) 000212 1768008852.648900 mapwrite 0x005339 thru 0x0100c5 (0xad8d bytes) 000213 1768008852.681793 write 0x00d939 thru 0x01a962 (0xd02a bytes) 000215 1768008852.683512 read 0x00450b thru 0x013de4 (0xf8da bytes) 000216 1768008852.684086 trunc from 0x03bdf7 to 0x03821b (0x3bdb bytes) 000218 1768008852.699085 trunc from 0x03821c to 0x01e974 (0x198a7 bytes) 000220 1768008852.712449 mapwrite 0x02b7fb thru 0x03134f (0x5b55 bytes) 000221 1768008852.760509 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31350): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31350): Mirror 3: 0x3fb32145 (stale) Mirror 4: 0x9c299b3e (stale) Mirror 5: 0x3cb61548 (stale) CRC-32 checksum value for chunk [0, 0x31350): Mirror 1: 0x86e6ce2b Mirror 6: 0x86e6ce2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31350: skipped 000226 1768008853.237821 mapread 0x00abec thru 0x00ce26 (0x223b bytes) 000227 1768008853.255809 mapread 0x022038 thru 0x02cc8d (0xac56 bytes) 000228 1768008853.268769 read 0x0176ab thru 0x020837 (0x918d bytes) 000229 1768008853.277586 MIRROR_RESYNC 5 mirrors 000230 1768008853.792718 trunc from 0x031350 to 0x01f8b4 (0x11a9b bytes) 000231 1768008853.852140 write 0x02649f thru 0x02fcb0 (0x9812 bytes) 000232 1768008853.856316 read 0x016e53 thru 0x0176ac (0x85a bytes) 000234 1768008853.862260 mapread 0x028156 thru 0x02fcb0 (0x7b5b bytes) 000236 1768008853.867190 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fcb1): Mirror 4: 0xbebb464d (stale) Mirror 5: 0xbebb464d (stale) Mirror 6: 0xbebb464d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fcb1): Mirror 4: 0xbebb464d (stale) Mirror 5: 0xbebb464d (stale) Mirror 6: 0xbebb464d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fcb1: skipped 000238 1768008854.168090 trunc from 0x02fcb1 to 0x03a34f (0xa69f bytes) 000243 1768008854.186699 MIRROR_RESYNC 4 mirrors 000244 1768008854.689867 mapwrite 0x03b6c4 thru 0x03ffff (0x493c bytes) 000245 1768008854.833294 write 0x00773c thru 0x01408e (0xc953 bytes) 000247 1768008854.848483 read 0x01ea83 thru 0x02c0d4 (0xd652 bytes) 000248 1768008854.856993 mapwrite 0x00db80 thru 0x015c1a (0x809b bytes) 000252 1768008854.919761 mapread 0x00fdb9 thru 0x016740 (0x6988 bytes) 000253 1768008854.920721 mapwrite 0x0090e3 thru 0x011d96 (0x8cb4 bytes) 000254 1768008854.959250 trunc from 0x040000 to 0x01e714 (0x218eb bytes) 000256 1768008854.971759 mapwrite 0x01c1e0 thru 0x02a049 (0xde6a bytes) 000257 1768008855.028894 mapwrite 0x023dbe thru 0x02c2a1 (0x84e4 bytes) 000258 1768008855.077135 mapwrite 0x024084 thru 0x02c31c (0x8299 bytes) 000260 1768008855.133969 trunc from 0x02c31d to 0x00ec8a (0x1d692 bytes) 000261 1768008855.147919 mapwrite 0x0274aa thru 0x03139e (0x9ef5 bytes) 000264 1768008855.184542 mapwrite 0x03eb10 thru 0x03ffff (0x14f0 bytes) 000265 1768008855.240273 MIRROR_RESYNC 4 mirrors 000267 1768008855.741032 MIRROR_EXTEND to 5 mirrors 000269 1768008856.172488 MIRROR_RESYNC 5 mirrors 000270 1768008856.459718 mapwrite 0x01b1d6 thru 0x029208 (0xe033 bytes) 000272 1768008856.639235 mapwrite 0x0122ac thru 0x016f65 (0x4cba bytes) 000274 1768008856.701004 mapread 0x038877 thru 0x03fe56 (0x75e0 bytes) 000275 1768008856.708633 mapwrite 0x02d294 thru 0x036652 (0x93bf bytes) 000277 1768008856.739696 mapread 0x00c931 thru 0x01bf33 (0xf603 bytes) 000278 1768008856.740503 read 0x012c22 thru 0x015d57 (0x3136 bytes) 000279 1768008856.741171 trunc from 0x040000 to 0x012ebc (0x2d143 bytes) 000280 1768008856.772660 write 0x03d8ba thru 0x03ffff (0x2746 bytes) 000281 1768008856.773703 read 0x002d69 thru 0x006bf4 (0x3e8c bytes) 000282 1768008856.783604 write 0x03e3f3 thru 0x03ffff (0x1c0d bytes) 000283 1768008856.784402 read 0x00931b thru 0x011269 (0x7f4f bytes) 000284 1768008856.788360 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x269c0b69 (stale) Mirror 5: 0x269c0b69 (stale) Mirror 6: 0x269c0b69 (stale) Mirror 7: 0x269c0b69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcbfcde Mirror 8: 0xcbfcde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000286 1768008857.283056 trunc from 0x040000 to 0x01fdb1 (0x2024e bytes) 000290 1768008857.379915 write 0x015d41 thru 0x0180ef (0x23af bytes) 000291 1768008857.385137 mapwrite 0x0306f3 thru 0x03d00b (0xc919 bytes) 000294 1768008857.456278 write 0x03df40 thru 0x03f21c (0x12dd bytes) 000295 1768008857.457453 write 0x01e071 thru 0x0213ca (0x335a bytes) 000296 1768008857.463434 read 0x0080fb thru 0x011bde (0x9ae4 bytes) 000298 1768008857.469576 mapwrite 0x027f35 thru 0x03681e (0xe8ea bytes) 000299 1768008857.501846 mapwrite 0x03a06a thru 0x03ffff (0x5f96 bytes) 000305 1768008857.543284 trunc from 0x040000 to 0x03ad4b (0x52b4 bytes) 000309 1768008857.565656 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ad4c): Mirror 5: 0x15615a37 (stale) Mirror 6: 0x15615a37 (stale) Mirror 7: 0x15615a37 (stale) Mirror 8: 0xa805d22d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad4c): Mirror 5: 0x15615a37 (stale) Mirror 6: 0x15615a37 (stale) Mirror 7: 0x15615a37 (stale) Mirror 8: 0xa805d22d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad4c: skipped 000310 1768008857.873521 mapwrite 0x026e2a thru 0x030784 (0x995b bytes) 000311 1768008857.914819 read 0x022f7d thru 0x02e509 (0xb58d bytes) 000313 1768008857.915729 read 0x03a30a thru 0x03ad4b (0xa42 bytes) 000315 1768008857.930133 trunc from 0x03ad4c to 0x038d9b (0x1fb0 bytes) 000317 1768008857.943615 read 0x031317 thru 0x038d9b (0x7a85 bytes) 000318 1768008857.944177 mapread 0x012deb thru 0x01f6b7 (0xc8cd bytes) 000319 1768008857.945027 mapread 0x006f79 thru 0x009d22 (0x2daa bytes) 000320 1768008857.946386 mapwrite 0x009d9f thru 0x00edb6 (0x5018 bytes) 000321 1768008857.977591 mapwrite 0x01b11f thru 0x02997b (0xe85d bytes) 000322 1768008858.013697 write 0x014307 thru 0x021ac0 (0xd7ba bytes) 000323 1768008858.014949 mapwrite 0x039960 thru 0x03ffff (0x66a0 bytes) 000324 1768008858.067451 mapread 0x01a4ef thru 0x01feec (0x59fe bytes) 000325 1768008858.068207 write 0x02a891 thru 0x02f88f (0x4fff bytes) 000326 1768008858.068949 read 0x007947 thru 0x007e20 (0x4da bytes) 000328 1768008858.069681 mapwrite 0x01a486 thru 0x026029 (0xbba4 bytes) 000329 1768008858.097224 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x269c0b69 (stale) Mirror 6: 0x269c0b69 (stale) Mirror 7: 0x269c0b69 (stale) Mirror 8: 0xcbfcde (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66950bdb Mirror 9: 0x66950bdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000330 1768008858.576586 mapwrite 0x00336c thru 0x00c7df (0x9474 bytes) 000331 1768008858.679867 mapwrite 0x019873 thru 0x01f0c3 (0x5851 bytes) 000332 1768008858.712342 mapread 0x035428 thru 0x03ffff (0xabd8 bytes) 000334 1768008858.723258 trunc from 0x040000 to 0x0294e7 (0x16b18 bytes) 000337 1768008858.747276 read 0x027b86 thru 0x0294e7 (0x1962 bytes) 000339 1768008858.761154 MIRROR_RESYNC 6 mirrors 000340 1768008859.247604 mapread 0x0165ba thru 0x018abb (0x2502 bytes) 000343 1768008859.262796 read 0x016e6c thru 0x02140c (0xa5a1 bytes) 000344 1768008859.269677 read 0x012633 thru 0x013d34 (0x1702 bytes) 000347 1768008859.270457 mapread 0x011de0 thru 0x014dde (0x2fff bytes) 000348 1768008859.274418 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) 000350 1768008859.462715 write 0x03128a thru 0x03b3df (0xa156 bytes) 000351 1768008859.489829 read 0x033dd6 thru 0x03b136 (0x7361 bytes) 000353 1768008859.490968 mapwrite 0x0161c2 thru 0x020e40 (0xac7f bytes) 000354 1768008859.524160 read 0x02f8c3 thru 0x034bf8 (0x5336 bytes) 000356 1768008859.535152 MIRROR_RESYNC 5 mirrors truncating to largest ever: 0x3f024 000357 1768008859.947634 trunc from 0x03b3e0 to 0x03f023 (0x3c44 bytes) 000358 1768008859.992992 mapread 0x024801 thru 0x02a96b (0x616b bytes) 000360 1768008860.002723 write 0x01c423 thru 0x022034 (0x5c12 bytes) 000361 1768008860.011609 trunc from 0x03f024 to 0x03c665 (0x29be bytes) 000362 1768008860.021771 read 0x02844c thru 0x031b04 (0x96b9 bytes) 000363 1768008860.024991 mapread 0x02bf5e thru 0x02c1d1 (0x274 bytes) 000364 1768008860.026279 write 0x0364cb thru 0x03e9e3 (0x8519 bytes) 000366 1768008860.032857 mapread 0x023467 thru 0x03222f (0xedc9 bytes) 000367 1768008860.033859 mapwrite 0x0230e5 thru 0x02eea8 (0xbdc4 bytes) 000368 1768008860.059948 mapread 0x007e8c thru 0x012c55 (0xadca bytes) 000369 1768008860.071419 trunc from 0x03e9e4 to 0x013e2b (0x2abb8 bytes) 000370 1768008860.083347 read 0x003f8d thru 0x005d42 (0x1db6 bytes) 000371 1768008860.083930 write 0x00bfc2 thru 0x013a78 (0x7ab7 bytes) 000372 1768008860.084929 mapwrite 0x02660c thru 0x02ebb5 (0x85aa bytes) 000373 1768008860.137709 read 0x0243cd thru 0x02acb9 (0x68ed bytes) 000375 1768008860.138900 mapwrite 0x02d46d thru 0x02f066 (0x1bfa bytes) 000376 1768008860.175038 mapwrite 0x02aaad thru 0x02d878 (0x2dcc bytes) 000377 1768008860.191211 write 0x0259c0 thru 0x02b839 (0x5e7a bytes) 000378 1768008860.192080 write 0x02d43f thru 0x039173 (0xbd35 bytes) 000379 1768008860.193136 trunc from 0x039174 to 0x0018f5 (0x3787e bytes) 000381 1768008860.205959 write 0x026592 thru 0x0324ec (0xbf5b bytes) 000382 1768008860.207452 read 0x020803 thru 0x0278e0 (0x70de bytes) 000383 1768008860.212900 mapread 0x0058c8 thru 0x00e228 (0x8961 bytes) 000386 1768008860.214718 read 0x01528b thru 0x01f7af (0xa525 bytes) 000387 1768008860.215324 read 0x027135 thru 0x031217 (0xa0e3 bytes) 000388 1768008860.215901 write 0x018f25 thru 0x0289a3 (0xfa7f bytes) 000393 1768008860.218630 trunc from 0x0324ed to 0x0160c6 (0x1c426 bytes) 000394 1768008860.229682 mapread 0x007a28 thru 0x0160c6 (0xe69f bytes) 000396 1768008860.231121 read 0x009821 thru 0x0160c6 (0xc8a6 bytes) 000399 1768008860.232936 write 0x02aee3 thru 0x035f83 (0xb0a1 bytes) 000403 1768008860.235801 mapread 0x020fed thru 0x026bc0 (0x5bd4 bytes) 000405 1768008860.246903 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35f84): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f84): Mirror 5: 0xf3465bfc (stale) Mirror 6: 0xf3465bfc (stale) Mirror 8: 0xf3465bfc (stale) Mirror 9: 0xf3465bfc (stale) CRC-32 checksum value for chunk [0, 0x35f84): Mirror 1: 0x966dfd4 Mirror 10: 0x966dfd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f84: skipped 000406 1768008860.610872 read 0x0083ba thru 0x00e350 (0x5f97 bytes) 000407 1768008860.616843 mapread 0x016643 thru 0x022e9e (0xc85c bytes) 000408 1768008860.633376 mapread 0x00e349 thru 0x0150cb (0x6d83 bytes) 000411 1768008860.643871 trunc from 0x035f84 to 0x025d2c (0x10257 bytes) 000412 1768008860.683941 read 0x0211a4 thru 0x024b10 (0x396d bytes) 000415 1768008860.697563 read 0x008aff thru 0x00edb3 (0x62b5 bytes) 000416 1768008860.698072 mapread 0x01b23c thru 0x0257c3 (0xa588 bytes) 000418 1768008860.699343 write 0x02b571 thru 0x038ae0 (0xd570 bytes) 000419 1768008860.700925 write 0x011964 thru 0x01c997 (0xb034 bytes) 000421 1768008860.705551 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ae1): Mirror 5: 0x4ec5c0cb (stale) Mirror 6: 0x4ec5c0cb (stale) Mirror 8: 0x4ec5c0cb (stale) Mirror 9: 0x4ec5c0cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ae1): Mirror 5: 0x4ec5c0cb (stale) Mirror 6: 0x4ec5c0cb (stale) Mirror 8: 0x4ec5c0cb (stale) Mirror 9: 0x4ec5c0cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ae1: skipped 000422 1768008860.882843 mapwrite 0x004d5a thru 0x00a926 (0x5bcd bytes) 000424 1768008860.919424 trunc from 0x038ae1 to 0x015bd5 (0x22f0b bytes) 000427 1768008860.927135 trunc from 0x015bd6 to 0x018ea2 (0x32cd bytes) 000429 1768008860.934399 read 0x0012e6 thru 0x010a85 (0xf7a0 bytes) 000431 1768008860.937919 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x18ea3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ea3): Mirror 5: 0xf83ad480 (stale) Mirror 6: 0xf83ad480 (stale) Mirror 8: 0xf83ad480 (stale) Mirror 9: 0xf83ad480 (stale) CRC-32 checksum value for chunk [0, 0x18ea3): Mirror 1: 0xadf9cb88 Mirror 10: 0xadf9cb88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ea3: skipped 000432 1768008861.336970 trunc from 0x018ea3 to 0x02a5a1 (0x116ff bytes) 000433 1768008861.375395 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a5a2): Mirror 5: 0xe5fc3af (stale) Mirror 6: 0xe5fc3af (stale) Mirror 8: 0xe5fc3af (stale) Mirror 9: 0xe5fc3af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a5a2): Mirror 5: 0xe5fc3af (stale) Mirror 6: 0xe5fc3af (stale) Mirror 8: 0xe5fc3af (stale) Mirror 9: 0xe5fc3af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a5a2: skipped 000434 1768008861.567312 read 0x007c6c thru 0x0171f9 (0xf58e bytes) 000436 1768008861.573251 trunc from 0x02a5a2 to 0x021d89 (0x8818 bytes) 000439 1768008861.588826 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21d8a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d8a): Mirror 5: 0xb53dcf37 (stale) Mirror 6: 0xb53dcf37 (stale) Mirror 8: 0xb53dcf37 (stale) Mirror 9: 0xb53dcf37 (stale) CRC-32 checksum value for chunk [0, 0x21d8a): Mirror 1: 0x9067a766 Mirror 10: 0x9067a766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d8a: skipped 000441 1768008861.919803 trunc from 0x021d8a to 0x025d67 (0x3fde bytes) 000444 1768008861.958475 mapread 0x011070 thru 0x01216a (0x10fb bytes) 000445 1768008861.962938 read 0x019889 thru 0x025d67 (0xc4df bytes) 000449 1768008861.970994 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25d68): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d68): Mirror 5: 0x70bc2b37 (stale) Mirror 6: 0x70bc2b37 (stale) Mirror 8: 0x70bc2b37 (stale) Mirror 9: 0x70bc2b37 (stale) Mirror 10: 0x40ccb9ef (stale) CRC-32 checksum value for chunk [0, 0x25d68): Mirror 1: 0x40ccb9ef Mirror 11: 0x40ccb9ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d68: skipped 000451 1768008862.345203 mapread 0x008259 thru 0x00d269 (0x5011 bytes) 000452 1768008862.365748 write 0x034058 thru 0x03bdef (0x7d98 bytes) 000453 1768008862.394477 mapread 0x02966a thru 0x0310a0 (0x7a37 bytes) 000454 1768008862.400701 trunc from 0x03bdf0 to 0x00a968 (0x31487 bytes) 000455 1768008862.412716 read 0x00a564 thru 0x00a968 (0x405 bytes) 000457 1768008862.415682 mapread 0x0024c5 thru 0x009bee (0x772a bytes) 000458 1768008862.420499 trunc from 0x00a969 to 0x0027c1 (0x81a7 bytes) 000459 1768008862.427959 read 0x0007fe thru 0x0027c1 (0x1fc4 bytes) 000461 1768008862.431148 read 0x001e5a thru 0x0027c1 (0x968 bytes) 000462 1768008862.434041 MIRROR_RESYNC 7 mirrors 000463 1768008862.840196 MIRROR_EXTEND to 8 mirrors truncating to largest ever: 0x3f656 000464 1768008863.137512 trunc from 0x0027c2 to 0x03f655 (0x3ce94 bytes) 000465 1768008863.183829 trunc from 0x03f656 to 0x0301d5 (0xf480 bytes) 000468 1768008863.197624 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x301d6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301d6): Mirror 5: 0x1034c7f2 (stale) Mirror 6: 0x1034c7f2 (stale) Mirror 8: 0x1034c7f2 (stale) Mirror 9: 0x1034c7f2 (stale) Mirror 10: 0x1034c7f2 (stale) Mirror 11: 0x1034c7f2 (stale) Mirror 12: 0x1034c7f2 (stale) CRC-32 checksum value for chunk [0, 0x301d6): Mirror 1: 0x1034c7f2 Mirror 13: 0x1034c7f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301d6: skipped 000469 1768008863.599708 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301d6): Mirror 5: 0x1034c7f2 (stale) Mirror 6: 0x1034c7f2 (stale) Mirror 8: 0x1034c7f2 (stale) Mirror 9: 0x1034c7f2 (stale) Mirror 10: 0x1034c7f2 (stale) Mirror 11: 0x1034c7f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301d6): Mirror 5: 0x1034c7f2 (stale) Mirror 6: 0x1034c7f2 (stale) Mirror 8: 0x1034c7f2 (stale) Mirror 9: 0x1034c7f2 (stale) Mirror 10: 0x1034c7f2 (stale) Mirror 11: 0x1034c7f2 (stale) CRC-32 checksum value for chunk [0, 0x301d6): Mirror 1: 0x1034c7f2 Mirror 13: 0x1034c7f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301d6: skipped 000470 1768008864.030639 mapread 0x02f95e thru 0x0301d5 (0x878 bytes) 000471 1768008864.062362 read 0x008eec thru 0x0169c8 (0xdadd bytes) 000472 1768008864.077081 mapread 0x015884 thru 0x022d4e (0xd4cb bytes) 000477 1768008864.143572 read 0x024db1 thru 0x02bafe (0x6d4e bytes) 000480 1768008864.177587 read 0x004384 thru 0x0126d6 (0xe353 bytes) 000481 1768008864.186001 trunc from 0x0301d6 to 0x03546f (0x529a bytes) 000482 1768008864.217061 mapwrite 0x02da65 thru 0x02e509 (0xaa5 bytes) 000484 1768008864.248203 mapread 0x01cb2b thru 0x020c0c (0x40e2 bytes) 000487 1768008864.249901 trunc from 0x035470 to 0x00026b (0x35204 bytes) 000489 1768008864.261651 mapread 0x000182 thru 0x00026b (0xea bytes) 000492 1768008864.263300 write 0x02486d thru 0x033eff (0xf693 bytes) 000493 1768008864.264970 mapread 0x01b6d5 thru 0x01ddeb (0x2717 bytes) 000496 1768008864.275638 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33f00): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f00): Mirror 5: 0xac7baad3 (stale) Mirror 6: 0xac7baad3 (stale) Mirror 8: 0xac7baad3 (stale) Mirror 9: 0xac7baad3 (stale) Mirror 10: 0xac7baad3 (stale) Mirror 11: 0xac7baad3 (stale) Mirror 13: 0xac7baad3 (stale) CRC-32 checksum value for chunk [0, 0x33f00): Mirror 1: 0x48e49a27 Mirror 14: 0x48e49a27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f00: skipped 000498 1768008864.696865 mapread 0x01a8b3 thru 0x020088 (0x57d6 bytes) 000500 1768008864.721347 read 0x025265 thru 0x02aabd (0x5859 bytes) 000502 1768008864.735091 mapwrite 0x026123 thru 0x0270c1 (0xf9f bytes) 000506 1768008864.859625 write 0x0319cc thru 0x035049 (0x367e bytes) 000507 1768008864.870169 mapwrite 0x00098d thru 0x00d9a3 (0xd017 bytes) 000508 1768008864.916689 mapread 0x00179d thru 0x003953 (0x21b7 bytes) 000510 1768008864.924142 MIRROR_RESYNC 9 mirrors 000513 1768008865.719976 mapwrite 0x01502b thru 0x020dcf (0xbda5 bytes) 000514 1768008865.834196 mapread 0x01face thru 0x02db14 (0xe047 bytes) 000515 1768008865.841133 read 0x0092f9 thru 0x016642 (0xd34a bytes) 000516 1768008865.847181 mapwrite 0x007890 thru 0x014bbc (0xd32d bytes) 000518 1768008865.886161 read 0x020cf0 thru 0x02ce00 (0xc111 bytes) 000519 1768008865.886915 mapread 0x00358c thru 0x0118ab (0xe320 bytes) 000524 1768008865.890599 mapread 0x01c3cd thru 0x028f82 (0xcbb6 bytes) 000526 1768008865.891733 read 0x004a96 thru 0x0095f4 (0x4b5f bytes) 000529 1768008865.893034 mapread 0x008f0c thru 0x016b75 (0xdc6a bytes) 000531 1768008865.894240 read 0x030fe4 thru 0x035049 (0x4066 bytes) 000532 1768008865.900179 write 0x00cfe3 thru 0x012b3a (0x5b58 bytes) 000533 1768008865.901001 mapread 0x01703d thru 0x01a73d (0x3701 bytes) 000534 1768008865.905077 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3504a): Mirror 5: 0x627319a2 (stale) Mirror 6: 0x627319a2 (stale) Mirror 8: 0x627319a2 (stale) Mirror 9: 0x627319a2 (stale) Mirror 10: 0x627319a2 (stale) Mirror 11: 0x627319a2 (stale) Mirror 14: 0x627319a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3504a): Mirror 5: 0x627319a2 (stale) Mirror 6: 0x627319a2 (stale) Mirror 8: 0x627319a2 (stale) Mirror 9: 0x627319a2 (stale) Mirror 10: 0x627319a2 (stale) Mirror 11: 0x627319a2 (stale) Mirror 14: 0x627319a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3504a: skipped 000535 1768008866.225631 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3504a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3504a): Mirror 5: 0x627319a2 (stale) Mirror 6: 0x627319a2 (stale) Mirror 8: 0x627319a2 (stale) Mirror 9: 0x627319a2 (stale) Mirror 10: 0x627319a2 (stale) Mirror 11: 0x627319a2 (stale) Mirror 14: 0x627319a2 (stale) CRC-32 checksum value for chunk [0, 0x3504a): Mirror 1: 0xc2802efe Mirror 15: 0xc2802efe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3504a: skipped 000536 1768008866.780576 mapwrite 0x017878 thru 0x017fbe (0x747 bytes) 000538 1768008866.862869 read 0x02df95 thru 0x031470 (0x34dc bytes) 000539 1768008866.868991 mapwrite 0x039d37 thru 0x03ffff (0x62c9 bytes) 000540 1768008866.904732 read 0x037906 thru 0x03ffff (0x86fa bytes) 000542 1768008866.905674 trunc from 0x040000 to 0x02f573 (0x10a8c bytes) 000545 1768008866.917113 read 0x007a70 thru 0x014452 (0xc9e3 bytes) 000546 1768008866.923293 read 0x0044dd thru 0x012ca6 (0xe7ca bytes) 000548 1768008866.924041 read 0x0209af thru 0x02ee32 (0xe484 bytes) 000552 1768008866.925660 write 0x013bfb thru 0x01b6f9 (0x7aff bytes) 000553 1768008866.926779 read 0x0026e1 thru 0x00bd74 (0x9694 bytes) 000554 1768008866.927480 mapwrite 0x00510e thru 0x00b5e0 (0x64d3 bytes) 000556 1768008866.961026 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f574): Mirror 5: 0xde89d892 (stale) Mirror 6: 0xde89d892 (stale) Mirror 8: 0xde89d892 (stale) Mirror 9: 0xde89d892 (stale) Mirror 11: 0xde89d892 (stale) Mirror 14: 0xde89d892 (stale) Mirror 15: 0x5d1af15e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f574): Mirror 5: 0xde89d892 (stale) Mirror 6: 0xde89d892 (stale) Mirror 8: 0xde89d892 (stale) Mirror 9: 0xde89d892 (stale) Mirror 11: 0xde89d892 (stale) Mirror 14: 0xde89d892 (stale) Mirror 15: 0x5d1af15e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f574: skipped 000557 1768008867.186881 mapwrite 0x027c89 thru 0x032396 (0xa70e bytes) 000559 1768008867.234816 mapread 0x02d05a thru 0x0311b7 (0x415e bytes) 000561 1768008867.236154 trunc from 0x032397 to 0x014e3c (0x1d55a bytes) 000563 1768008867.248986 mapread 0x00ee93 thru 0x014e3c (0x5faa bytes) 000564 1768008867.255953 mapread 0x009655 thru 0x014e3c (0xb7e8 bytes) 000566 1768008867.257345 write 0x03dd32 thru 0x03e7a1 (0xa70 bytes) 000567 1768008867.262600 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e7a2): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7a2): Mirror 5: 0x41762f76 (stale) Mirror 6: 0x41762f76 (stale) Mirror 8: 0x41762f76 (stale) Mirror 9: 0x41762f76 (stale) Mirror 11: 0x41762f76 (stale) Mirror 14: 0x41762f76 (stale) Mirror 15: 0x29995ff (stale) CRC-32 checksum value for chunk [0, 0x3e7a2): Mirror 1: 0x9423c7d5 Mirror 16: 0x9423c7d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7a2: skipped 000568 1768008867.744715 mapread 0x01f51e thru 0x02743c (0x7f1f bytes) 000569 1768008867.759846 mapread 0x02cf07 thru 0x02e5c7 (0x16c1 bytes) 000572 1768008867.777933 write 0x016458 thru 0x01be3d (0x59e6 bytes) 000574 1768008867.836386 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e7a2): Mirror 5: 0x41762f76 (stale) Mirror 6: 0x41762f76 (stale) Mirror 8: 0x41762f76 (stale) Mirror 9: 0x41762f76 (stale) Mirror 11: 0x41762f76 (stale) Mirror 14: 0x41762f76 (stale) Mirror 16: 0x9423c7d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7a2): Mirror 5: 0x41762f76 (stale) Mirror 6: 0x41762f76 (stale) Mirror 8: 0x41762f76 (stale) Mirror 9: 0x41762f76 (stale) Mirror 11: 0x41762f76 (stale) Mirror 14: 0x41762f76 (stale) Mirror 16: 0x9423c7d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7a2: skipped 000575 1768008868.130515 trunc from 0x03e7a2 to 0x01a06e (0x24733 bytes) 000576 1768008868.158046 trunc from 0x01a06f to 0x03ab91 (0x20b23 bytes) 000579 1768008868.176431 write 0x0035cf thru 0x00d90e (0xa340 bytes) 000580 1768008868.183258 read 0x032585 thru 0x03ab91 (0x860d bytes) 000581 1768008868.193394 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: 0x3d58c09f (stale) Mirror 6: 0x3d58c09f (stale) Mirror 8: 0x3d58c09f (stale) Mirror 9: 0x3d58c09f (stale) Mirror 11: 0x3d58c09f (stale) Mirror 16: 0x3ff81886 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: 0x3d58c09f (stale) Mirror 6: 0x3d58c09f (stale) Mirror 8: 0x3d58c09f (stale) Mirror 9: 0x3d58c09f (stale) Mirror 11: 0x3d58c09f (stale) Mirror 16: 0x3ff81886 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab92: skipped 000582 1768008868.492559 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: 0x3d58c09f (stale) Mirror 6: 0x3d58c09f (stale) Mirror 8: 0x3d58c09f (stale) Mirror 9: 0x3d58c09f (stale) Mirror 11: 0x3d58c09f (stale) Mirror 16: 0x3ff81886 (stale) CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 1: 0x96121359 Mirror 17: 0x96121359 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab92: skipped 000583 1768008868.959064 mapread 0x025621 thru 0x02a0d2 (0x4ab2 bytes) 000585 1768008869.001000 read 0x023eba thru 0x02b1e3 (0x732a bytes) 000587 1768008869.010556 read 0x0128e8 thru 0x01eb5f (0xc278 bytes) 000588 1768008869.027045 MIRROR_RESYNC 8 mirrors 000590 1768008869.734210 write 0x022d36 thru 0x029c5c (0x6f27 bytes) 000592 1768008869.791668 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: 0x96121359 (stale) Mirror 6: 0x96121359 (stale) Mirror 8: 0x96121359 (stale) Mirror 9: 0x96121359 (stale) Mirror 11: 0x96121359 (stale) Mirror 17: 0x96121359 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab92): Mirror 5: 0x96121359 (stale) Mirror 6: 0x96121359 (stale) Mirror 8: 0x96121359 (stale) Mirror 9: 0x96121359 (stale) Mirror 11: 0x96121359 (stale) Mirror 17: 0x96121359 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab92: skipped 000595 1768008870.091405 mapread 0x0162b0 thru 0x01ff30 (0x9c81 bytes) 000596 1768008870.100706 trunc from 0x03ab92 to 0x031939 (0x9258 bytes) 000597 1768008870.122449 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3193a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3193a): Mirror 5: 0x33f7333e (stale) Mirror 6: 0x33f7333e (stale) Mirror 8: 0x33f7333e (stale) Mirror 9: 0x33f7333e (stale) Mirror 11: 0x33f7333e (stale) Mirror 17: 0x33f7333e (stale) CRC-32 checksum value for chunk [0, 0x3193a): Mirror 1: 0xc86f4e91 Mirror 18: 0xc86f4e91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3193a: skipped 000598 1768008870.657382 read 0x00f3ec thru 0x019411 (0xa026 bytes) 000599 1768008870.670934 MIRROR_RESYNC 8 mirrors 000603 1768008871.294320 MIRROR_RESYNC 8 mirrors 000604 1768008871.562833 mapread 0x0235ac thru 0x0264ca (0x2f1f bytes) 000605 1768008871.581251 write 0x035b54 thru 0x03ffff (0xa4ac bytes) 000606 1768008871.621540 mapread 0x0203ef thru 0x02eb16 (0xe728 bytes) 000607 1768008871.639510 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6fe27a41 (stale) Mirror 6: 0x6fe27a41 (stale) Mirror 9: 0x6fe27a41 (stale) Mirror 11: 0x6fe27a41 (stale) Mirror 17: 0x6fe27a41 (stale) Mirror 18: 0x6fe27a41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6fe27a41 (stale) Mirror 6: 0x6fe27a41 (stale) Mirror 9: 0x6fe27a41 (stale) Mirror 11: 0x6fe27a41 (stale) Mirror 17: 0x6fe27a41 (stale) Mirror 18: 0x6fe27a41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1768008871.918034 mapwrite 0x037464 thru 0x03cb85 (0x5722 bytes) 000613 1768008871.970733 trunc from 0x040000 to 0x039481 (0x6b7e bytes) 000615 1768008871.988029 mapwrite 0x037d24 thru 0x03c048 (0x4325 bytes) 000616 1768008872.035263 mapread 0x00ca8a thru 0x01b85f (0xedd6 bytes) 000617 1768008872.051887 MIRROR_RESYNC 7 mirrors 000619 1768008872.755700 read 0x02cd33 thru 0x02e98b (0x1c59 bytes) 000620 1768008872.763023 mapwrite 0x013d45 thru 0x01b6b3 (0x796f bytes) 000622 1768008872.869560 mapwrite 0x01c98a thru 0x01d0ec (0x763 bytes) 000623 1768008872.904231 mapread 0x00bf57 thru 0x01242b (0x64d5 bytes) 000625 1768008872.910643 read 0x035bb3 thru 0x036603 (0xa51 bytes) 000626 1768008872.920638 mapread 0x02534a thru 0x0329d2 (0xd689 bytes) 000627 1768008872.921665 mapwrite 0x01a80f thru 0x022450 (0x7c42 bytes) 000628 1768008872.956656 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c049): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c049): Mirror 5: 0x5fc4109e (stale) Mirror 6: 0x5fc4109e (stale) Mirror 9: 0x5fc4109e (stale) Mirror 11: 0x5fc4109e (stale) Mirror 17: 0x5fc4109e (stale) Mirror 18: 0x5fc4109e (stale) CRC-32 checksum value for chunk [0, 0x3c049): Mirror 1: 0xe29c9bfa Mirror 19: 0xe29c9bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c049: skipped 000629 1768008873.573337 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c049): Mirror 5: 0x5fc4109e (stale) Mirror 6: 0x5fc4109e (stale) Mirror 11: 0x5fc4109e (stale) Mirror 17: 0x5fc4109e (stale) Mirror 18: 0x5fc4109e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c049): Mirror 5: 0x5fc4109e (stale) Mirror 6: 0x5fc4109e (stale) Mirror 11: 0x5fc4109e (stale) Mirror 17: 0x5fc4109e (stale) Mirror 18: 0x5fc4109e (stale) CRC-32 checksum value for chunk [0, 0x3c049): Mirror 1: 0xe29c9bfa Mirror 19: 0xe29c9bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c049: skipped 000630 1768008874.153429 mapwrite 0x00db99 thru 0x014595 (0x69fd bytes) 000631 1768008874.297013 read 0x00e65d thru 0x01cd50 (0xe6f4 bytes) 000633 1768008874.307934 trunc from 0x03c049 to 0x014900 (0x27748 bytes) 000634 1768008874.330403 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14901): Mirror 6: 0xd17e9c26 (stale) Mirror 11: 0xd17e9c26 (stale) Mirror 17: 0xd17e9c26 (stale) Mirror 18: 0xd17e9c26 (stale) Mirror 19: 0x93f9e014 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14901): Mirror 6: 0xd17e9c26 (stale) Mirror 11: 0xd17e9c26 (stale) Mirror 17: 0xd17e9c26 (stale) Mirror 18: 0xd17e9c26 (stale) Mirror 19: 0x93f9e014 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14901: skipped 000635 1768008874.589748 read 0x00b2da thru 0x014900 (0x9627 bytes) 000638 1768008874.596543 trunc from 0x014901 to 0x028dae (0x144ae bytes) 000639 1768008874.617762 mapread 0x00cb60 thru 0x01ba7e (0xef1f bytes) 000640 1768008874.636051 read 0x01f077 thru 0x028dae (0x9d38 bytes) 000641 1768008874.642326 read 0x000215 thru 0x008427 (0x8213 bytes) 000642 1768008874.643243 mapread 0x00b877 thru 0x015c32 (0xa3bc bytes) 000644 1768008874.644766 write 0x01ca24 thru 0x029e2c (0xd409 bytes) 000645 1768008874.647304 mapread 0x01078b thru 0x015589 (0x4dff bytes) 000646 1768008874.657545 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e2d): Mirror 6: 0x31602bdc (stale) Mirror 11: 0x31602bdc (stale) Mirror 17: 0x31602bdc (stale) Mirror 19: 0x8ae1656f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e2d): Mirror 6: 0x31602bdc (stale) Mirror 11: 0x31602bdc (stale) Mirror 17: 0x31602bdc (stale) Mirror 19: 0x8ae1656f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e2d: skipped 000647 1768008874.927004 mapread 0x01608c thru 0x024d81 (0xecf6 bytes) 000648 1768008874.951211 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e2d): Mirror 11: 0x31602bdc (stale) Mirror 17: 0x31602bdc (stale) Mirror 19: 0x8ae1656f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e2d): Mirror 11: 0x31602bdc (stale) Mirror 17: 0x31602bdc (stale) Mirror 19: 0x8ae1656f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e2d: skipped 000650 1768008875.164741 trunc from 0x029e2d to 0x016ef5 (0x12f37 bytes) 000651 1768008875.178303 read 0x00eeb9 thru 0x016ef5 (0x803d bytes) 000652 1768008875.184526 write 0x02d986 thru 0x0335e4 (0x5c5f bytes) 000655 1768008875.186434 write 0x016318 thru 0x01fac7 (0x97b0 bytes) 000656 1768008875.190850 read 0x02ae7a thru 0x031a3f (0x6bc6 bytes) 000658 1768008875.197675 trunc from 0x0335e5 to 0x036e54 (0x3870 bytes) 000659 1768008875.207174 mapread 0x003785 thru 0x0133e1 (0xfc5d bytes) 000660 1768008875.208214 read 0x014d22 thru 0x021cf4 (0xcfd3 bytes) 000662 1768008875.208933 mapread 0x001c4c thru 0x002cde (0x1093 bytes) 000664 1768008875.210118 mapread 0x0346ce thru 0x036e54 (0x2787 bytes) 000665 1768008875.214612 trunc from 0x036e55 to 0x0069a3 (0x304b1 bytes) 000667 1768008875.234822 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x69a4): Mirror 17: 0xbc37cd80 (stale) Mirror 19: 0xbc37cd80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x69a4): Mirror 17: 0xbc37cd80 (stale) Mirror 19: 0xbc37cd80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x69a4: skipped 000668 1768008875.440197 read 0x00321e thru 0x0069a3 (0x3786 bytes) 000669 1768008875.447060 trunc from 0x0069a4 to 0x034e45 (0x2e4a2 bytes) 000671 1768008875.454466 read 0x02c2a1 thru 0x02f6a3 (0x3403 bytes) 000672 1768008875.470740 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e46): Mirror 17: 0x2f7efe6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e46): Mirror 17: 0x2f7efe6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e46: skipped 000675 1768008875.639733 read 0x008bee thru 0x018035 (0xf448 bytes) 000677 1768008875.652982 mapwrite 0x03b66f thru 0x03ffff (0x4991 bytes) 000678 1768008875.712859 mapwrite 0x0057ee thru 0x006c8f (0x14a2 bytes) 000681 1768008875.741960 mapread 0x0069cb thru 0x00fff8 (0x962e bytes) 000682 1768008875.742960 trunc from 0x040000 to 0x0306b7 (0xf948 bytes) 000683 1768008875.757301 write 0x02ea56 thru 0x033e68 (0x5413 bytes) 000684 1768008875.773732 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33e69): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e69): Mirror 17: 0x2872fe6d (stale) CRC-32 checksum value for chunk [0, 0x33e69): Mirror 1: 0x9337e5f9 Mirror 18: 0x9337e5f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e69: skipped 000685 1768008876.199761 mapwrite 0x006cb0 thru 0x0136e1 (0xca32 bytes) 000689 1768008876.346484 mapwrite 0x031b24 thru 0x039754 (0x7c31 bytes) 000690 1768008876.428190 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39755): Mirror 18: 0x59b05aa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39755): Mirror 18: 0x59b05aa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39755: skipped 000691 1768008876.651283 MIRROR_RESYNC 2 mirrors 000693 1768008877.014220 mapread 0x034d2a thru 0x036e03 (0x20da bytes) 000694 1768008877.029532 mapwrite 0x01f1a3 thru 0x0254c4 (0x6322 bytes) 000695 1768008877.121209 write 0x030ba7 thru 0x03b452 (0xa8ac bytes) 000696 1768008877.128757 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b453): Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b453): Mirror 18: 0x327d49a0 (stale) CRC-32 checksum value for chunk [0, 0x3b453): Mirror 1: 0x9893811e Mirror 19: 0x9893811e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b453: skipped 000698 1768008877.628892 read 0x01907d thru 0x025cb5 (0xcc39 bytes) 000700 1768008877.652655 trunc from 0x03b453 to 0x01fe30 (0x1b622 bytes) 000701 1768008877.720967 mapwrite 0x017ec3 thru 0x019ee9 (0x2027 bytes) 000704 1768008877.805026 MIRROR_RESYNC 3 mirrors 000706 1768008878.280168 trunc from 0x01fe31 to 0x033144 (0x13314 bytes) 000708 1768008878.333814 trunc from 0x033145 to 0x020fc9 (0x1217b bytes) 000713 1768008878.352043 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20fca): Mirror 19: 0x4b7a300e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fca): Mirror 19: 0x4b7a300e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fca: skipped 000714 1768008878.496806 mapwrite 0x009bf1 thru 0x01865d (0xea6d bytes) 000715 1768008878.558083 trunc from 0x020fca to 0x02838b (0x73c2 bytes) 000717 1768008878.574350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) 000718 1768008878.641081 trunc from 0x02838c to 0x007408 (0x20f83 bytes) 000719 1768008878.651949 trunc from 0x007409 to 0x031da2 (0x2a99a bytes) 000720 1768008878.666668 mapwrite 0x03318d thru 0x03ffff (0xce73 bytes) 000721 1768008878.745877 mapread 0x000258 thru 0x008f8a (0x8d33 bytes) 000725 1768008878.755067 mapwrite 0x025da6 thru 0x03137b (0xb5d6 bytes) 000728 1768008878.790399 write 0x034eb4 thru 0x03ef8d (0xa0da bytes) 000731 1768008878.791725 mapwrite 0x025e02 thru 0x02e857 (0x8a56 bytes) 000735 1768008878.834170 write 0x014af9 thru 0x0246b8 (0xfbc0 bytes) 000736 1768008878.843655 write 0x037594 thru 0x03c38e (0x4dfb bytes) 000737 1768008878.844560 mapwrite 0x003156 thru 0x00cf1a (0x9dc5 bytes) 000744 1768008878.903523 mapwrite 0x00d9a7 thru 0x0126b7 (0x4d11 bytes) 000747 1768008878.946603 MIRROR_EXTEND to 2 mirrors 000748 1768008879.364979 MIRROR_EXTEND to 3 mirrors 000749 1768008879.714146 trunc from 0x040000 to 0x0198a6 (0x26759 bytes) 000750 1768008879.744374 mapread 0x010b9b thru 0x0198a6 (0x8d0c bytes) 000751 1768008879.756892 write 0x00eea4 thru 0x01181b (0x2978 bytes) 000757 1768008879.760500 read 0x00b92c thru 0x00f95f (0x4034 bytes) truncating to largest ever: 0x3fb7a 000758 1768008879.761076 trunc from 0x0198a7 to 0x03fb79 (0x262d3 bytes) 000759 1768008879.775829 mapwrite 0x01552b thru 0x019343 (0x3e19 bytes) 000760 1768008879.803352 write 0x02f66e thru 0x03e23c (0xebcf bytes) 000761 1768008879.816239 write 0x0224a9 thru 0x0246a2 (0x21fa bytes) 000762 1768008879.825417 write 0x00d723 thru 0x01d29e (0xfb7c bytes) 000763 1768008879.826766 read 0x03e02f thru 0x03fb79 (0x1b4b bytes) 000764 1768008879.832668 mapwrite 0x015e0c thru 0x01ef85 (0x917a bytes) 000766 1768008879.882445 trunc from 0x03fb7a to 0x01c26c (0x2390d bytes) 000768 1768008879.894451 write 0x02cdd1 thru 0x030479 (0x36a9 bytes) 000771 1768008879.896704 mapread 0x02e913 thru 0x030479 (0x1b67 bytes) 000772 1768008879.900935 MIRROR_RESYNC 3 mirrors 000775 1768008880.266155 mapwrite 0x030229 thru 0x03afb4 (0xad8c bytes) 000776 1768008880.352592 mapread 0x031064 thru 0x0370b4 (0x6051 bytes) 000780 1768008880.354737 trunc from 0x03afb5 to 0x01e6a7 (0x1c90d bytes) 000782 1768008880.382614 mapwrite 0x0013e0 thru 0x0058d6 (0x44f7 bytes) 000783 1768008880.430992 read 0x00bf73 thru 0x00dec0 (0x1f4e bytes) 000784 1768008880.431590 mapwrite 0x003695 thru 0x0053e6 (0x1d52 bytes) 000785 1768008880.460999 read 0x001d6a thru 0x0034dd (0x1774 bytes) 000787 1768008880.463014 trunc from 0x01e6a8 to 0x001e3f (0x1c868 bytes) 000788 1768008880.477649 mapwrite 0x00d097 thru 0x01b25a (0xe1c4 bytes) 000789 1768008880.536174 read 0x00a249 thru 0x00d64a (0x3402 bytes) 000790 1768008880.539854 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b25b): Mirror 3: 0x4c790652 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b25b): Mirror 3: 0x4c790652 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b25b: skipped 000791 1768008880.693712 trunc from 0x01b25b to 0x017909 (0x3951 bytes) 000795 1768008880.705511 mapread 0x00780c thru 0x00f61a (0x7e0f bytes) 000797 1768008880.720416 mapread 0x00084d thru 0x000a8d (0x241 bytes) 000798 1768008880.724605 trunc from 0x01790a to 0x0050a7 (0x12862 bytes) 000799 1768008880.736000 mapwrite 0x024cef thru 0x034488 (0xf79a bytes) 000801 1768008880.805331 mapread 0x01a2ad thru 0x0286f4 (0xe448 bytes) 000802 1768008880.814497 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34489): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34489): Mirror 3: 0x24cbc5e9 (stale) CRC-32 checksum value for chunk [0, 0x34489): Mirror 1: 0xe15ea6b7 Mirror 4: 0xe15ea6b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34489: skipped 000803 1768008881.159319 read 0x004f58 thru 0x012e98 (0xdf41 bytes) 000804 1768008881.166811 read 0x009270 thru 0x00ac1e (0x19af bytes) 000805 1768008881.171012 write 0x020d65 thru 0x022157 (0x13f3 bytes) 000807 1768008881.220719 trunc from 0x034489 to 0x010d35 (0x23753 bytes) 000810 1768008881.240947 mapwrite 0x02f197 thru 0x03c35e (0xd1c8 bytes) 000813 1768008881.304641 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c35f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c35f): Mirror 3: 0x9650c00b (stale) Mirror 4: 0x686ef57c (stale) CRC-32 checksum value for chunk [0, 0x3c35f): Mirror 1: 0xb9929fc Mirror 5: 0xb9929fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c35f: skipped 000815 1768008881.686182 mapwrite 0x0065a2 thru 0x00eed2 (0x8931 bytes) 000818 1768008881.804554 read 0x02d7bb thru 0x030768 (0x2fae bytes) 000819 1768008881.809761 write 0x01fc40 thru 0x01fce2 (0xa3 bytes) 000822 1768008881.823944 trunc from 0x03c35f to 0x01d414 (0x1ef4a bytes) 000823 1768008881.861233 mapwrite 0x0100eb thru 0x01ada5 (0xacbb bytes) 000824 1768008881.912414 write 0x00deeb thru 0x01b521 (0xd637 bytes) 000827 1768008881.920398 MIRROR_RESYNC 4 mirrors 000830 1768008882.412847 trunc from 0x01d415 to 0x02983e (0xc42a bytes) 000832 1768008882.487061 mapread 0x025d9c thru 0x028285 (0x24ea bytes) 000833 1768008882.502871 mapread 0x01f880 thru 0x022b05 (0x3286 bytes) 000836 1768008882.514980 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2983f): Mirror 3: 0xecc174e1 (stale) Mirror 5: 0xecc174e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2983f): Mirror 3: 0xecc174e1 (stale) Mirror 5: 0xecc174e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2983f: skipped 000837 1768008882.711202 write 0x022346 thru 0x024c4a (0x2905 bytes) 000838 1768008882.716934 mapwrite 0x023d36 thru 0x0259cb (0x1c96 bytes) 000839 1768008882.749538 mapread 0x01a55a thru 0x027deb (0xd892 bytes) 000841 1768008882.757216 mapread 0x013c3d thru 0x017e45 (0x4209 bytes) 000842 1768008882.757773 mapread 0x025cd7 thru 0x02983e (0x3b68 bytes) 000843 1768008882.758389 read 0x00fd4b thru 0x01a1ac (0xa462 bytes) 000844 1768008882.763989 mapwrite 0x02a978 thru 0x037bab (0xd234 bytes) 000846 1768008882.825538 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37bac): Mirror 5: 0xb2406ba1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bac): Mirror 5: 0xb2406ba1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bac: skipped 000847 1768008882.978157 mapread 0x02b64f thru 0x02bb66 (0x518 bytes) 000848 1768008882.982675 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37bac): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bac): Mirror 5: 0xb2406ba1 (stale) CRC-32 checksum value for chunk [0, 0x37bac): Mirror 1: 0x94ae2253 Mirror 6: 0x94ae2253 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bac: skipped 000849 1768008883.483482 mapread 0x010e6a thru 0x01ebd2 (0xdd69 bytes) 000850 1768008883.521625 write 0x01e807 thru 0x0211e6 (0x29e0 bytes) 000851 1768008883.572134 trunc from 0x037bac to 0x028e94 (0xed17 bytes) 000852 1768008883.583208 mapread 0x022e98 thru 0x028e94 (0x5ffd bytes) 000853 1768008883.588778 trunc from 0x028e95 to 0x01608c (0x12e08 bytes) 000854 1768008883.602000 mapread 0x009c27 thru 0x01608c (0xc466 bytes) 000857 1768008883.611131 write 0x019265 thru 0x0207e0 (0x757c bytes) 000860 1768008883.615460 mapwrite 0x036aff thru 0x03ffff (0x9501 bytes) 000861 1768008883.704673 mapread 0x037546 thru 0x0380ce (0xb89 bytes) 000862 1768008883.705531 mapwrite 0x02025d thru 0x02a53a (0xa2de bytes) 000865 1768008883.775042 write 0x01ade6 thru 0x026234 (0xb44f bytes) 000869 1768008883.778838 write 0x01ce73 thru 0x027265 (0xa3f3 bytes) 000873 1768008883.782201 mapread 0x00a5f8 thru 0x016799 (0xc1a2 bytes) 000874 1768008883.783393 trunc from 0x040000 to 0x024c14 (0x1b3eb bytes) 000875 1768008883.805438 mapwrite 0x028a15 thru 0x032518 (0x9b04 bytes) 000876 1768008883.879189 MIRROR_RESYNC 3 mirrors 000877 1768008884.330435 trunc from 0x032519 to 0x016e31 (0x1b6e7 bytes) 000879 1768008884.373374 mapread 0x005dd3 thru 0x010df1 (0xb01f bytes) 000881 1768008884.386072 trunc from 0x016e32 to 0x00a65e (0xc7d3 bytes) 000883 1768008884.399467 mapread 0x003742 thru 0x00a65e (0x6f1d bytes) 000884 1768008884.400512 mapwrite 0x02ba7d thru 0x0328c7 (0x6e4b bytes) 000885 1768008884.471215 mapwrite 0x0170be thru 0x01dafd (0x6a40 bytes) 000886 1768008884.520748 MIRROR_RESYNC 3 mirrors 000889 1768008884.873185 write 0x024447 thru 0x027c83 (0x383d bytes) 000890 1768008884.927393 write 0x01aafd thru 0x0287d3 (0xdcd7 bytes) 000891 1768008884.942148 mapwrite 0x02e24c thru 0x03a1d9 (0xbf8e bytes) 000892 1768008884.996847 write 0x00e21d thru 0x00fea8 (0x1c8c bytes) 000894 1768008885.002205 trunc from 0x03a1da to 0x0300d8 (0xa101 bytes) 000895 1768008885.011242 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x300d9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300d9): Mirror 5: 0x65b671ab (stale) Mirror 6: 0x65b671ab (stale) CRC-32 checksum value for chunk [0, 0x300d9): Mirror 1: 0x8132e679 Mirror 7: 0x8132e679 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300d9: skipped 000901 1768008885.481647 write 0x0123cb thru 0x018110 (0x5d46 bytes) 000904 1768008885.511935 read 0x017d23 thru 0x01d05a (0x5338 bytes) 000908 1768008885.518141 trunc from 0x0300d9 to 0x028c12 (0x74c6 bytes) 000909 1768008885.530085 mapread 0x001bd0 thru 0x00f7b3 (0xdbe4 bytes) 000912 1768008885.538769 mapread 0x01da29 thru 0x028c12 (0xb1ea bytes) 000913 1768008885.554387 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x368:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c13): Mirror 6: 0x5d1535a0 (stale) Mirror 7: 0x46d0a7e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c13): Mirror 6: 0x5d1535a0 (stale) Mirror 7: 0x46d0a7e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c13: skipped 000914 1768008885.704660 read 0x0210fe thru 0x028c12 (0x7b15 bytes) 000915 1768008885.708883 mapread 0x01f2db thru 0x01f6e8 (0x40e bytes) 000919 1768008885.713326 read 0x00cd03 thru 0x011c48 (0x4f46 bytes) 000922 1768008885.718441 read 0x00769c thru 0x014e80 (0xd7e5 bytes) 000923 1768008885.719135 mapwrite 0x0228b8 thru 0x027e30 (0x5579 bytes) 000925 1768008885.737900 write 0x00c255 thru 0x014649 (0x83f5 bytes) 000926 1768008885.742267 MIRROR_RESYNC 3 mirrors 000928 1768008886.064341 trunc from 0x028c13 to 0x0335ae (0xa99c bytes) 000930 1768008886.101751 write 0x032ea1 thru 0x03570b (0x286b bytes) 000932 1768008886.109049 mapread 0x033795 thru 0x03570b (0x1f77 bytes) 000934 1768008886.110512 mapread 0x010fd3 thru 0x0190e3 (0x8111 bytes) 000937 1768008886.123467 mapread 0x024cf3 thru 0x0263ad (0x16bb bytes) 000939 1768008886.129283 write 0x014293 thru 0x01ecfd (0xaa6b bytes) 000941 1768008886.130960 write 0x022c6b thru 0x031282 (0xe618 bytes) 000944 1768008886.141090 write 0x0097fa thru 0x0145a4 (0xadab bytes) 000945 1768008886.149283 MIRROR_RESYNC 3 mirrors 000948 1768008886.451845 read 0x000d68 thru 0x007b90 (0x6e29 bytes) 000949 1768008886.457211 write 0x003c06 thru 0x008d6a (0x5165 bytes) 000950 1768008886.486127 mapread 0x01844c thru 0x01fe50 (0x7a05 bytes) 000951 1768008886.491806 mapread 0x00f005 thru 0x015574 (0x6570 bytes) 000952 1768008886.497019 write 0x00f885 thru 0x01cc00 (0xd37c bytes) 000953 1768008886.498316 trunc from 0x03570c to 0x02bcec (0x9a1f bytes) 000954 1768008886.520021 mapread 0x026f32 thru 0x02bcec (0x4dbb bytes) 000955 1768008886.525243 trunc from 0x02bced to 0x025f3b (0x5db1 bytes) 000957 1768008886.538148 mapread 0x014ef8 thru 0x023202 (0xe30b bytes) 000959 1768008886.542464 MIRROR_RESYNC 3 mirrors 000960 1768008886.838559 mapread 0x019637 thru 0x02172d (0x80f7 bytes) 000961 1768008886.848572 read 0x0097f9 thru 0x01287b (0x9083 bytes) 000963 1768008886.853920 write 0x03a19d thru 0x03ffff (0x5e63 bytes) 000964 1768008886.886226 mapread 0x0315fe thru 0x03ffff (0xea02 bytes) 000965 1768008886.896242 read 0x02b530 thru 0x033ffc (0x8acd bytes) 000968 1768008886.898674 mapwrite 0x0399a9 thru 0x03ffff (0x6657 bytes) 000969 1768008886.933231 mapwrite 0x02b44a thru 0x032395 (0x6f4c bytes) 000970 1768008886.961237 trunc from 0x040000 to 0x0063b0 (0x39c4f bytes) 000972 1768008886.981186 write 0x019344 thru 0x02304d (0x9d0a bytes) 000975 1768008886.983917 read 0x005ad1 thru 0x00c63c (0x6b6c bytes) 000976 1768008886.991660 trunc from 0x02304e to 0x001e7c (0x211d1 bytes) 000978 1768008887.008699 mapread 0x001268 thru 0x001e7c (0xc15 bytes) 000979 1768008887.009558 mapread 0x00026e thru 0x001e7c (0x1c0f bytes) 000980 1768008887.010224 read 0x001745 thru 0x001e7c (0x738 bytes) 000983 1768008887.013783 mapread 0x00029e thru 0x001e7c (0x1bdf bytes) 000984 1768008887.014801 mapread 0x000f4f thru 0x001e7c (0xf2e bytes) 000985 1768008887.015714 trunc from 0x001e7d to 0x0290be (0x27242 bytes) 000986 1768008887.025135 trunc from 0x0290bf to 0x015a0d (0x136b1 bytes) 000987 1768008887.036464 mapread 0x003a43 thru 0x011960 (0xdf1e bytes) 000988 1768008887.042423 write 0x025ea3 thru 0x02b6c5 (0x5823 bytes) 000989 1768008887.043660 mapwrite 0x03fd11 thru 0x03ffff (0x2ef bytes) 000990 1768008887.087699 mapwrite 0x016181 thru 0x01a92d (0x47ad bytes) 000991 1768008887.107848 read 0x02dbd8 thru 0x031c6c (0x4095 bytes) 000992 1768008887.108347 read 0x023b12 thru 0x03170b (0xdbfa bytes) 000996 1768008887.110658 mapread 0x01ae4a thru 0x0279fe (0xcbb5 bytes) 000997 1768008887.111612 read 0x020bf5 thru 0x02150e (0x91a bytes) 000998 1768008887.112170 mapwrite 0x0229f2 thru 0x02f221 (0xc830 bytes) 000999 1768008887.137607 mapwrite 0x017929 thru 0x02173b (0x9e13 bytes) 001000 1768008887.158233 read 0x03bcb6 thru 0x03ffff (0x434a bytes) All operations completed A-OK! PASS 70a (44s) == sanity-flr test 71: check mirror extend parent fid ==== 20:34:49 (1768008889) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 20:34:51 (1768008891) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10162 s, 20.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107891 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 20:34:55 (1768008895) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5268288 .. Extending file size to 6904224 .. failed Extending file size to 8531232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8639616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8758560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9082464 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8650144 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8802560 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9145280 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56399 56400 56403 56404 56405 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:36:03 (1768008963) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2118016 .. Extending file size to 6631168 .. done Extending file size to 2512064 .. Extending file size to 7143904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5373408 .. Extending file size to 8292928 .. Extending file size to 8707072 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9298272 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8717408 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9236576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59178 59179 59181 59183 59185 failed 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:37:10 (1768009030) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0432082 s, 48.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134526 s, 15.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xefb57f29 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:37:18 (1768009038) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0583627 s, 35.9 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 20:37:21 (1768009041) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3b1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bc:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x4b7:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:37:24 (1768009044) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0647802 s, 32.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:37:27 (1768009047) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105458 s, 19.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bb:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:37:30 (1768009050) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0777948 s, 27.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bd:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:37:55 (1768009075) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344585 s, 30.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c2:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:38:36 (1768009116) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0527796 s, 39.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:39:24 (1768009164) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. Creating new pool test_204f oleg460-server: Pool lustre.test_204f created Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.058943 s, 35.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg460-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:40:22 (1768009222) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:40:25 (1768009225) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:40:28 (1768009228) oleg460-server: Pool lustre.test_206 created Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg460-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:40:44 (1768009244) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276188 s, 38.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126368 s, 83.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9a75695c Mirror 3: 0x9a75695c PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:40:48 (1768009248) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:40:49 (1768009249) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:40:50 (1768009250) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0537246 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x900000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa90000 Read this (4 bytes): BBBB 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:41:09 (1768009269) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0061479 s, 1.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:41:11 (1768009271) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201583 s, 52.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3d6 lmm_fid: [0x200000404:0x3d6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3d6 lmm_fid: [0x200000404:0x3d6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4cf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834902 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-09 20:41:12.000000000 -0500 Modify: 2026-01-09 20:41:12.000000000 -0500 Change: 2026-01-09 20:41:12.000000000 -0500 Birth: 2026-01-09 20:41:12.000000000 -0500 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:41:15 (1768009275) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:41:16 (1768009276) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.169411 s, 242 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0265839 s, 154 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 9 20:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 9 20:41 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:41:19 (1768009279) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:41:22 (1768009282) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 1014 sec =========== 20:41:25 (1768009285) === sanity-flr: start cleanup 20:41:26 (1768009286) === === sanity-flr: finish cleanup 20:41:29 (1768009289) ===