-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 12:00:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg117-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:01:00 (1788796860) === oleg117-client.virtnet: executing check_config_client /mnt/lustre oleg117-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg117-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d7bd8bd2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d7bd8bd2000.idle_timeout=debug disable quota as required oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:01:45 (1788796905) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 12:01:49 (1788796909) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:02:10 (1788796930) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:02:15 (1788796935) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:02:20 (1788796940) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:02:53 (1788796973) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:02:56 (1788796976) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:03:00 (1788796980) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d7bd8bd2000.stats=clear osc.lustre-OST0001-osc-ffff8d7bd8bd2000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:03:10 (1788796990) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:03:27 (1788797007) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:03:37 (1788797017) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:03:41 (1788797021) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:03:51 (1788797031) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 12:04:01 (1788797041) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:04:09 (1788797049) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:04:14 (1788797054) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:04:24 (1788797064) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:04:37 (1788797077) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:04:45 (1788797085) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.95934 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224706 s, 4.7 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:04:59 (1788797099) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.171124 s, 42.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:05:08 (1788797108) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:05:19 (1788797119) Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:06:41 (1788797201) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.292258 s, 14.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:07:26 (1788797246) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:08:54 (1788797334) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.71716 s, 9.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 12:09 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.454s user 0m0.001s sys 0m0.160s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.310s user 0m0.001s sys 0m0.211s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.396s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d7bd8bd2000.ost_server_uuid in FULL state after 0 sec PASS 33b (110s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:10:44 (1788797444) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:10:49 (1788797449) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:10:53 (1788797453) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:10:56 (1788797456) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 12:11:05 (1788797465) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.664147 s, 18.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:11:20 (1788797480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446241 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542656 s, 19.3 MB/s PASS 36b (11s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:11:31 (1788797491) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:11:41 (1788797501) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000993937 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000972962 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00140179 s, 730 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157239 s, 651 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 12:11:55 (1788797515) create test files with size 15752709 .. 0321433ec39c478f671d5a1009f29ca6 - 98ab7308c32910a601f8ca616658c5a4 - f9883b5a349ee273a4c8270249da8681 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 12:12:26 (1788797546) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:12:45 (1788797565) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:12:52 (1788797572) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149553 s, 7.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466579 s, 22.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 12:13:01 (1788797581) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148992 s, 14.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.475115 s, 8.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187988 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 12:13:20 (1788797600) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:13:25 (1788797605) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:13:29 (1788797609) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024255 s, 43.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 12:13:39 (1788797619) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:13:43 (1788797623) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:13:55 (1788797635) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.369331 s, 28.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:14:09 (1788797649) llite.lustre-ffff8d7bd0a25800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.338257 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d7bd1147000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:14:25 (1788797665) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:15:11 (1788797711) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (52s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:16:03 (1788797763) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 12:16:08 (1788797768) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.181226 s, 17.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.147522 s, 21.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4250856b (nosync,stale) Mirror 2: 0x4250856b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 389d8c333c356b105113993b07cae1ec - mirror 1 checksum: a1e8cd46002c29225a8a5af0c7508f1d - mirror 2 checksum: 389d8c333c356b105113993b07cae1ec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T12:16:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T12:16:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 389d8c333c356b105113993b07cae1ec - mirror 1 checksum: a1e8cd46002c29225a8a5af0c7508f1d - mirror 2 checksum: a1e8cd46002c29225a8a5af0c7508f1d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T12:16:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T12:16:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:16:22 (1788797782) Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 12:16:30 (1788797790) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server Failover mds1 to oleg117-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 12:17:03 (1788797823) targets are mounted 12:17:03 (1788797823) facet_failover done oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375109 s, 28.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:17:22 (1788797842) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:17:27 (1788797847) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:17:30 (1788797850) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00665873 s, 2.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:17:39 (1788797859) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:17:43 (1788797863) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00863273 s, 474 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.12329 s, 33.2 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:17:53 (1788797873) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:18:20 (1788797900) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:18:53 (1788797933) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:19:18 (1788797958) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:19:44 (1788797984) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 12:19:53 (1788797993) mirror create pid 54375 mirror split pid 54376 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54375 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54376 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:21:00 (1788798060) Seed set to 8062 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1788798063.138783 write 0x01f274 thru 0x02196c (0x26f9 bytes) truncating to largest ever: 0x2b0bb 000007 1788798063.154420 trunc from 0x02196d to 0x02b0ba (0x974e bytes) 000009 1788798063.199229 write 0x019303 thru 0x022a5d (0x975b bytes) 000010 1788798063.225252 read 0x006ba3 thru 0x00dc2d (0x708b bytes) 000014 1788798063.256242 punch from 0x007ee4 to 0x016a8a (0xeba7 bytes) truncating to largest ever: 0x39a7a 000015 1788798063.283295 trunc from 0x02b0bb to 0x039a79 (0xe9bf bytes) 000017 1788798063.313537 mapwrite 0x032462 thru 0x03e77e (0xc31d bytes) 000018 1788798063.515305 write 0x01ab16 thru 0x025c8c (0xb177 bytes) 000019 1788798063.526002 punch from 0x0398c2 to 0x03e77e (0x4ebd bytes) 000021 1788798063.589842 MIRROR_EXTEND to 2 mirrors 000023 1788798064.523909 read 0x013f1e thru 0x0144cf (0x5b2 bytes) 000024 1788798064.547331 mapwrite 0x015668 thru 0x0195c5 (0x3f5e bytes) 000025 1788798064.903692 mapread 0x026b05 thru 0x033375 (0xc871 bytes) 000026 1788798064.935333 trunc from 0x03e77f to 0x015ba9 (0x28bd5 bytes) 000028 1788798064.995221 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15baa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15baa): Mirror 2: 0x87a319cc (stale) CRC-32 checksum value for chunk [0, 0x15baa): Mirror 1: 0xf5cab7e2 Mirror 3: 0xf5cab7e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15baa: skipped 000029 1788798066.206606 mapread 0x0011da thru 0x002a77 (0x189e bytes) 000030 1788798066.235102 write 0x00ea82 thru 0x0186c1 (0x9c40 bytes) 000031 1788798066.353291 punch from 0x00dddd to 0x0186c1 (0xa8e5 bytes) 000032 1788798066.371892 trunc from 0x0186c2 to 0x02636f (0xdcae bytes) 000035 1788798066.399431 read 0x0018af thru 0x008248 (0x699a bytes) 000036 1788798066.413900 write 0x00c191 thru 0x0156a7 (0x9517 bytes) 000037 1788798066.427369 trunc from 0x026370 to 0x0057a7 (0x20bc8 bytes) 000038 1788798066.459243 MIRROR_RESYNC 3 mirrors 000039 1788798067.523034 trunc from 0x0057a8 to 0x031f2a (0x2c783 bytes) 000040 1788798067.657611 read 0x015925 thru 0x019ee7 (0x45c3 bytes) 000042 1788798067.679134 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31f2b): Mirror 2: 0x68f84c27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f2b): Mirror 2: 0x68f84c27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f2b: skipped 000046 1788798068.066629 read 0x02d6b3 thru 0x031f2a (0x4878 bytes) 000047 1788798068.073147 write 0x00ab95 thru 0x011b9b (0x7007 bytes) 000049 1788798068.080611 read 0x0047ba thru 0x005d45 (0x158c bytes) 000051 1788798068.090392 trunc from 0x031f2b to 0x02d637 (0x48f3 bytes) 000053 1788798068.139629 mapwrite 0x02ff43 thru 0x0381f7 (0x82b5 bytes) 000054 1788798068.279878 mapwrite 0x033ca1 thru 0x03f5a4 (0xb904 bytes) 000056 1788798068.387438 write 0x009098 thru 0x01901b (0xff84 bytes) 000057 1788798068.392344 punch from 0x0383c0 to 0x03b3f2 (0x3033 bytes) 000058 1788798068.412381 read 0x014360 thru 0x01917d (0x4e1e bytes) 000061 1788798068.416606 read 0x00891c thru 0x01111a (0x87ff bytes) 000062 1788798068.419117 mapwrite 0x025906 thru 0x027a8b (0x2186 bytes) 000066 1788798068.496057 trunc from 0x03f5a5 to 0x02a796 (0x14e0e bytes) 000067 1788798068.530305 read 0x00d406 thru 0x013ec1 (0x6abc bytes) 000068 1788798068.531361 write 0x0144a6 thru 0x023ba2 (0xf6fd bytes) 000069 1788798068.534685 mapwrite 0x01a2eb thru 0x0297a1 (0xf4b7 bytes) 000071 1788798068.635844 MIRROR_RESYNC 2 mirrors 000072 1788798069.692243 mapread 0x029123 thru 0x02a796 (0x1674 bytes) 000075 1788798069.723339 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000076 1788798069.927455 mapread 0x018a24 thru 0x01f128 (0x6705 bytes) 000077 1788798069.974494 mapwrite 0x00c739 thru 0x00f5e5 (0x2ead bytes) 000078 1788798070.107160 write 0x0205c6 thru 0x0249fa (0x4435 bytes) 000079 1788798070.117890 mapread 0x01ffe6 thru 0x02a796 (0xa7b1 bytes) 000080 1788798070.161900 mapread 0x005b59 thru 0x00cd29 (0x71d1 bytes) 000081 1788798070.162966 mapwrite 0x00b6df thru 0x00f088 (0x39aa bytes) 000082 1788798070.251913 mapread 0x008397 thru 0x00ce7b (0x4ae5 bytes) 000083 1788798070.252764 write 0x01fe24 thru 0x02839b (0x8578 bytes) 000084 1788798070.253962 mapwrite 0x016640 thru 0x0218d7 (0xb298 bytes) 000085 1788798070.378341 write 0x01b197 thru 0x02a8bf (0xf729 bytes) 000088 1788798070.380329 mapread 0x0000ea thru 0x0085e1 (0x84f8 bytes) 000089 1788798070.401896 punch from 0x01f42e to 0x02a8bf (0xb492 bytes) 000091 1788798070.421370 read 0x010954 thru 0x018249 (0x78f6 bytes) 000093 1788798070.423151 mapread 0x021b37 thru 0x02a8bf (0x8d89 bytes) 000095 1788798070.504500 trunc from 0x02a8c0 to 0x028163 (0x275c bytes) 000096 1788798070.538715 mapread 0x02204a thru 0x0252d0 (0x3287 bytes) 000097 1788798070.548503 mapwrite 0x00a224 thru 0x0159e5 (0xb7c2 bytes) 000099 1788798070.642298 mapwrite 0x02c3a2 thru 0x02d32b (0xf8a bytes) 000102 1788798070.810604 read 0x004d5e thru 0x00a812 (0x5ab5 bytes) 000103 1788798070.811519 punch from 0x009cd1 to 0x014818 (0xab48 bytes) 000107 1788798070.820414 read 0x009764 thru 0x009a17 (0x2b4 bytes) 000108 1788798070.831497 punch from 0x0025b0 to 0x00dccb (0xb71c bytes) 000109 1788798070.835944 trunc from 0x02d32c to 0x011888 (0x1baa3 bytes) 000110 1788798070.871596 read 0x00ff31 thru 0x011888 (0x1958 bytes) 000111 1788798070.872713 write 0x0082d0 thru 0x010f7b (0x8cac bytes) 000112 1788798070.886237 mapread 0x00caa5 thru 0x011888 (0x4de4 bytes) 000114 1788798070.888872 write 0x020e4e thru 0x02b7bd (0xa970 bytes) 000115 1788798070.891750 trunc from 0x02b7be to 0x028b48 (0x2c75 bytes) 000116 1788798070.935727 mapwrite 0x0187eb thru 0x0202b3 (0x7ac9 bytes) 000117 1788798071.017631 mapwrite 0x015235 thru 0x019385 (0x4151 bytes) 000118 1788798071.075506 trunc from 0x028b49 to 0x032e0e (0xa2c6 bytes) 000119 1788798071.100689 read 0x000fc8 thru 0x0037bb (0x27f4 bytes) 000120 1788798071.102703 trunc from 0x032e0f to 0x02c31d (0x6af1 bytes) 000124 1788798071.135911 mapwrite 0x03db03 thru 0x03ffff (0x24fd bytes) 000126 1788798071.321812 mapread 0x011c6e thru 0x01d7d0 (0xbb63 bytes) 000127 1788798071.322928 mapread 0x039d18 thru 0x03f569 (0x5852 bytes) 000129 1788798071.325008 punch from 0x008923 to 0x00e779 (0x5e57 bytes) 000130 1788798071.333806 mapread 0x030b4b thru 0x03a733 (0x9be9 bytes) 000131 1788798071.334843 write 0x001dcd thru 0x002168 (0x39c bytes) 000132 1788798071.335739 mapread 0x010120 thru 0x0196b9 (0x959a bytes) 000133 1788798071.348471 trunc from 0x040000 to 0x0112f5 (0x2ed0a bytes) 000134 1788798071.377724 mapread 0x001402 thru 0x004931 (0x3530 bytes) 000135 1788798071.381233 trunc from 0x0112f6 to 0x00ec67 (0x268e bytes) 000136 1788798071.405429 trunc from 0x00ec68 to 0x014bbf (0x5f58 bytes) 000137 1788798071.431742 write 0x038360 thru 0x03ffff (0x7ca0 bytes) 000139 1788798071.433513 trunc from 0x040000 to 0x026eba (0x19145 bytes) 000140 1788798071.471354 trunc from 0x026ebb to 0x01bd28 (0xb192 bytes) 000142 1788798071.491221 mapwrite 0x03c020 thru 0x03ffff (0x3fe0 bytes) 000143 1788798071.722269 read 0x013bdb thru 0x0228cf (0xecf5 bytes) 000144 1788798071.753766 mapwrite 0x02f3b1 thru 0x0344bb (0x510b bytes) 000145 1788798071.883791 read 0x03610a thru 0x03ba1a (0x5911 bytes) 000146 1788798071.884495 trunc from 0x040000 to 0x02750f (0x18af0 bytes) 000149 1788798071.914538 trunc from 0x027510 to 0x01ffde (0x7531 bytes) 000150 1788798071.937327 punch from 0x00a142 to 0x00b2b0 (0x116f bytes) 000158 1788798071.952949 mapread 0x010d26 thru 0x0121b1 (0x148c bytes) 000159 1788798071.966832 MIRROR_EXTEND to 2 mirrors 000160 1788798073.106945 punch from 0x00381d to 0x006ccb (0x34af bytes) 000162 1788798073.166164 write 0x00894e thru 0x014116 (0xb7c9 bytes) 000164 1788798073.185006 read 0x01c522 thru 0x01ffde (0x3abd bytes) 000166 1788798073.200609 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ffdf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ffdf): Mirror 2: 0xed24ec6c (stale) CRC-32 checksum value for chunk [0, 0x1ffdf): Mirror 1: 0xc4767ab4 Mirror 3: 0xc4767ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ffdf: skipped 000167 1788798074.160336 punch from 0x00f1ef to 0x014667 (0x5479 bytes) 000169 1788798074.245309 mapwrite 0x021b8b thru 0x02213c (0x5b2 bytes) 000170 1788798074.344520 trunc from 0x02213d to 0x009ed2 (0x1826a bytes) 000172 1788798074.380635 mapwrite 0x02cfe2 thru 0x02f6ca (0x26e9 bytes) 000173 1788798074.441259 punch from 0x00d60d to 0x01d31d (0xfd11 bytes) 000174 1788798074.458625 trunc from 0x02f6cb to 0x02b365 (0x4365 bytes) 000176 1788798074.482363 write 0x0143e8 thru 0x01b9fd (0x7616 bytes) 000177 1788798074.492812 mapwrite 0x02f395 thru 0x02fac1 (0x72d bytes) 000178 1788798074.575362 mapread 0x013977 thru 0x016533 (0x2bbd bytes) 000179 1788798074.584372 trunc from 0x02fac2 to 0x034168 (0x46a7 bytes) 000181 1788798074.601280 mapread 0x01b762 thru 0x024874 (0x9113 bytes) truncating to largest ever: 0x3e01f 000182 1788798074.615208 trunc from 0x034169 to 0x03e01e (0x9eb6 bytes) 000183 1788798074.647414 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 3: 0x95f6882b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 3: 0x95f6882b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000184 1788798074.900626 MIRROR_RESYNC 2 mirrors 000185 1788798075.597037 read 0x000240 thru 0x00bbde (0xb99f bytes) 000188 1788798075.611402 read 0x034a20 thru 0x03e01e (0x95ff bytes) 000189 1788798075.622271 mapread 0x02e394 thru 0x0340a4 (0x5d11 bytes) 000190 1788798075.626963 MIRROR_EXTEND to 3 mirrors 000191 1788798076.600750 mapread 0x03305a thru 0x03d493 (0xa43a bytes) 000192 1788798076.639522 read 0x03241c thru 0x033779 (0x135e bytes) 000193 1788798076.639972 mapread 0x033233 thru 0x03b8f7 (0x86c5 bytes) 000194 1788798076.643575 punch from 0x019b30 to 0x027d7a (0xe24b bytes) 000195 1788798076.832465 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: 0x672345b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: 0x672345b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000197 1788798077.143261 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: 0x672345b8 (stale) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 1: 0xf5a00ebb Mirror 5: 0xf5a00ebb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000198 1788798078.193967 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: 0x672345b8 (stale) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 1: 0xf5a00ebb Mirror 5: 0xf5a00ebb Mirror 6: 0xf5a00ebb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000199 1788798079.238492 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 4: 0x672345b8 (stale) CRC-32 checksum value for chunk [0, 0x3e01f): Mirror 1: 0xf5a00ebb Mirror 5: 0xf5a00ebb Mirror 6: 0xf5a00ebb Mirror 7: 0xf5a00ebb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e01f: skipped 000201 1788798080.399578 mapwrite 0x038d6f thru 0x03ffff (0x7291 bytes) 000202 1788798080.616469 trunc from 0x040000 to 0x02faca (0x10535 bytes) 000203 1788798080.654670 read 0x019230 thru 0x01d78d (0x455e bytes) 000205 1788798080.669897 mapread 0x00725b thru 0x00866f (0x1415 bytes) 000207 1788798080.688222 punch from 0x0112e5 to 0x012906 (0x1622 bytes) 000208 1788798080.703467 trunc from 0x02facb to 0x01feb5 (0xfc15 bytes) 000210 1788798080.771274 mapread 0x00e28e thru 0x0193eb (0xb15e bytes) 000211 1788798080.788508 trunc from 0x01feb6 to 0x02553b (0x5686 bytes) 000212 1788798080.819080 mapwrite 0x01a190 thru 0x023b70 (0x99e1 bytes) 000214 1788798080.904864 trunc from 0x02553c to 0x031142 (0xbc07 bytes) 000215 1788798080.934806 trunc from 0x031143 to 0x024952 (0xc7f0 bytes) 000216 1788798080.956777 punch from 0x01579a to 0x024560 (0xedc7 bytes) 000217 1788798080.967415 mapwrite 0x02ddd8 thru 0x030e3e (0x3067 bytes) 000219 1788798081.131478 write 0x02fa39 thru 0x0384e7 (0x8aaf bytes) 000221 1788798081.138477 mapread 0x032696 thru 0x0384e7 (0x5e52 bytes) 000222 1788798081.139620 mapread 0x0115c9 thru 0x015651 (0x4089 bytes) 000224 1788798081.181901 MIRROR_RESYNC 5 mirrors 000225 1788798082.300831 MIRROR_EXTEND to 6 mirrors 000226 1788798083.277998 trunc from 0x0384e8 to 0x0100b1 (0x28436 bytes) 000228 1788798083.449508 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x100b2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x100b2): Mirror 4: 0x5393fe84 (stale) Mirror 5: 0x5393fe84 (stale) Mirror 6: 0x5393fe84 (stale) Mirror 7: 0x5393fe84 (stale) Mirror 8: 0x5393fe84 (stale) CRC-32 checksum value for chunk [0, 0x100b2): Mirror 1: 0x5393fe84 Mirror 9: 0x5393fe84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x100b2: skipped 000229 1788798084.725465 mapwrite 0x0099b2 thru 0x016fc5 (0xd614 bytes) 000230 1788798085.103345 mapwrite 0x016091 thru 0x01db7f (0x7aef bytes) 000231 1788798085.279005 write 0x0252c1 thru 0x0263b7 (0x10f7 bytes) 000232 1788798085.280045 mapwrite 0x03ed08 thru 0x03ffff (0x12f8 bytes) 000235 1788798085.381902 mapwrite 0x01fad9 thru 0x0235b2 (0x3ada bytes) 000236 1788798085.454930 read 0x02dece thru 0x03964d (0xb780 bytes) 000238 1788798085.472031 read 0x02123e thru 0x0296fd (0x84c0 bytes) 000239 1788798085.482474 mapwrite 0x0094d1 thru 0x01748c (0xdfbc bytes) 000240 1788798085.531571 write 0x000f6c thru 0x0079da (0x6a6f bytes) 000241 1788798085.536152 read 0x023b3a thru 0x02b09a (0x7561 bytes) 000242 1788798085.537301 mapread 0x019448 thru 0x01ffbc (0x6b75 bytes) 000243 1788798085.543366 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1826b194 (stale) Mirror 5: 0x1826b194 (stale) Mirror 6: 0x1826b194 (stale) Mirror 7: 0x1826b194 (stale) Mirror 8: 0x1826b194 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1826b194 (stale) Mirror 5: 0x1826b194 (stale) Mirror 6: 0x1826b194 (stale) Mirror 7: 0x1826b194 (stale) Mirror 8: 0x1826b194 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000244 1788798086.092135 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1826b194 (stale) Mirror 5: 0x1826b194 (stale) Mirror 6: 0x1826b194 (stale) Mirror 7: 0x1826b194 (stale) Mirror 8: 0x1826b194 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18a49065 Mirror 9: 0x18a49065 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000245 1788798087.361808 trunc from 0x040000 to 0x0086b2 (0x3794d bytes) 000246 1788798087.473122 mapwrite 0x0145da thru 0x020df6 (0xc81d bytes) 000247 1788798087.580450 trunc from 0x020df7 to 0x029713 (0x891d bytes) 000248 1788798087.600515 write 0x02de03 thru 0x0302f4 (0x24f2 bytes) 000250 1788798087.602295 mapread 0x021c55 thru 0x02bc4a (0x9ff6 bytes) 000251 1788798087.617743 mapwrite 0x03200d thru 0x03ff53 (0xdf47 bytes) 000254 1788798087.702250 write 0x023053 thru 0x02eedf (0xbe8d bytes) 000255 1788798087.706381 mapwrite 0x027381 thru 0x0370c7 (0xfd47 bytes) 000256 1788798087.789917 read 0x0020f9 thru 0x00e44e (0xc356 bytes) 000257 1788798087.815104 trunc from 0x03ff54 to 0x0346dd (0xb876 bytes) 000258 1788798087.867021 mapwrite 0x02a8e8 thru 0x02eca1 (0x43ba bytes) 000259 1788798087.940616 trunc from 0x0346de to 0x0293c2 (0xb31b bytes) 000261 1788798087.975128 punch from 0x004b7f to 0x00b9a1 (0x6e23 bytes) 000263 1788798087.984718 mapwrite 0x02ea5b thru 0x0303fd (0x19a3 bytes) 000264 1788798088.070316 mapwrite 0x02f410 thru 0x032958 (0x3549 bytes) 000265 1788798088.160213 MIRROR_RESYNC 7 mirrors 000266 1788798089.920636 mapread 0x0280c9 thru 0x032958 (0xa890 bytes) 000267 1788798089.945977 punch from 0x02eb84 to 0x031396 (0x2813 bytes) 000268 1788798090.018533 trunc from 0x032959 to 0x02fffa (0x295e bytes) 000269 1788798090.037924 trunc from 0x02fffb to 0x01cdc6 (0x13234 bytes) 000271 1788798090.049929 trunc from 0x01cdc7 to 0x025c9a (0x8ed4 bytes) 000272 1788798090.064385 trunc from 0x025c9b to 0x02a750 (0x4ab6 bytes) 000273 1788798090.090155 mapread 0x01b674 thru 0x02498d (0x931a bytes) 000275 1788798090.125437 mapwrite 0x0333c2 thru 0x039204 (0x5e43 bytes) 000276 1788798090.198283 punch from 0x02feb2 to 0x036981 (0x6ad0 bytes) 000277 1788798090.204828 trunc from 0x039205 to 0x027d44 (0x114c0 bytes) 000279 1788798090.222461 mapread 0x01d515 thru 0x01fc7d (0x2769 bytes) 000281 1788798090.223960 punch from 0x008167 to 0x010608 (0x84a2 bytes) 000282 1788798090.230596 mapwrite 0x004078 thru 0x008026 (0x3faf bytes) 000283 1788798090.325517 read 0x024a42 thru 0x027d44 (0x3303 bytes) 000284 1788798090.326753 mapwrite 0x02bf38 thru 0x038018 (0xc0e1 bytes) 000286 1788798090.422402 write 0x039dcc thru 0x03f728 (0x595d bytes) 000287 1788798090.429296 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f729): Mirror 4: 0xb8da185f (stale) Mirror 5: 0xb8da185f (stale) Mirror 6: 0xb8da185f (stale) Mirror 7: 0xb8da185f (stale) Mirror 8: 0xb8da185f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f729): Mirror 4: 0xb8da185f (stale) Mirror 5: 0xb8da185f (stale) Mirror 6: 0xb8da185f (stale) Mirror 7: 0xb8da185f (stale) Mirror 8: 0xb8da185f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f729: skipped 000289 1788798090.885039 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f729): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f729): Mirror 4: 0xb8da185f (stale) Mirror 5: 0xb8da185f (stale) Mirror 6: 0xb8da185f (stale) Mirror 7: 0xb8da185f (stale) Mirror 8: 0xb8da185f (stale) CRC-32 checksum value for chunk [0, 0x3f729): Mirror 1: 0x88335a3d Mirror 9: 0x88335a3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f729: skipped 000290 1788798092.292105 mapread 0x006811 thru 0x00af5c (0x474c bytes) 000291 1788798092.348419 write 0x00873d thru 0x00ff62 (0x7826 bytes) 000293 1788798092.442430 trunc from 0x03f729 to 0x00a4ff (0x35229 bytes) 000294 1788798092.465203 read 0x007995 thru 0x00a4ff (0x2b6b bytes) 000296 1788798092.475818 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: 0x55da7468 (stale) Mirror 5: 0x55da7468 (stale) Mirror 6: 0x55da7468 (stale) Mirror 7: 0x55da7468 (stale) Mirror 8: 0x55da7468 (stale) Mirror 9: 0x12487540 (stale) CRC-32 checksum value for chunk [0, 0xa500): Mirror 1: 0x55752926 Mirror 10: 0x55752926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa500: skipped 000297 1788798093.591615 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: 0x55da7468 (stale) Mirror 5: 0x55da7468 (stale) Mirror 6: 0x55da7468 (stale) Mirror 7: 0x55da7468 (stale) Mirror 8: 0x55da7468 (stale) Mirror 9: 0x12487540 (stale) CRC-32 checksum value for chunk [0, 0xa500): Mirror 1: 0x55752926 Mirror 10: 0x55752926 Mirror 11: 0x55752926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa500: skipped 000298 1788798094.544127 read 0x009568 thru 0x00a4ff (0xf98 bytes) 000299 1788798094.549239 read 0x008381 thru 0x00a4ff (0x217f bytes) 000300 1788798094.560210 read 0x002567 thru 0x00a4e6 (0x7f80 bytes) 000301 1788798094.564277 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: 0x55da7468 (stale) Mirror 5: 0x55da7468 (stale) Mirror 7: 0x55da7468 (stale) Mirror 8: 0x55da7468 (stale) Mirror 9: 0x12487540 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa500): Mirror 4: 0x55da7468 (stale) Mirror 5: 0x55da7468 (stale) Mirror 7: 0x55da7468 (stale) Mirror 8: 0x55da7468 (stale) Mirror 9: 0x12487540 (stale) CRC-32 checksum value for chunk [0, 0xa500): Mirror 1: 0x55752926 Mirror 10: 0x55752926 Mirror 11: 0x55752926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa500: skipped 000302 1788798095.656929 read 0x006828 thru 0x00a4ff (0x3cd8 bytes) 000304 1788798095.705267 read 0x004afa thru 0x00a4ff (0x5a06 bytes) 000305 1788798095.741050 mapread 0x0093aa thru 0x00a4ff (0x1156 bytes) 000307 1788798095.800384 punch from 0x00615c to 0x00a4ff (0x43a4 bytes) 000308 1788798095.952352 trunc from 0x00a500 to 0x018da9 (0xe8aa bytes) 000309 1788798096.004403 trunc from 0x018daa to 0x00b11a (0xdc8f bytes) 000310 1788798096.030506 trunc from 0x00b11b to 0x01054b (0x5431 bytes) 000312 1788798096.057332 MIRROR_RESYNC 8 mirrors 000315 1788798098.040995 trunc from 0x01054c to 0x03ab86 (0x2a63b bytes) 000316 1788798098.137749 write 0x009928 thru 0x00f577 (0x5c50 bytes) 000317 1788798098.151632 mapwrite 0x010529 thru 0x013952 (0x342a bytes) 000318 1788798098.327137 write 0x0174a8 thru 0x0262a6 (0xedff bytes) 000319 1788798098.334575 punch from 0x01ea24 to 0x0294b1 (0xaa8e bytes) 000320 1788798098.382591 read 0x00b22d thru 0x00f9cf (0x47a3 bytes) 000321 1788798098.383844 punch from 0x0198e7 to 0x0282c1 (0xe9db bytes) 000322 1788798098.391942 write 0x006228 thru 0x0063b3 (0x18c bytes) 000323 1788798098.398573 punch from 0x027381 to 0x0291ae (0x1e2e bytes) 000324 1788798098.418369 write 0x02fa7b thru 0x03da8e (0xe014 bytes) 000327 1788798098.428618 mapwrite 0x009d38 thru 0x00cba9 (0x2e72 bytes) 000329 1788798098.587077 trunc from 0x03da8f to 0x0353c2 (0x86cc bytes) 000330 1788798098.648994 MIRROR_RESYNC 8 mirrors 000332 1788798100.549137 write 0x00d47d thru 0x018644 (0xb1c8 bytes) 000333 1788798100.706837 read 0x00bf2c thru 0x018155 (0xc22a bytes) 000334 1788798100.720128 mapread 0x0123ea thru 0x021cc3 (0xf8da bytes) 000336 1788798100.763978 read 0x003f55 thru 0x005e3b (0x1ee7 bytes) 000337 1788798100.777637 read 0x0351bf thru 0x0353c2 (0x204 bytes) 000341 1788798100.781065 mapwrite 0x0228eb thru 0x02c207 (0x991d bytes) 000343 1788798100.889841 punch from 0x003aa0 to 0x011046 (0xd5a7 bytes) 000344 1788798100.906471 write 0x03ca58 thru 0x03ffff (0x35a8 bytes) 000345 1788798100.907572 mapwrite 0x000454 thru 0x009715 (0x92c2 bytes) 000347 1788798101.029497 write 0x000405 thru 0x004349 (0x3f45 bytes) 000350 1788798101.046893 MIRROR_RESYNC 8 mirrors 000351 1788798102.320772 punch from 0x039c22 to 0x03ffff (0x63de bytes) 000352 1788798102.432084 mapwrite 0x024948 thru 0x0292d8 (0x4991 bytes) 000353 1788798102.496336 read 0x006454 thru 0x00c0cd (0x5c7a bytes) 000354 1788798102.511506 write 0x01b45c thru 0x01e6ff (0x32a4 bytes) 000355 1788798102.526321 mapread 0x02685f thru 0x034a78 (0xe21a bytes) 000356 1788798102.540098 punch from 0x037afb to 0x03ffff (0x8505 bytes) 000357 1788798102.569488 punch from 0x02aa32 to 0x039410 (0xe9df bytes) 000361 1788798102.594312 MIRROR_RESYNC 8 mirrors 000362 1788798104.271755 mapread 0x033e3a thru 0x03d5bb (0x9782 bytes) 000363 1788798104.289388 read 0x036689 thru 0x037fe5 (0x195d bytes) 000365 1788798104.290090 punch from 0x0075f7 to 0x00778a (0x194 bytes) 000369 1788798104.351912 write 0x03acd7 thru 0x03ffff (0x5329 bytes) 000371 1788798104.360599 write 0x0047c0 thru 0x00bd5a (0x759b bytes) 000372 1788798104.371298 read 0x03ab9e thru 0x03ffff (0x5462 bytes) 000373 1788798104.373912 mapwrite 0x02e19c thru 0x02e1b9 (0x1e bytes) 000374 1788798104.438203 read 0x03165e thru 0x03ffff (0xe9a2 bytes) 000375 1788798104.443054 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe219ee9f (stale) Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 11: 0xe219ee9f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd521ea5 Mirror 12: 0xbd521ea5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1788798105.499150 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 11: 0xe219ee9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 11: 0xe219ee9f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd521ea5 Mirror 12: 0xbd521ea5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000377 1788798106.544212 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 11: 0xe219ee9f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd521ea5 Mirror 12: 0xbd521ea5 Mirror 13: 0xbd521ea5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000378 1788798108.372556 mapwrite 0x0219ed thru 0x03010d (0xe721 bytes) 000379 1788798108.631508 read 0x025a57 thru 0x029e9e (0x4448 bytes) 000380 1788798108.641331 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 12: 0xbd521ea5 (stale) Mirror 13: 0xbd521ea5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe219ee9f (stale) Mirror 7: 0xe219ee9f (stale) Mirror 8: 0xe219ee9f (stale) Mirror 9: 0xe219ee9f (stale) Mirror 10: 0xe219ee9f (stale) Mirror 12: 0xbd521ea5 (stale) Mirror 13: 0xbd521ea5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1788798109.342134 write 0x014f81 thru 0x015499 (0x519 bytes) 000382 1788798109.358733 write 0x018e0c thru 0x028bcf (0xfdc4 bytes) 000383 1788798109.371359 mapread 0x03fe13 thru 0x03ffff (0x1ed bytes) truncating to largest ever: 0x3f4c7 000384 1788798109.377271 trunc from 0x040000 to 0x03f4c6 (0xb39 bytes) 000386 1788798109.420534 mapread 0x03df52 thru 0x03f4c6 (0x1575 bytes) 000387 1788798109.436610 punch from 0x01bf43 to 0x02bd03 (0xfdc1 bytes) 000388 1788798109.464949 MIRROR_RESYNC 8 mirrors 000390 1788798111.312861 mapwrite 0x01f12b thru 0x0267fa (0x76d0 bytes) 000391 1788798111.682246 punch from 0x028937 to 0x03316f (0xa839 bytes) 000392 1788798111.692479 mapwrite 0x03c8db thru 0x03ffff (0x3725 bytes) 000393 1788798111.814247 trunc from 0x040000 to 0x02cb00 (0x134ff bytes) 000396 1788798111.854874 punch from 0x017d16 to 0x01def8 (0x61e3 bytes) 000398 1788798111.870091 mapread 0x014ed8 thru 0x01d17a (0x82a3 bytes) 000399 1788798111.893129 write 0x01fbe2 thru 0x020671 (0xa90 bytes) 000402 1788798111.900944 mapread 0x002733 thru 0x012065 (0xf933 bytes) 000403 1788798111.922078 read 0x012144 thru 0x01ca87 (0xa944 bytes) 000405 1788798111.924823 mapwrite 0x0149da thru 0x024776 (0xfd9d bytes) 000406 1788798112.012766 trunc from 0x02cb01 to 0x00db15 (0x1efeb bytes) 000407 1788798112.035511 punch from 0x007d0c to 0x00db15 (0x5e0a bytes) 000408 1788798112.041074 write 0x02182d thru 0x027c0c (0x63e0 bytes) 000409 1788798112.043123 trunc from 0x027c0d to 0x013305 (0x14907 bytes) 000410 1788798112.070928 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13306): Mirror 5: 0xb14582b6 (stale) Mirror 8: 0xb14582b6 (stale) Mirror 9: 0xb14582b6 (stale) Mirror 10: 0xb14582b6 (stale) Mirror 12: 0xb14582b6 (stale) Mirror 13: 0xb14582b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13306): Mirror 5: 0xb14582b6 (stale) Mirror 8: 0xb14582b6 (stale) Mirror 9: 0xb14582b6 (stale) Mirror 10: 0xb14582b6 (stale) Mirror 12: 0xb14582b6 (stale) Mirror 13: 0xb14582b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13306: skipped 000411 1788798112.610592 trunc from 0x013306 to 0x03bf29 (0x28c24 bytes) 000412 1788798112.647837 MIRROR_RESYNC 7 mirrors 000413 1788798114.707709 MIRROR_EXTEND to 8 mirrors 000414 1788798115.814766 mapwrite 0x01ed9a thru 0x02bb8a (0xcdf1 bytes) 000415 1788798116.182890 read 0x02566e thru 0x034a31 (0xf3c4 bytes) 000416 1788798116.203539 mapwrite 0x00dbc2 thru 0x01d8c3 (0xfd02 bytes) 000418 1788798116.321474 read 0x01c9c0 thru 0x027208 (0xa849 bytes) 000420 1788798116.330256 mapread 0x02190b thru 0x0233ec (0x1ae2 bytes) 000421 1788798116.339518 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 5: 0xf636b531 (stale) Mirror 8: 0xf636b531 (stale) Mirror 9: 0xf636b531 (stale) Mirror 10: 0xf636b531 (stale) Mirror 12: 0xf636b531 (stale) Mirror 13: 0xf636b531 (stale) Mirror 14: 0xf636b531 (stale) CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 1: 0xd64d1c40 Mirror 15: 0xd64d1c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf2a: skipped 000423 1788798117.949095 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 5: 0xf636b531 (stale) Mirror 8: 0xf636b531 (stale) Mirror 9: 0xf636b531 (stale) Mirror 10: 0xf636b531 (stale) Mirror 13: 0xf636b531 (stale) Mirror 14: 0xf636b531 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 5: 0xf636b531 (stale) Mirror 8: 0xf636b531 (stale) Mirror 9: 0xf636b531 (stale) Mirror 10: 0xf636b531 (stale) Mirror 13: 0xf636b531 (stale) Mirror 14: 0xf636b531 (stale) CRC-32 checksum value for chunk [0, 0x3bf2a): Mirror 1: 0xd64d1c40 Mirror 15: 0xd64d1c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf2a: skipped 000424 1788798119.002031 trunc from 0x03bf2a to 0x024c66 (0x172c3 bytes) 000425 1788798119.112523 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c67): Mirror 5: 0xf805ece4 (stale) Mirror 8: 0xf805ece4 (stale) Mirror 9: 0xf805ece4 (stale) Mirror 10: 0xf805ece4 (stale) Mirror 13: 0xf805ece4 (stale) Mirror 14: 0xf805ece4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c67): Mirror 5: 0xf805ece4 (stale) Mirror 8: 0xf805ece4 (stale) Mirror 9: 0xf805ece4 (stale) Mirror 10: 0xf805ece4 (stale) Mirror 13: 0xf805ece4 (stale) Mirror 14: 0xf805ece4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c67: skipped 000426 1788798120.059345 write 0x03b9d9 thru 0x03ed35 (0x335d bytes) 000429 1788798120.065328 punch from 0x02ecd1 to 0x03a0f9 (0xb429 bytes) 000430 1788798120.086441 mapwrite 0x033314 thru 0x03ffff (0xccec bytes) 000431 1788798120.201050 read 0x0332f7 thru 0x03e5cd (0xb2d7 bytes) 000432 1788798120.201832 mapwrite 0x017b08 thru 0x021f41 (0xa43a bytes) 000433 1788798120.315169 mapread 0x01199a thru 0x020a07 (0xf06e bytes) 000434 1788798120.317460 mapread 0x01b5ec thru 0x0241a1 (0x8bb6 bytes) 000435 1788798120.319174 punch from 0x0106e6 to 0x0109e4 (0x2ff bytes) 000436 1788798120.327878 read 0x03a8d9 thru 0x03ffff (0x5727 bytes) 000437 1788798120.340732 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x80d8def9 (stale) Mirror 8: 0x80d8def9 (stale) Mirror 9: 0x80d8def9 (stale) Mirror 10: 0x80d8def9 (stale) Mirror 13: 0x80d8def9 (stale) Mirror 14: 0x80d8def9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f905fbd Mirror 15: 0x9f905fbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000438 1788798122.035253 punch from 0x01e82d to 0x0285c9 (0x9d9d bytes) 000439 1788798122.121161 trunc from 0x040000 to 0x01de9a (0x22165 bytes) 000440 1788798122.143900 read 0x0120c2 thru 0x01de9a (0xbdd9 bytes) 000441 1788798122.156044 punch from 0x008b9a to 0x00fa36 (0x6e9d bytes) 000442 1788798122.159901 punch from 0x010459 to 0x01c8b0 (0xc458 bytes) 000443 1788798122.164289 trunc from 0x01de9b to 0x01fc12 (0x1d78 bytes) 000446 1788798122.200303 punch from 0x00513d to 0x00d01f (0x7ee3 bytes) 000448 1788798122.209561 mapwrite 0x020989 thru 0x02f74e (0xedc6 bytes) 000450 1788798122.405233 mapread 0x01ade8 thru 0x02567e (0xa897 bytes) 000453 1788798122.424359 mapread 0x027b6f thru 0x02f74e (0x7be0 bytes) 000454 1788798122.427235 mapread 0x0254f8 thru 0x02f74e (0xa257 bytes) 000455 1788798122.431907 mapwrite 0x023a4f thru 0x0294be (0x5a70 bytes) 000456 1788798122.494183 mapread 0x00e5ef thru 0x018028 (0x9a3a bytes) 000457 1788798122.508332 read 0x01a1a6 thru 0x024457 (0xa2b2 bytes) 000459 1788798122.510571 mapread 0x00786d thru 0x012d70 (0xb504 bytes) 000460 1788798122.511997 mapwrite 0x0027a0 thru 0x006917 (0x4178 bytes) 000461 1788798122.569147 mapwrite 0x01bd07 thru 0x01dbcc (0x1ec6 bytes) 000462 1788798122.615281 write 0x031a86 thru 0x039a52 (0x7fcd bytes) 000463 1788798122.616689 mapwrite 0x026edc thru 0x02e86f (0x7994 bytes) 000464 1788798122.678541 read 0x0002d0 thru 0x002eb7 (0x2be8 bytes) 000465 1788798122.681953 write 0x01a63f thru 0x0207c3 (0x6185 bytes) 000467 1788798122.688359 trunc from 0x039a53 to 0x00bf35 (0x2db1d bytes) 000469 1788798122.720372 write 0x0104de thru 0x01ba48 (0xb56b bytes) 000470 1788798122.721825 mapwrite 0x02d2ac thru 0x03d0b3 (0xfe08 bytes) 000471 1788798122.799556 trunc from 0x03d0b4 to 0x00a600 (0x32ab3 bytes) 000472 1788798122.818953 read 0x004e20 thru 0x00a600 (0x57e1 bytes) 000473 1788798122.826584 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xa601): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa601): Mirror 5: 0x6a422fce (stale) Mirror 8: 0x6a422fce (stale) Mirror 9: 0x6a422fce (stale) Mirror 10: 0x6a422fce (stale) Mirror 13: 0x6a422fce (stale) Mirror 14: 0x6a422fce (stale) Mirror 15: 0x6a422fce (stale) CRC-32 checksum value for chunk [0, 0xa601): Mirror 1: 0xe16b107e Mirror 16: 0xe16b107e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa601: skipped 000474 1788798124.130479 write 0x00f853 thru 0x01b4e9 (0xbc97 bytes) 000476 1788798124.244372 mapwrite 0x03c50d thru 0x03fa57 (0x354b bytes) 000477 1788798124.387495 MIRROR_RESYNC 9 mirrors 000478 1788798127.610558 read 0x025f07 thru 0x0314e4 (0xb5de bytes) 000479 1788798127.631937 trunc from 0x03fa58 to 0x017121 (0x28936 bytes) 000480 1788798127.904765 mapwrite 0x015b25 thru 0x0250f6 (0xf5d2 bytes) 000482 1788798128.031695 mapwrite 0x004444 thru 0x00ec45 (0xa802 bytes) 000483 1788798128.117604 read 0x01a923 thru 0x0250f6 (0xa7d4 bytes) 000484 1788798128.118874 punch from 0x007e79 to 0x00a9cd (0x2b55 bytes) 000485 1788798128.125344 read 0x015bb9 thru 0x019ba5 (0x3fed bytes) 000486 1788798128.125966 write 0x024f69 thru 0x029729 (0x47c1 bytes) 000487 1788798128.127815 read 0x019870 thru 0x01f702 (0x5e93 bytes) 000488 1788798128.129437 punch from 0x01aba8 to 0x0224bd (0x7916 bytes) 000495 1788798128.152304 read 0x00ff39 thru 0x010706 (0x7ce bytes) 000496 1788798128.153210 trunc from 0x02972a to 0x0215b9 (0x8170 bytes) 000497 1788798128.202559 punch from 0x01e50b to 0x01f4f1 (0xfe7 bytes) 000498 1788798128.209329 read 0x0132f4 thru 0x016140 (0x2e4d bytes) 000499 1788798128.210361 write 0x02bddf thru 0x033421 (0x7643 bytes) 000500 1788798128.213519 punch from 0x01eba3 to 0x02b8b0 (0xcd0e bytes) 000503 1788798128.241557 mapread 0x00b3f6 thru 0x00b4b2 (0xbd bytes) 000505 1788798128.245313 punch from 0x024af0 to 0x02decc (0x93dd bytes) 000506 1788798128.250448 mapwrite 0x004d3c thru 0x00d4cb (0x8790 bytes) 000507 1788798128.349382 punch from 0x003f14 to 0x009cc9 (0x5db6 bytes) 000510 1788798128.371428 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33422): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33422): Mirror 5: 0xc6462935 (stale) Mirror 8: 0xc6462935 (stale) Mirror 9: 0xc6462935 (stale) Mirror 10: 0xc6462935 (stale) Mirror 13: 0xc6462935 (stale) Mirror 14: 0xc6462935 (stale) Mirror 15: 0xc6462935 (stale) Mirror 16: 0xc6462935 (stale) CRC-32 checksum value for chunk [0, 0x33422): Mirror 1: 0x91d37a64 Mirror 17: 0x91d37a64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33422: skipped 000511 1788798130.437403 trunc from 0x033422 to 0x0308db (0x2b46 bytes) 000512 1788798130.538608 mapread 0x02861b thru 0x0308db (0x82c1 bytes) 000515 1788798130.600192 read 0x009f47 thru 0x0125d1 (0x868b bytes) 000516 1788798130.630285 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x308dc): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308dc): Mirror 5: 0x1f086e01 (stale) Mirror 8: 0x1f086e01 (stale) Mirror 9: 0x1f086e01 (stale) Mirror 10: 0x1f086e01 (stale) Mirror 13: 0x1f086e01 (stale) Mirror 14: 0x1f086e01 (stale) Mirror 15: 0x1f086e01 (stale) Mirror 16: 0x1f086e01 (stale) Mirror 17: 0xd56eb59c (stale) CRC-32 checksum value for chunk [0, 0x308dc): Mirror 1: 0xd56eb59c Mirror 18: 0xd56eb59c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308dc: skipped 000517 1788798132.523712 trunc from 0x0308dc to 0x033c23 (0x3348 bytes) 000518 1788798132.740276 punch from 0x01a27c to 0x01b943 (0x16c8 bytes) 000519 1788798132.750375 mapread 0x0040d7 thru 0x00c211 (0x813b bytes) 000521 1788798132.833747 read 0x00007e thru 0x004c8f (0x4c12 bytes) 000522 1788798132.843929 read 0x013152 thru 0x019493 (0x6342 bytes) 000523 1788798132.857390 trunc from 0x033c24 to 0x030237 (0x39ec bytes) 000524 1788798132.893545 write 0x038d9b thru 0x03ffff (0x7265 bytes) 000526 1788798132.901325 mapread 0x02e555 thru 0x02fe96 (0x1942 bytes) 000532 1788798132.917035 read 0x02c59d thru 0x032aeb (0x654f bytes) 000533 1788798132.927385 read 0x03f794 thru 0x03ffff (0x86c bytes) 000536 1788798132.934281 mapwrite 0x01b6b6 thru 0x01c753 (0x109e bytes) 000537 1788798133.028994 trunc from 0x040000 to 0x006035 (0x39fca bytes) 000538 1788798133.084557 mapread 0x005774 thru 0x006035 (0x8c2 bytes) 000540 1788798133.094416 mapread 0x003c29 thru 0x006035 (0x240d bytes) 000541 1788798133.096734 trunc from 0x006036 to 0x002a01 (0x3634 bytes) 000542 1788798133.144248 read 0x0010ba thru 0x002a01 (0x1948 bytes) 000543 1788798133.145007 mapread 0x000488 thru 0x002538 (0x20b1 bytes) 000544 1788798133.145878 read 0x0012cd thru 0x002a01 (0x1735 bytes) 000545 1788798133.149950 punch from 0x000f10 to 0x002a01 (0x1af2 bytes) 000548 1788798133.156765 mapread 0x0007e3 thru 0x002a01 (0x221f bytes) 000550 1788798133.165491 punch from 0x000894 to 0x002a01 (0x216e bytes) 000552 1788798133.172695 write 0x02669e thru 0x035014 (0xe977 bytes) 000553 1788798133.174605 trunc from 0x035015 to 0x02ff87 (0x508d bytes) 000555 1788798133.194261 trunc from 0x02ff88 to 0x036b88 (0x6c01 bytes) 000556 1788798133.225756 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36b89): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b89): Mirror 5: 0xbaa8029d (stale) Mirror 8: 0xbaa8029d (stale) Mirror 9: 0xbaa8029d (stale) Mirror 10: 0xbaa8029d (stale) Mirror 13: 0xbaa8029d (stale) Mirror 14: 0xbaa8029d (stale) Mirror 15: 0xbaa8029d (stale) Mirror 16: 0xbaa8029d (stale) Mirror 17: 0x4a7916f (stale) Mirror 18: 0xcb626e96 (stale) CRC-32 checksum value for chunk [0, 0x36b89): Mirror 1: 0xcb374611 Mirror 19: 0xcb374611 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b89: skipped 000557 1788798135.668153 mapwrite 0x00f225 thru 0x0146bb (0x5497 bytes) 000558 1788798135.905819 mapwrite 0x032a88 thru 0x035f1f (0x3498 bytes) 000559 1788798136.008506 mapwrite 0x03db96 thru 0x03ffff (0x246a bytes) 000560 1788798136.124417 punch from 0x02ebe8 to 0x031ade (0x2ef7 bytes) 000562 1788798136.134736 mapread 0x027e0e thru 0x028c4e (0xe41 bytes) 000563 1788798136.154429 mapread 0x0094a2 thru 0x009917 (0x476 bytes) 000564 1788798136.155400 punch from 0x01f352 to 0x022fc1 (0x3c70 bytes) 000565 1788798136.163149 read 0x004c09 thru 0x011c6e (0xd066 bytes) 000566 1788798136.183570 trunc from 0x040000 to 0x00e634 (0x319cb bytes) 000567 1788798136.201554 read 0x00ab9c thru 0x00e634 (0x3a99 bytes) 000568 1788798136.212695 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xe635): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe635): Mirror 5: 0xf42d0760 (stale) Mirror 8: 0xf42d0760 (stale) Mirror 9: 0xf42d0760 (stale) Mirror 10: 0xf42d0760 (stale) Mirror 13: 0xf42d0760 (stale) Mirror 14: 0xf42d0760 (stale) Mirror 15: 0xf42d0760 (stale) Mirror 16: 0xf42d0760 (stale) Mirror 17: 0x2886f6c7 (stale) Mirror 18: 0x2886f6c7 (stale) Mirror 19: 0x815af595 (stale) CRC-32 checksum value for chunk [0, 0xe635): Mirror 1: 0x815af595 Mirror 20: 0x815af595 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe635: skipped 000569 1788798137.554676 trunc from 0x00e635 to 0x0254c4 (0x16e90 bytes) 000570 1788798137.679196 write 0x0345fa thru 0x03a2da (0x5ce1 bytes) 000571 1788798137.681361 punch from 0x01c380 to 0x028ff5 (0xcc76 bytes) 000573 1788798137.702697 read 0x0264b8 thru 0x0329d8 (0xc521 bytes) 000575 1788798137.735258 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3a2db): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2db): Mirror 5: 0x47515db4 (stale) Mirror 8: 0x47515db4 (stale) Mirror 9: 0x47515db4 (stale) Mirror 10: 0x47515db4 (stale) Mirror 13: 0x47515db4 (stale) Mirror 14: 0x47515db4 (stale) Mirror 15: 0x47515db4 (stale) Mirror 16: 0x47515db4 (stale) Mirror 17: 0x5da4a534 (stale) Mirror 18: 0x99e69199 (stale) Mirror 19: 0xda45de0 (stale) Mirror 20: 0x1a1b5b9f (stale) CRC-32 checksum value for chunk [0, 0x3a2db): Mirror 1: 0x2f47c0ac Mirror 21: 0x2f47c0ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2db: skipped 000577 1788798139.881943 trunc from 0x03a2db to 0x020eed (0x193ed bytes) 000578 1788798139.964630 mapread 0x019469 thru 0x020eed (0x7a85 bytes) 000579 1788798139.987317 mapwrite 0x018aad thru 0x018f15 (0x469 bytes) 000581 1788798140.054382 mapread 0x0073f4 thru 0x009295 (0x1ea2 bytes) 000582 1788798140.092551 trunc from 0x020eee to 0x0061c9 (0x1ad24 bytes) 000583 1788798140.149038 write 0x01280f thru 0x018028 (0x581a bytes) 000585 1788798140.156253 read 0x0131bb thru 0x0177cb (0x4611 bytes) 000586 1788798140.158356 read 0x013f2b thru 0x018028 (0x40fe bytes) 000587 1788798140.162577 mapwrite 0x00a2e6 thru 0x014446 (0xa161 bytes) 000588 1788798140.284018 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x18029): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18029): Mirror 5: 0xfde35f4b (stale) Mirror 8: 0xfde35f4b (stale) Mirror 9: 0xfde35f4b (stale) Mirror 10: 0xfde35f4b (stale) Mirror 13: 0xfde35f4b (stale) Mirror 14: 0xfde35f4b (stale) Mirror 15: 0xfde35f4b (stale) Mirror 16: 0xfde35f4b (stale) Mirror 17: 0x78ee094a (stale) Mirror 18: 0x78ee094a (stale) Mirror 19: 0x92f49865 (stale) Mirror 20: 0x92f49865 (stale) Mirror 21: 0x92f49865 (stale) CRC-32 checksum value for chunk [0, 0x18029): Mirror 1: 0xdd6db7ea Mirror 22: 0xdd6db7ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18029: skipped 000589 1788798141.934604 punch from 0x00988e to 0x011b6e (0x82e1 bytes) 000590 1788798142.131578 mapread 0x00037c thru 0x003562 (0x31e7 bytes) 000591 1788798142.192383 punch from 0x00491b to 0x00c5b0 (0x7c96 bytes) 000592 1788798142.199898 trunc from 0x018029 to 0x024fdc (0xcfb4 bytes) 000593 1788798142.316995 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24fdd): Mirror 5: 0x1eb44a26 (stale) Mirror 8: 0x1eb44a26 (stale) Mirror 9: 0x1eb44a26 (stale) Mirror 10: 0x1eb44a26 (stale) Mirror 13: 0x1eb44a26 (stale) Mirror 14: 0x1eb44a26 (stale) Mirror 15: 0x1eb44a26 (stale) Mirror 16: 0x1eb44a26 (stale) Mirror 17: 0x690b176 (stale) Mirror 18: 0x690b176 (stale) Mirror 19: 0x5591f280 (stale) Mirror 20: 0x5591f280 (stale) Mirror 21: 0x5591f280 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fdd): Mirror 5: 0x1eb44a26 (stale) Mirror 8: 0x1eb44a26 (stale) Mirror 9: 0x1eb44a26 (stale) Mirror 10: 0x1eb44a26 (stale) Mirror 13: 0x1eb44a26 (stale) Mirror 14: 0x1eb44a26 (stale) Mirror 15: 0x1eb44a26 (stale) Mirror 16: 0x1eb44a26 (stale) Mirror 17: 0x690b176 (stale) Mirror 18: 0x690b176 (stale) Mirror 19: 0x5591f280 (stale) Mirror 20: 0x5591f280 (stale) Mirror 21: 0x5591f280 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fdd: skipped 000594 1788798143.521135 mapwrite 0x03c6d0 thru 0x03f4de (0x2e0f bytes) 000596 1788798143.623393 punch from 0x0357e6 to 0x03f4de (0x9cf9 bytes) 000598 1788798143.634444 read 0x039092 thru 0x03f4de (0x644d bytes) 000599 1788798143.651956 MIRROR_RESYNC 14 mirrors 000600 1788798147.561146 trunc from 0x03f4df to 0x02ea48 (0x10a96 bytes) 000601 1788798147.681319 mapread 0x00f29e thru 0x01be12 (0xcb75 bytes) 000602 1788798147.768355 mapread 0x000dfe thru 0x00e1ef (0xd3f2 bytes) 000603 1788798147.783425 read 0x015ba2 thru 0x01e056 (0x84b5 bytes) 000604 1788798147.784536 punch from 0x0109d4 to 0x01442f (0x3a5c bytes) 000606 1788798147.807249 read 0x013de5 thru 0x018a4d (0x4c69 bytes) 000608 1788798147.835569 mapwrite 0x01ccbf thru 0x022c40 (0x5f82 bytes) 000609 1788798147.898924 mapread 0x0125e7 thru 0x0146ef (0x2109 bytes) 000612 1788798147.924482 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ea49): Mirror 8: 0x8d2c86a9 (stale) Mirror 9: 0x8d2c86a9 (stale) Mirror 10: 0x8d2c86a9 (stale) Mirror 13: 0x8d2c86a9 (stale) Mirror 14: 0x8d2c86a9 (stale) Mirror 15: 0x8d2c86a9 (stale) Mirror 16: 0x8d2c86a9 (stale) Mirror 17: 0x8d2c86a9 (stale) Mirror 18: 0x8d2c86a9 (stale) Mirror 19: 0x8d2c86a9 (stale) Mirror 20: 0x8d2c86a9 (stale) Mirror 21: 0x8d2c86a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea49): Mirror 8: 0x8d2c86a9 (stale) Mirror 9: 0x8d2c86a9 (stale) Mirror 10: 0x8d2c86a9 (stale) Mirror 13: 0x8d2c86a9 (stale) Mirror 14: 0x8d2c86a9 (stale) Mirror 15: 0x8d2c86a9 (stale) Mirror 16: 0x8d2c86a9 (stale) Mirror 17: 0x8d2c86a9 (stale) Mirror 18: 0x8d2c86a9 (stale) Mirror 19: 0x8d2c86a9 (stale) Mirror 20: 0x8d2c86a9 (stale) Mirror 21: 0x8d2c86a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea49: skipped 000614 1788798149.069102 write 0x003c33 thru 0x00f699 (0xba67 bytes) 000615 1788798149.087294 read 0x00580c thru 0x0093eb (0x3be0 bytes) 000616 1788798149.087921 punch from 0x023d8e to 0x02ea48 (0xacbb bytes) 000617 1788798149.115466 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ea49): Mirror 9: 0x8d2c86a9 (stale) Mirror 10: 0x8d2c86a9 (stale) Mirror 13: 0x8d2c86a9 (stale) Mirror 14: 0x8d2c86a9 (stale) Mirror 15: 0x8d2c86a9 (stale) Mirror 16: 0x8d2c86a9 (stale) Mirror 17: 0x8d2c86a9 (stale) Mirror 18: 0x8d2c86a9 (stale) Mirror 19: 0x8d2c86a9 (stale) Mirror 20: 0x8d2c86a9 (stale) Mirror 21: 0x8d2c86a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea49): Mirror 9: 0x8d2c86a9 (stale) Mirror 10: 0x8d2c86a9 (stale) Mirror 13: 0x8d2c86a9 (stale) Mirror 14: 0x8d2c86a9 (stale) Mirror 15: 0x8d2c86a9 (stale) Mirror 16: 0x8d2c86a9 (stale) Mirror 17: 0x8d2c86a9 (stale) Mirror 18: 0x8d2c86a9 (stale) Mirror 19: 0x8d2c86a9 (stale) Mirror 20: 0x8d2c86a9 (stale) Mirror 21: 0x8d2c86a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea49: skipped 000619 1788798150.234403 read 0x01cc67 thru 0x026522 (0x98bc bytes) 000621 1788798150.256400 trunc from 0x02ea49 to 0x024f57 (0x9af1 bytes) 000622 1788798150.294049 read 0x00d2b7 thru 0x01a85e (0xd5a8 bytes) 000623 1788798150.312410 trunc from 0x024f58 to 0x03ea2d (0x19ad6 bytes) 000624 1788798150.332230 mapread 0x02ff74 thru 0x031042 (0x10cf bytes) 000625 1788798150.344156 trunc from 0x03ea2e to 0x009f48 (0x34ae5 bytes) 000626 1788798150.367154 mapwrite 0x002f0c thru 0x0071d7 (0x42cc bytes) 000628 1788798150.477193 punch from 0x0005f3 to 0x009d87 (0x9795 bytes) 000629 1788798150.484764 mapread 0x0038c6 thru 0x004725 (0xe60 bytes) 000631 1788798150.530090 write 0x0382ff thru 0x03cfe0 (0x4ce2 bytes) 000632 1788798150.533244 punch from 0x0032b7 to 0x00b93a (0x8684 bytes) 000633 1788798150.561116 read 0x03c0e4 thru 0x03cfe0 (0xefd bytes) 000635 1788798150.564180 trunc from 0x03cfe1 to 0x0313cf (0xbc11 bytes) 000636 1788798150.604200 mapwrite 0x02fdef thru 0x03487d (0x4a8f bytes) 000637 1788798150.760283 MIRROR_RESYNC 12 mirrors 000639 1788798154.360069 read 0x022234 thru 0x0265e9 (0x43b6 bytes) 000641 1788798154.374200 MIRROR_RESYNC 12 mirrors 000642 1788798155.511464 write 0x0192e9 thru 0x01bac7 (0x27df bytes) 000643 1788798155.704758 trunc from 0x03487e to 0x00754d (0x2d330 bytes) 000644 1788798155.796984 mapread 0x002bd2 thru 0x00754d (0x497c bytes) 000645 1788798155.861188 punch from 0x005abf to 0x00754d (0x1a8f bytes) 000647 1788798155.870787 mapwrite 0x034549 thru 0x03df2d (0x99e5 bytes) 000648 1788798155.985703 mapwrite 0x03ad18 thru 0x03ffff (0x52e8 bytes) 000650 1788798156.074341 mapread 0x01a86c thru 0x01d7d5 (0x2f6a bytes) 000651 1788798156.087243 read 0x00a927 thru 0x018c98 (0xe372 bytes) 000652 1788798156.099314 mapread 0x00c2bd thru 0x016710 (0xa454 bytes) 000655 1788798156.108995 MIRROR_RESYNC 12 mirrors 000656 1788798159.385107 read 0x0220c1 thru 0x027b41 (0x5a81 bytes) 000657 1788798159.405621 mapwrite 0x01d12a thru 0x027be4 (0xaabb bytes) 000659 1788798159.855141 write 0x01f689 thru 0x025a85 (0x63fd bytes) 000660 1788798159.862056 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5921ed58 (stale) Mirror 10: 0x5921ed58 (stale) Mirror 13: 0x5921ed58 (stale) Mirror 14: 0x5921ed58 (stale) Mirror 15: 0x5921ed58 (stale) Mirror 16: 0x5921ed58 (stale) Mirror 17: 0x5921ed58 (stale) Mirror 18: 0x5921ed58 (stale) Mirror 19: 0x5921ed58 (stale) Mirror 20: 0x5921ed58 (stale) Mirror 21: 0x5921ed58 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2628130 Mirror 22: 0xd2628130 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000661 1788798161.575418 punch from 0x01e784 to 0x024f6d (0x67ea bytes) 000663 1788798161.695281 MIRROR_RESYNC 13 mirrors 000665 1788798165.350795 mapread 0x03b61b thru 0x03ffff (0x49e5 bytes) 000667 1788798165.400584 mapread 0x01904f thru 0x0262be (0xd270 bytes) 000668 1788798165.471965 punch from 0x02809f to 0x031d39 (0x9c9b bytes) 000669 1788798165.671736 mapwrite 0x03080e thru 0x03cd1b (0xc50e bytes) 000672 1788798166.018820 write 0x017b1d thru 0x0242b0 (0xc794 bytes) 000673 1788798166.071085 write 0x023271 thru 0x027631 (0x43c1 bytes) 000674 1788798166.093619 read 0x004d4c thru 0x011993 (0xcc48 bytes) 000675 1788798166.138505 mapread 0x038bcc thru 0x03a8f6 (0x1d2b bytes) 000677 1788798166.140027 mapwrite 0x02d82c thru 0x035616 (0x7deb bytes) 000678 1788798166.316894 punch from 0x0301f4 to 0x035a5e (0x586b bytes) 000680 1788798166.339620 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6512ce79 (stale) Mirror 10: 0x6512ce79 (stale) Mirror 13: 0x6512ce79 (stale) Mirror 14: 0x6512ce79 (stale) Mirror 15: 0x6512ce79 (stale) Mirror 16: 0x6512ce79 (stale) Mirror 17: 0x6512ce79 (stale) Mirror 18: 0x6512ce79 (stale) Mirror 19: 0x6512ce79 (stale) Mirror 20: 0x6512ce79 (stale) Mirror 21: 0x6512ce79 (stale) Mirror 22: 0x6512ce79 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1944e33e Mirror 23: 0x1944e33e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000682 1788798168.579601 punch from 0x014f46 to 0x0242bf (0xf37a bytes) 000683 1788798168.673743 trunc from 0x040000 to 0x0302af (0xfd50 bytes) 000684 1788798168.701120 mapwrite 0x035606 thru 0x03af30 (0x592b bytes) 000685 1788798168.783926 punch from 0x0055db to 0x00bad3 (0x64f9 bytes) 000688 1788798168.796763 read 0x02bcad thru 0x03563b (0x998f bytes) 000691 1788798168.813526 MIRROR_RESYNC 14 mirrors 000692 1788798173.420936 MIRROR_RESYNC 14 mirrors 000694 1788798174.627443 punch from 0x01e41f to 0x01e8bb (0x49d bytes) 000695 1788798174.807924 MIRROR_RESYNC 14 mirrors 000696 1788798178.981437 mapread 0x00e47e thru 0x01b240 (0xcdc3 bytes) 000697 1788798179.072630 mapread 0x000d1a thru 0x003ce8 (0x2fcf bytes) 000698 1788798179.106701 write 0x00a8b4 thru 0x0140b0 (0x97fd bytes) 000699 1788798179.346924 write 0x03b4ec thru 0x03ffff (0x4b14 bytes) 000700 1788798179.363360 MIRROR_RESYNC 14 mirrors 000701 1788798183.488794 trunc from 0x040000 to 0x039db2 (0x624d bytes) 000702 1788798183.641860 mapwrite 0x004885 thru 0x0114aa (0xcc26 bytes) 000703 1788798183.862721 trunc from 0x039db3 to 0x03d135 (0x3383 bytes) 000704 1788798183.939438 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d136): Mirror 10: 0xcf7c0dc0 (stale) Mirror 13: 0xcf7c0dc0 (stale) Mirror 14: 0xcf7c0dc0 (stale) Mirror 15: 0xcf7c0dc0 (stale) Mirror 16: 0xcf7c0dc0 (stale) Mirror 17: 0xcf7c0dc0 (stale) Mirror 18: 0xcf7c0dc0 (stale) Mirror 19: 0xcf7c0dc0 (stale) Mirror 20: 0xcf7c0dc0 (stale) Mirror 21: 0xcf7c0dc0 (stale) Mirror 22: 0xcf7c0dc0 (stale) Mirror 23: 0xcf7c0dc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d136): Mirror 10: 0xcf7c0dc0 (stale) Mirror 13: 0xcf7c0dc0 (stale) Mirror 14: 0xcf7c0dc0 (stale) Mirror 15: 0xcf7c0dc0 (stale) Mirror 16: 0xcf7c0dc0 (stale) Mirror 17: 0xcf7c0dc0 (stale) Mirror 18: 0xcf7c0dc0 (stale) Mirror 19: 0xcf7c0dc0 (stale) Mirror 20: 0xcf7c0dc0 (stale) Mirror 21: 0xcf7c0dc0 (stale) Mirror 22: 0xcf7c0dc0 (stale) Mirror 23: 0xcf7c0dc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d136: skipped 000705 1788798184.846623 read 0x03b2c1 thru 0x03d135 (0x1e75 bytes) 000706 1788798184.853582 mapwrite 0x0129a1 thru 0x01c626 (0x9c86 bytes) 000707 1788798184.969104 read 0x01fb9a thru 0x0278ba (0x7d21 bytes) 000708 1788798184.994752 write 0x00a8f2 thru 0x0111e3 (0x68f2 bytes) 000709 1788798184.996083 read 0x00621d thru 0x010afc (0xa8e0 bytes) 000711 1788798185.000012 trunc from 0x03d136 to 0x038c19 (0x451c bytes) 000712 1788798185.055445 mapread 0x00878a thru 0x00fff6 (0x786d bytes) 000713 1788798185.056350 read 0x0188e0 thru 0x02336c (0xaa8d bytes) 000715 1788798185.069482 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 10: 0x912ab7c3 (stale) Mirror 13: 0x912ab7c3 (stale) Mirror 14: 0x912ab7c3 (stale) Mirror 15: 0x912ab7c3 (stale) Mirror 16: 0x912ab7c3 (stale) Mirror 17: 0x912ab7c3 (stale) Mirror 18: 0x912ab7c3 (stale) Mirror 19: 0x912ab7c3 (stale) Mirror 20: 0x912ab7c3 (stale) Mirror 21: 0x912ab7c3 (stale) Mirror 23: 0x912ab7c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 10: 0x912ab7c3 (stale) Mirror 13: 0x912ab7c3 (stale) Mirror 14: 0x912ab7c3 (stale) Mirror 15: 0x912ab7c3 (stale) Mirror 16: 0x912ab7c3 (stale) Mirror 17: 0x912ab7c3 (stale) Mirror 18: 0x912ab7c3 (stale) Mirror 19: 0x912ab7c3 (stale) Mirror 20: 0x912ab7c3 (stale) Mirror 21: 0x912ab7c3 (stale) Mirror 23: 0x912ab7c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1a: skipped 000716 1788798185.777040 mapread 0x032777 thru 0x038c19 (0x64a3 bytes) 000719 1788798185.810751 trunc from 0x038c1a to 0x00f5d5 (0x29644 bytes) 000721 1788798185.849207 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xf5d6): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf5d6): Mirror 10: 0x596fcbf2 (stale) Mirror 13: 0x596fcbf2 (stale) Mirror 14: 0x596fcbf2 (stale) Mirror 15: 0x596fcbf2 (stale) Mirror 16: 0x596fcbf2 (stale) Mirror 17: 0x596fcbf2 (stale) Mirror 18: 0x596fcbf2 (stale) Mirror 19: 0x596fcbf2 (stale) Mirror 20: 0x596fcbf2 (stale) Mirror 21: 0x596fcbf2 (stale) Mirror 23: 0x596fcbf2 (stale) CRC-32 checksum value for chunk [0, 0xf5d6): Mirror 1: 0xdc09054b Mirror 24: 0xdc09054b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf5d6: skipped 000722 1788798186.939446 write 0x01f9a0 thru 0x0296e2 (0x9d43 bytes) 000723 1788798187.073523 trunc from 0x0296e3 to 0x016b37 (0x12bab bytes) 000724 1788798187.104498 mapread 0x012983 thru 0x016b37 (0x41b5 bytes) 000725 1788798187.126554 mapread 0x004e81 thru 0x00cea0 (0x8020 bytes) 000726 1788798187.140731 read 0x004848 thru 0x00e7d8 (0x9f91 bytes) 000727 1788798187.142285 punch from 0x005666 to 0x00c694 (0x702f bytes) 000729 1788798187.155547 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16b38): Mirror 10: 0xf5c8ea7b (stale) Mirror 13: 0xf5c8ea7b (stale) Mirror 14: 0xf5c8ea7b (stale) Mirror 15: 0xf5c8ea7b (stale) Mirror 17: 0xf5c8ea7b (stale) Mirror 18: 0xf5c8ea7b (stale) Mirror 19: 0xf5c8ea7b (stale) Mirror 20: 0xf5c8ea7b (stale) Mirror 21: 0xf5c8ea7b (stale) Mirror 23: 0xf5c8ea7b (stale) Mirror 24: 0xdd7726cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b38): Mirror 10: 0xf5c8ea7b (stale) Mirror 13: 0xf5c8ea7b (stale) Mirror 14: 0xf5c8ea7b (stale) Mirror 15: 0xf5c8ea7b (stale) Mirror 17: 0xf5c8ea7b (stale) Mirror 18: 0xf5c8ea7b (stale) Mirror 19: 0xf5c8ea7b (stale) Mirror 20: 0xf5c8ea7b (stale) Mirror 21: 0xf5c8ea7b (stale) Mirror 23: 0xf5c8ea7b (stale) Mirror 24: 0xdd7726cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b38: skipped 000730 1788798187.856380 punch from 0x0168ab to 0x016b37 (0x28d bytes) 000731 1788798187.865461 punch from 0x015aaf to 0x016b37 (0x1089 bytes) 000732 1788798187.880362 mapwrite 0x003761 thru 0x006e42 (0x36e2 bytes) 000734 1788798188.065727 read 0x006569 thru 0x00bd2b (0x57c3 bytes) 000736 1788798188.066871 read 0x00533f thru 0x011dc6 (0xca88 bytes) 000737 1788798188.093585 trunc from 0x016b38 to 0x0256dc (0xeba5 bytes) 000745 1788798188.153009 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x256dd): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256dd): Mirror 10: 0xfc607f (stale) Mirror 13: 0xfc607f (stale) Mirror 14: 0xfc607f (stale) Mirror 15: 0xfc607f (stale) Mirror 17: 0xfc607f (stale) Mirror 18: 0xfc607f (stale) Mirror 19: 0xfc607f (stale) Mirror 20: 0xfc607f (stale) Mirror 21: 0xfc607f (stale) Mirror 23: 0xfc607f (stale) Mirror 24: 0x5dd78562 (stale) CRC-32 checksum value for chunk [0, 0x256dd): Mirror 1: 0x5f758db0 Mirror 25: 0x5f758db0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256dd: skipped 000747 1788798190.548214 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x256dd): Mirror 10: 0xfc607f (stale) Mirror 14: 0xfc607f (stale) Mirror 15: 0xfc607f (stale) Mirror 17: 0xfc607f (stale) Mirror 18: 0xfc607f (stale) Mirror 19: 0xfc607f (stale) Mirror 20: 0xfc607f (stale) Mirror 21: 0xfc607f (stale) Mirror 23: 0xfc607f (stale) Mirror 24: 0x5dd78562 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256dd): Mirror 10: 0xfc607f (stale) Mirror 14: 0xfc607f (stale) Mirror 15: 0xfc607f (stale) Mirror 17: 0xfc607f (stale) Mirror 18: 0xfc607f (stale) Mirror 19: 0xfc607f (stale) Mirror 20: 0xfc607f (stale) Mirror 21: 0xfc607f (stale) Mirror 23: 0xfc607f (stale) Mirror 24: 0x5dd78562 (stale) CRC-32 checksum value for chunk [0, 0x256dd): Mirror 1: 0x5f758db0 Mirror 25: 0x5f758db0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256dd: skipped 000749 1788798192.220499 mapread 0x00c29e thru 0x0110a5 (0x4e08 bytes) 000750 1788798192.322138 mapwrite 0x02771f thru 0x02f836 (0x8118 bytes) 000751 1788798192.768686 trunc from 0x02f837 to 0x02201d (0xd819 bytes) 000752 1788798192.843799 trunc from 0x02201e to 0x032cd6 (0x10cb9 bytes) 000753 1788798192.924701 mapwrite 0x00baba thru 0x0101e2 (0x4729 bytes) 000754 1788798193.046501 punch from 0x032756 to 0x032cd6 (0x581 bytes) 000755 1788798193.063397 write 0x003864 thru 0x006bff (0x339c bytes) 000756 1788798193.081103 punch from 0x013e68 to 0x013f58 (0xf1 bytes) 000758 1788798193.113193 mapread 0x008129 thru 0x013516 (0xb3ee bytes) 000762 1788798193.129407 mapread 0x001a7d thru 0x00627e (0x4802 bytes) 000763 1788798193.137896 punch from 0x01f2c0 to 0x026a80 (0x77c1 bytes) 000764 1788798193.149482 mapwrite 0x024412 thru 0x02f900 (0xb4ef bytes) 000765 1788798193.248445 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32cd7): Mirror 10: 0xb574e7f4 (stale) Mirror 14: 0xb574e7f4 (stale) Mirror 15: 0xb574e7f4 (stale) Mirror 17: 0xb574e7f4 (stale) Mirror 19: 0xb574e7f4 (stale) Mirror 20: 0xb574e7f4 (stale) Mirror 21: 0xb574e7f4 (stale) Mirror 23: 0xb574e7f4 (stale) Mirror 24: 0x27c1fee0 (stale) Mirror 25: 0x7968f9a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32cd7): Mirror 10: 0xb574e7f4 (stale) Mirror 14: 0xb574e7f4 (stale) Mirror 15: 0xb574e7f4 (stale) Mirror 17: 0xb574e7f4 (stale) Mirror 19: 0xb574e7f4 (stale) Mirror 20: 0xb574e7f4 (stale) Mirror 21: 0xb574e7f4 (stale) Mirror 23: 0xb574e7f4 (stale) Mirror 24: 0x27c1fee0 (stale) Mirror 25: 0x7968f9a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32cd7: skipped 000767 1788798194.033581 punch from 0x023604 to 0x028a18 (0x5415 bytes) 000768 1788798194.040627 write 0x034a9d thru 0x0373cb (0x292f bytes) 000769 1788798194.043235 trunc from 0x0373cc to 0x02d310 (0xa0bb bytes) 000770 1788798194.078601 trunc from 0x02d311 to 0x010ab1 (0x1c85f bytes) 000771 1788798194.100692 read 0x00d9c8 thru 0x010ab1 (0x30ea bytes) 000775 1788798194.120595 write 0x025afd thru 0x02f8a4 (0x9da8 bytes) 000776 1788798194.122634 mapwrite 0x0076a6 thru 0x0145b7 (0xcf12 bytes) 000777 1788798194.266800 trunc from 0x02f8a5 to 0x03a0cf (0xa82b bytes) 000778 1788798194.292187 write 0x011922 thru 0x01acf8 (0x93d7 bytes) 000779 1788798194.294806 write 0x0122da thru 0x015d18 (0x3a3f bytes) 000780 1788798194.300159 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 10: 0xf7f31c53 (stale) Mirror 14: 0xf7f31c53 (stale) Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 1: 0x216f844e Mirror 26: 0x216f844e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d0: skipped 000781 1788798196.202220 read 0x000531 thru 0x002e1f (0x28ef bytes) 000782 1788798196.224141 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 10: 0xf7f31c53 (stale) Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 10: 0xf7f31c53 (stale) Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 1: 0x216f844e Mirror 26: 0x216f844e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d0: skipped 000783 1788798198.457135 mapwrite 0x027272 thru 0x03098d (0x971c bytes) 000785 1788798198.823280 read 0x00ca71 thru 0x00d46d (0x9fd bytes) 000787 1788798198.835854 write 0x00320f thru 0x00a80c (0x75fe bytes) 000789 1788798198.878561 read 0x00f292 thru 0x012a54 (0x37c3 bytes) 000790 1788798198.914450 read 0x010dac thru 0x0175b8 (0x680d bytes) 000791 1788798198.924876 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) Mirror 26: 0x216f844e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) Mirror 26: 0x216f844e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d0: skipped 000792 1788798200.186239 write 0x00dc75 thru 0x00fe62 (0x21ee bytes) 000795 1788798200.211028 mapread 0x0177b1 thru 0x01b338 (0x3b88 bytes) 000796 1788798200.319453 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 15: 0xf7f31c53 (stale) Mirror 17: 0xf7f31c53 (stale) Mirror 19: 0xf7f31c53 (stale) Mirror 20: 0xf7f31c53 (stale) Mirror 21: 0xf7f31c53 (stale) Mirror 23: 0xf7f31c53 (stale) Mirror 24: 0x51c26fd4 (stale) Mirror 25: 0x74b2ade9 (stale) Mirror 26: 0x216f844e (stale) CRC-32 checksum value for chunk [0, 0x3a0d0): Mirror 1: 0x18a12abd Mirror 27: 0x18a12abd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d0: skipped 000798 1788798202.115570 punch from 0x01ed33 to 0x01ed50 (0x1e bytes) 000799 1788798202.312275 MIRROR_RESYNC 11 mirrors 000800 1788798205.437046 MIRROR_RESYNC 11 mirrors 000801 1788798206.188016 mapread 0x01224e thru 0x01c2ab (0xa05e bytes) 000802 1788798206.209467 trunc from 0x03a0d0 to 0x02633d (0x13d92 bytes) 000804 1788798206.303569 trunc from 0x02633e to 0x036061 (0xfd24 bytes) 000806 1788798206.362105 punch from 0x0179a9 to 0x0276bc (0xfd14 bytes) 000807 1788798206.400044 punch from 0x00d25f to 0x017533 (0xa2d5 bytes) 000808 1788798206.420946 punch from 0x015013 to 0x022937 (0xd925 bytes) 000812 1788798206.432462 read 0x00f5f7 thru 0x01b58d (0xbf97 bytes) 000813 1788798206.449304 write 0x01965d thru 0x01cba4 (0x3548 bytes) 000814 1788798206.457259 trunc from 0x036062 to 0x0243a0 (0x11cc1 bytes) 000815 1788798206.488295 trunc from 0x0243a1 to 0x002f04 (0x2149c bytes) 000816 1788798206.518434 read 0x001f11 thru 0x002f04 (0xff4 bytes) 000818 1788798206.535046 write 0x0083f0 thru 0x00b48b (0x309c bytes) 000819 1788798206.536867 read 0x002e9c thru 0x00b48b (0x85f0 bytes) 000821 1788798206.547858 write 0x03fd6f thru 0x03ffff (0x291 bytes) 000822 1788798206.549236 read 0x028903 thru 0x036604 (0xdd02 bytes) 000823 1788798206.565254 mapwrite 0x0105c8 thru 0x01a5c0 (0x9ff9 bytes) 000824 1788798206.628636 read 0x02c7ff thru 0x0332e6 (0x6ae8 bytes) 000826 1788798206.629986 mapread 0x001a7d thru 0x00f5ed (0xdb71 bytes) 000827 1788798206.638073 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc95161f4 (stale) Mirror 17: 0xc95161f4 (stale) Mirror 19: 0xc95161f4 (stale) Mirror 20: 0xc95161f4 (stale) Mirror 21: 0xc95161f4 (stale) Mirror 23: 0xc95161f4 (stale) Mirror 24: 0xc95161f4 (stale) Mirror 25: 0xc95161f4 (stale) Mirror 26: 0xc95161f4 (stale) Mirror 27: 0xc95161f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x961029fa Mirror 28: 0x961029fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000828 1788798208.020969 write 0x00f21e thru 0x01cfc3 (0xdda6 bytes) 000829 1788798208.149999 mapread 0x02f9a1 thru 0x0345f4 (0x4c54 bytes) 000830 1788798208.170256 mapwrite 0x008e90 thru 0x010258 (0x73c9 bytes) 000831 1788798208.230918 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc95161f4 (stale) Mirror 17: 0xc95161f4 (stale) Mirror 19: 0xc95161f4 (stale) Mirror 20: 0xc95161f4 (stale) Mirror 21: 0xc95161f4 (stale) Mirror 23: 0xc95161f4 (stale) Mirror 24: 0xc95161f4 (stale) Mirror 25: 0xc95161f4 (stale) Mirror 26: 0xc95161f4 (stale) Mirror 27: 0xc95161f4 (stale) Mirror 28: 0x961029fa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe2197439 Mirror 29: 0xe2197439 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000834 1788798209.356809 punch from 0x01f06a to 0x022350 (0x32e7 bytes) 000835 1788798209.542992 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc95161f4 (stale) Mirror 17: 0xc95161f4 (stale) Mirror 19: 0xc95161f4 (stale) Mirror 20: 0xc95161f4 (stale) Mirror 21: 0xc95161f4 (stale) Mirror 23: 0xc95161f4 (stale) Mirror 24: 0xc95161f4 (stale) Mirror 25: 0xc95161f4 (stale) Mirror 27: 0xc95161f4 (stale) Mirror 28: 0x961029fa (stale) Mirror 29: 0xe2197439 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc95161f4 (stale) Mirror 17: 0xc95161f4 (stale) Mirror 19: 0xc95161f4 (stale) Mirror 20: 0xc95161f4 (stale) Mirror 21: 0xc95161f4 (stale) Mirror 23: 0xc95161f4 (stale) Mirror 24: 0xc95161f4 (stale) Mirror 25: 0xc95161f4 (stale) Mirror 27: 0xc95161f4 (stale) Mirror 28: 0x961029fa (stale) Mirror 29: 0xe2197439 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000837 1788798210.136839 read 0x00fab4 thru 0x01413d (0x468a bytes) 000838 1788798210.145159 punch from 0x0264d0 to 0x031be1 (0xb712 bytes) 000839 1788798210.149314 trunc from 0x040000 to 0x0273ed (0x18c12 bytes) 000840 1788798210.189080 MIRROR_RESYNC 12 mirrors 000842 1788798212.871108 mapwrite 0x017391 thru 0x01c8d1 (0x5541 bytes) 000843 1788798213.187814 mapread 0x01d460 thru 0x0273ed (0x9f8e bytes) 000844 1788798213.201460 mapread 0x01a6eb thru 0x0233a5 (0x8cbb bytes) 000845 1788798213.202741 read 0x014ff1 thru 0x01a73f (0x574f bytes) 000846 1788798213.204833 mapwrite 0x00acb8 thru 0x018a9e (0xdde7 bytes) 000847 1788798213.341759 write 0x03d1f4 thru 0x03ffff (0x2e0c bytes) 000848 1788798213.343862 mapwrite 0x021207 thru 0x02cfea (0xbde4 bytes) 000849 1788798213.452154 mapwrite 0x037844 thru 0x03ffff (0x87bc bytes) 000852 1788798213.552464 write 0x013243 thru 0x015b8b (0x2949 bytes) 000854 1788798213.555047 trunc from 0x040000 to 0x011edd (0x2e122 bytes) 000855 1788798213.588027 write 0x005e9e thru 0x00917c (0x32df bytes) 000856 1788798213.595838 trunc from 0x011ede to 0x03b408 (0x2952b bytes) 000857 1788798213.624984 punch from 0x01f9f0 to 0x02038f (0x9a0 bytes) 000858 1788798213.658458 read 0x01c8af thru 0x0263b8 (0x9b0a bytes) 000859 1788798213.683537 mapwrite 0x017c83 thru 0x023e15 (0xc193 bytes) 000861 1788798213.746870 trunc from 0x03b409 to 0x03cc96 (0x188e bytes) 000862 1788798213.789197 trunc from 0x03cc97 to 0x00119f (0x3baf7 bytes) 000863 1788798213.816633 mapread 0x000b33 thru 0x00119f (0x66d bytes) 000864 1788798213.817697 mapread 0x000f8e thru 0x00119f (0x212 bytes) 000865 1788798213.821232 trunc from 0x0011a0 to 0x018ba6 (0x17a07 bytes) 000866 1788798213.859853 write 0x02fc34 thru 0x035085 (0x5452 bytes) 000867 1788798213.863308 trunc from 0x035086 to 0x03a0d8 (0x5053 bytes) 000868 1788798213.898510 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a0d9): Mirror 15: 0x7d0babe3 (stale) Mirror 17: 0x7d0babe3 (stale) Mirror 19: 0x7d0babe3 (stale) Mirror 20: 0x7d0babe3 (stale) Mirror 21: 0x7d0babe3 (stale) Mirror 23: 0x7d0babe3 (stale) Mirror 25: 0x7d0babe3 (stale) Mirror 27: 0x7d0babe3 (stale) Mirror 28: 0x7d0babe3 (stale) Mirror 29: 0x7d0babe3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d9): Mirror 15: 0x7d0babe3 (stale) Mirror 17: 0x7d0babe3 (stale) Mirror 19: 0x7d0babe3 (stale) Mirror 20: 0x7d0babe3 (stale) Mirror 21: 0x7d0babe3 (stale) Mirror 23: 0x7d0babe3 (stale) Mirror 25: 0x7d0babe3 (stale) Mirror 27: 0x7d0babe3 (stale) Mirror 28: 0x7d0babe3 (stale) Mirror 29: 0x7d0babe3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d9: skipped 000869 1788798214.422632 mapwrite 0x03bdb6 thru 0x03ffff (0x424a bytes) 000870 1788798214.482851 punch from 0x03e521 to 0x03ffff (0x1adf bytes) 000872 1788798214.487847 mapwrite 0x0030ad thru 0x004dab (0x1cff bytes) 000873 1788798214.528312 read 0x03d06e thru 0x03e04d (0xfe0 bytes) 000875 1788798214.542033 mapwrite 0x0363ca thru 0x03ffff (0x9c36 bytes) 000876 1788798214.613833 mapread 0x02fbd0 thru 0x03ea59 (0xee8a bytes) 000878 1788798214.625017 trunc from 0x040000 to 0x00c440 (0x33bbf bytes) 000879 1788798214.640308 punch from 0x00a07f to 0x00c440 (0x23c2 bytes) 000880 1788798214.652114 trunc from 0x00c441 to 0x0230ff (0x16cbf bytes) 000881 1788798214.680628 read 0x003d03 thru 0x012045 (0xe343 bytes) 000883 1788798214.695537 mapwrite 0x03e7e1 thru 0x03ffff (0x181f bytes) 000884 1788798214.797454 trunc from 0x040000 to 0x025f31 (0x1a0ce bytes) 000887 1788798214.827877 mapwrite 0x020dba thru 0x0278b7 (0x6afe bytes) 000888 1788798214.919285 punch from 0x01e6b1 to 0x026282 (0x7bd2 bytes) 000890 1788798214.929997 mapread 0x014628 thru 0x01641c (0x1df5 bytes) 000892 1788798214.937160 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x278b8): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278b8): Mirror 15: 0x43ce24ae (stale) Mirror 17: 0x43ce24ae (stale) Mirror 19: 0x43ce24ae (stale) Mirror 20: 0x43ce24ae (stale) Mirror 21: 0x43ce24ae (stale) Mirror 23: 0x43ce24ae (stale) Mirror 25: 0x43ce24ae (stale) Mirror 27: 0x43ce24ae (stale) Mirror 28: 0x43ce24ae (stale) Mirror 29: 0x43ce24ae (stale) CRC-32 checksum value for chunk [0, 0x278b8): Mirror 1: 0x70a47da9 Mirror 30: 0x70a47da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278b8: skipped 000893 1788798217.287832 trunc from 0x0278b8 to 0x02ce1e (0x5567 bytes) 000894 1788798217.456781 punch from 0x00b7cc to 0x00c5ec (0xe21 bytes) 000896 1788798217.463779 trunc from 0x02ce1f to 0x00b383 (0x21a9b bytes) 000897 1788798217.483733 write 0x02bcf9 thru 0x032801 (0x6b09 bytes) 000898 1788798217.486108 trunc from 0x032802 to 0x00c4eb (0x26316 bytes) 000899 1788798217.503402 mapread 0x00a621 thru 0x00c4eb (0x1ecb bytes) 000900 1788798217.523518 read 0x00532c thru 0x00c4eb (0x71c0 bytes) 000903 1788798217.525954 trunc from 0x00c4ec to 0x0039d2 (0x8b19 bytes) 000904 1788798217.547823 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d3): Mirror 15: 0x912115ba (stale) Mirror 17: 0x912115ba (stale) Mirror 19: 0x912115ba (stale) Mirror 20: 0x912115ba (stale) Mirror 21: 0x912115ba (stale) Mirror 23: 0x912115ba (stale) Mirror 25: 0x912115ba (stale) Mirror 28: 0x912115ba (stale) Mirror 29: 0x912115ba (stale) Mirror 30: 0x16169c45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d3): Mirror 15: 0x912115ba (stale) Mirror 17: 0x912115ba (stale) Mirror 19: 0x912115ba (stale) Mirror 20: 0x912115ba (stale) Mirror 21: 0x912115ba (stale) Mirror 23: 0x912115ba (stale) Mirror 25: 0x912115ba (stale) Mirror 28: 0x912115ba (stale) Mirror 29: 0x912115ba (stale) Mirror 30: 0x16169c45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d3: skipped 000905 1788798217.981318 trunc from 0x0039d3 to 0x00c1a6 (0x87d4 bytes) 000906 1788798218.004469 trunc from 0x00c1a7 to 0x016a8c (0xa8e6 bytes) 000911 1788798218.032130 write 0x035f51 thru 0x03ffff (0xa0af bytes) 000913 1788798218.041809 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xaa272d58 (stale) Mirror 17: 0xaa272d58 (stale) Mirror 19: 0xaa272d58 (stale) Mirror 20: 0xaa272d58 (stale) Mirror 21: 0xaa272d58 (stale) Mirror 23: 0xaa272d58 (stale) Mirror 25: 0xaa272d58 (stale) Mirror 28: 0xaa272d58 (stale) Mirror 29: 0xaa272d58 (stale) Mirror 30: 0x33af02b5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x644c8c21 Mirror 31: 0x644c8c21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000914 1788798219.862734 write 0x014db7 thru 0x01d4fe (0x8748 bytes) 000916 1788798219.955460 trunc from 0x040000 to 0x0300c0 (0xff3f bytes) 000917 1788798219.980573 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x300c1): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x300c1): Mirror 15: 0x5e11f032 (stale) Mirror 17: 0x5e11f032 (stale) Mirror 19: 0x5e11f032 (stale) Mirror 20: 0x5e11f032 (stale) Mirror 21: 0x5e11f032 (stale) Mirror 23: 0x5e11f032 (stale) Mirror 25: 0x5e11f032 (stale) Mirror 28: 0x5e11f032 (stale) Mirror 29: 0x5e11f032 (stale) Mirror 30: 0x805624b (stale) Mirror 31: 0xa9687a6b (stale) CRC-32 checksum value for chunk [0, 0x300c1): Mirror 1: 0xe90c5c2 Mirror 32: 0xe90c5c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x300c1: skipped 000918 1788798222.332267 read 0x004af4 thru 0x0090b0 (0x45bd bytes) 000919 1788798222.361626 MIRROR_RESYNC 13 mirrors 000920 1788798226.604456 read 0x01f256 thru 0x023dc5 (0x4b70 bytes) 000921 1788798226.615325 mapwrite 0x005b2d thru 0x00f126 (0x95fa bytes) 000922 1788798226.808122 punch from 0x00db42 to 0x01b0e2 (0xd5a1 bytes) 000923 1788798226.814489 trunc from 0x0300c1 to 0x00845a (0x27c66 bytes) 000924 1788798226.856897 MIRROR_RESYNC 13 mirrors 000925 1788798228.824681 mapread 0x003ec7 thru 0x004c2c (0xd66 bytes) 000926 1788798228.836801 punch from 0x007850 to 0x00845a (0xc0b bytes) 000927 1788798228.956491 write 0x002689 thru 0x00bdd6 (0x974e bytes) 000929 1788798228.967616 MIRROR_RESYNC 13 mirrors 000931 1788798230.956600 mapread 0x00a031 thru 0x00bdd6 (0x1da6 bytes) 000932 1788798230.975326 MIRROR_RESYNC 13 mirrors 000933 1788798231.680846 punch from 0x001e0e to 0x00b5a8 (0x979b bytes) 000935 1788798231.764358 mapread 0x003545 thru 0x00ae8f (0x794b bytes) 000936 1788798231.801429 MIRROR_RESYNC 13 mirrors 000939 1788798233.724694 MIRROR_EXTEND to 14 mirrors 000940 1788798234.843508 punch from 0x005ebe to 0x00bdd6 (0x5f19 bytes) 000941 1788798234.940788 read 0x009a80 thru 0x00bdd6 (0x2357 bytes) 000943 1788798234.950598 read 0x0070b4 thru 0x007743 (0x690 bytes) 000945 1788798234.961497 mapwrite 0x018ba0 thru 0x024db4 (0xc215 bytes) 000946 1788798235.069922 punch from 0x013983 to 0x02125d (0xd8db bytes) 000947 1788798235.078836 mapwrite 0x0057f1 thru 0x00f30d (0x9b1d bytes) 000948 1788798235.123917 mapread 0x0039c2 thru 0x003b78 (0x1b7 bytes) 000949 1788798235.124866 write 0x0273a9 thru 0x02ea82 (0x76da bytes) 000950 1788798235.126111 trunc from 0x02ea83 to 0x02cced (0x1d95 bytes) 000951 1788798235.152544 read 0x00d0ba thru 0x01ad4d (0xdc94 bytes) 000952 1788798235.172233 write 0x00e444 thru 0x019dd4 (0xb991 bytes) 000953 1788798235.174598 trunc from 0x02ccee to 0x01fb55 (0xd198 bytes) 000954 1788798235.205455 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 15: 0x8c5e2abe (stale) Mirror 17: 0x8c5e2abe (stale) Mirror 19: 0x8c5e2abe (stale) Mirror 20: 0x8c5e2abe (stale) Mirror 21: 0x8c5e2abe (stale) Mirror 23: 0x8c5e2abe (stale) Mirror 25: 0x8c5e2abe (stale) Mirror 28: 0x8c5e2abe (stale) Mirror 29: 0x8c5e2abe (stale) Mirror 30: 0x8c5e2abe (stale) Mirror 31: 0x8c5e2abe (stale) Mirror 32: 0x8c5e2abe (stale) Mirror 33: 0x8c5e2abe (stale) CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 1: 0x6b8103c0 Mirror 34: 0x6b8103c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb56: skipped 000958 1788798236.745942 mapread 0x0074e3 thru 0x0090c1 (0x1bdf bytes) 000960 1788798236.781084 mapread 0x00e2f8 thru 0x0113e2 (0x30eb bytes) 000961 1788798236.790427 mapwrite 0x014456 thru 0x019da7 (0x5952 bytes) 000962 1788798236.995174 mapread 0x01e4a0 thru 0x01fb55 (0x16b6 bytes) 000964 1788798237.005728 mapread 0x00a81a thru 0x00c0f7 (0x18de bytes) 000965 1788798237.017161 mapread 0x006b38 thru 0x015061 (0xe52a bytes) 000966 1788798237.018279 mapread 0x00e019 thru 0x0156eb (0x76d3 bytes) 000967 1788798237.023198 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 17: 0x8c5e2abe (stale) Mirror 19: 0x8c5e2abe (stale) Mirror 20: 0x8c5e2abe (stale) Mirror 21: 0x8c5e2abe (stale) Mirror 23: 0x8c5e2abe (stale) Mirror 25: 0x8c5e2abe (stale) Mirror 28: 0x8c5e2abe (stale) Mirror 29: 0x8c5e2abe (stale) Mirror 30: 0x8c5e2abe (stale) Mirror 31: 0x8c5e2abe (stale) Mirror 32: 0x8c5e2abe (stale) Mirror 33: 0x8c5e2abe (stale) Mirror 34: 0x6b8103c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 17: 0x8c5e2abe (stale) Mirror 19: 0x8c5e2abe (stale) Mirror 20: 0x8c5e2abe (stale) Mirror 21: 0x8c5e2abe (stale) Mirror 23: 0x8c5e2abe (stale) Mirror 25: 0x8c5e2abe (stale) Mirror 28: 0x8c5e2abe (stale) Mirror 29: 0x8c5e2abe (stale) Mirror 30: 0x8c5e2abe (stale) Mirror 31: 0x8c5e2abe (stale) Mirror 32: 0x8c5e2abe (stale) Mirror 33: 0x8c5e2abe (stale) Mirror 34: 0x6b8103c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb56: skipped 000969 1788798237.633433 punch from 0x00a964 to 0x00ebec (0x4289 bytes) 000970 1788798237.643479 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 17: 0x8c5e2abe (stale) Mirror 19: 0x8c5e2abe (stale) Mirror 20: 0x8c5e2abe (stale) Mirror 21: 0x8c5e2abe (stale) Mirror 23: 0x8c5e2abe (stale) Mirror 25: 0x8c5e2abe (stale) Mirror 28: 0x8c5e2abe (stale) Mirror 29: 0x8c5e2abe (stale) Mirror 30: 0x8c5e2abe (stale) Mirror 31: 0x8c5e2abe (stale) Mirror 32: 0x8c5e2abe (stale) Mirror 33: 0x8c5e2abe (stale) Mirror 34: 0x6b8103c0 (stale) CRC-32 checksum value for chunk [0, 0x1fb56): Mirror 1: 0x494d3925 Mirror 35: 0x494d3925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb56: skipped 000972 1788798239.140835 mapread 0x018fee thru 0x01b091 (0x20a4 bytes) 000974 1788798239.197482 mapwrite 0x02e02b thru 0x038407 (0xa3dd bytes) 000976 1788798239.453883 MIRROR_RESYNC 15 mirrors 000977 1788798242.808905 mapread 0x02b2c5 thru 0x038294 (0xcfd0 bytes) 000978 1788798242.876550 trunc from 0x038408 to 0x0347fe (0x3c09 bytes) 000979 1788798242.986796 write 0x016bce thru 0x01aac3 (0x3ef6 bytes) 000980 1788798243.026816 punch from 0x00e4e7 to 0x015791 (0x72ab bytes) 000982 1788798243.057031 trunc from 0x0347ff to 0x03ade3 (0x65e5 bytes) 000983 1788798243.111278 mapwrite 0x013f67 thru 0x018382 (0x441c bytes) 000985 1788798243.206026 mapread 0x02688c thru 0x02976f (0x2ee4 bytes) 000986 1788798243.223783 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ade4): Mirror 17: 0xa7525991 (stale) Mirror 19: 0xa7525991 (stale) Mirror 20: 0xa7525991 (stale) Mirror 21: 0xa7525991 (stale) Mirror 23: 0xa7525991 (stale) Mirror 28: 0xa7525991 (stale) Mirror 29: 0xa7525991 (stale) Mirror 30: 0xa7525991 (stale) Mirror 31: 0xa7525991 (stale) Mirror 32: 0xa7525991 (stale) Mirror 33: 0xa7525991 (stale) Mirror 34: 0xa7525991 (stale) Mirror 35: 0xa7525991 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ade4): Mirror 17: 0xa7525991 (stale) Mirror 19: 0xa7525991 (stale) Mirror 20: 0xa7525991 (stale) Mirror 21: 0xa7525991 (stale) Mirror 23: 0xa7525991 (stale) Mirror 28: 0xa7525991 (stale) Mirror 29: 0xa7525991 (stale) Mirror 30: 0xa7525991 (stale) Mirror 31: 0xa7525991 (stale) Mirror 32: 0xa7525991 (stale) Mirror 33: 0xa7525991 (stale) Mirror 34: 0xa7525991 (stale) Mirror 35: 0xa7525991 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ade4: skipped 000987 1788798244.528437 mapread 0x006f67 thru 0x00ced4 (0x5f6e bytes) 000990 1788798244.574530 trunc from 0x03ade4 to 0x018536 (0x228ad bytes) 000992 1788798244.623784 read 0x01479c thru 0x018536 (0x3d9b bytes) 000993 1788798244.647841 mapwrite 0x0140c8 thru 0x01eec3 (0xadfc bytes) 000998 1788798244.776750 read 0x006816 thru 0x01316c (0xc957 bytes) 001000 1788798244.807313 MIRROR_RESYNC 14 mirrors All operations completed A-OK! PASS 70a (192s) == sanity-flr test 71: check mirror extend parent fid ==== 12:24:13 (1788798253) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 12:24:16 (1788798256) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31258 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142525 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 12:24:28 (1788798268) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7250976 .. Extending file size to 4887296 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6099040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8144864 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8372800 .. Extending file size to 8689376 .. Extending file size to 8019008 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8350880 .. Extending file size to 8821536 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9197248 .. Extending file size to 8886240 .. Extending file size to 8979840 .. Waiting 57518 57520 57522 57523 57524 failed 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:25:51 (1788798351) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4409152 .. Extending file size to 3715680 .. failed Extending file size to 8540960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7117440 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8287168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8605568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8926528 .. Extending file size to 9361440 .. Extending file size to 8662528 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59069 59070 59072 59074 59075 done 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:27:15 (1788798435) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107134 s, 19.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.507949 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf0415ff8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:27:38 (1788798458) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100624 s, 20.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 12:27:46 (1788798466) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:27:54 (1788798474) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330657 s, 6.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:28:04 (1788798484) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127184 s, 16.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:28:12 (1788798492) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208554 s, 10.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:28:46 (1788798526) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143974 s, 7.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:29:50 (1788798590) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.325778 s, 6.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:30:53 (1788798653) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. Creating new pool test_204f oleg117-server: Pool lustre.test_204f created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315821 s, 6.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg117-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:32:14 (1788798734) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:32:23 (1788798743) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:32:31 (1788798751) oleg117-server: Pool lustre.test_206 created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg117-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:32:58 (1788798778) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139399 s, 7.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934321 s, 11.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2eb71d33 Mirror 3: 0x2eb71d33 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:33:09 (1788798789) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:33:14 (1788798794) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:33:19 (1788798799) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.196333 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1790000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1790000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1790000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1af0000 Read this (4 bytes): BBBB 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:34:06 (1788798846) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0327148 s, 250 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:34:14 (1788798854) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652045 s, 16.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20c lmm_fid: [0x200000404:0x20c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20c lmm_fid: [0x200000404:0x20c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834444 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 12:34:16.000000000 -0400 Modify: 2026-09-07 12:34:16.000000000 -0400 Change: 2026-09-07 12:34:17.000000000 -0400 Birth: 2026-09-07 12:34:15.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:34:24 (1788798864) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:34:29 (1788798869) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.486862 s, 84.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0597364 s, 68.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 12:34 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 12:34 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:34:38 (1788798878) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:34:46 (1788798886) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2057 sec =========== 12:34:58 (1788798898) === sanity-flr: start cleanup 12:35:00 (1788798900) === === sanity-flr: finish cleanup 12:35:08 (1788798908) ===