-----============= acceptance-small: sanity-flr ============----- Fri Feb 27 01:57:56 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:58:15 (1772175495) === oleg420-client.virtnet: executing check_config_client /mnt/lustre oleg420-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg420-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff902644ddc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff902644ddc000.idle_timeout=debug disable quota as required oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:58:49 (1772175529) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7558] == sanity-flr test 0a: lfs mirror create with -N option == 01:58:52 (1772175532) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:59:08 (1772175548) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:59:12 (1772175552) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:59:15 (1772175555) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:59:39 (1772175579) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:59:42 (1772175582) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:59:46 (1772175586) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff902644ddc000.stats=clear osc.lustre-OST0001-osc-ffff902644ddc000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:59:54 (1772175594) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:00:03 (1772175603) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:00:13 (1772175613) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:00:16 (1772175616) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:00:24 (1772175624) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:00:31 (1772175631) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:00:35 (1772175635) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:00:41 (1772175641) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:00:50 (1772175650) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:00:56 (1772175656) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.0921 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103795 s, 10.1 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:01:06 (1772175666) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324407 s, 32.3 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:01:13 (1772175673) Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (63s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:02:16 (1772175736) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.537744 s, 19.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:02:50 (1772175770) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:03:57 (1772175837) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.02314 s, 13.0 MB/s -rw-r--r-- 1 root root 52428800 Feb 27 02:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.155s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.084s user 0m0.001s sys 0m0.180s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.317s user 0m0.002s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902644ddc000.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:05:18 (1772175918) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:05:21 (1772175921) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:05:25 (1772175925) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:05:28 (1772175928) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 02:05:35 (1772175935) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.490876 s, 25.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:05:47 (1772175947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291359 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04975 s, 21.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:05:54 (1772175954) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:06:02 (1772175962) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000874632 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00209563 s, 489 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105523 s, 970 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00238124 s, 430 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 02:06:13 (1772175973) create test files with size 15747183 .. 19f62c4ee358f150a6189e940076a16c - 29d112c993b43fbcdb4c85adefce1afc - 37ed9062eeaa709807ba18722c685d54 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 02:06:37 (1772175997) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:06:52 (1772176012) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:06:58 (1772176018) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953317 s, 11.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426353 s, 24.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:07:06 (1772176026) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113237 s, 18.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.257296 s, 16.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134006 s, 15.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 02:07:19 (1772176039) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:07:22 (1772176042) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:07:24 (1772176044) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234615 s, 44.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 02:07:31 (1772176051) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:07:34 (1772176054) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:07:44 (1772176064) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.235418 s, 44.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:07:55 (1772176075) llite.lustre-ffff902645bdc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.261627 s, 40.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff902644ddf000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:08:07 (1772176087) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:08:37 (1772176117) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:09:06 (1772176146) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:09:10 (1772176150) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241567 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138616 s, 22.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8b16d907 (stale) Mirror 2: 0x8b16d907 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 0fd076e66b96ddfbafbea982ea30f728 - mirror 1 checksum: 7459dc8b585e384ffa255a93c0eedeff - mirror 2 checksum: 0fd076e66b96ddfbafbea982ea30f728 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 27 02:09:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 27 02:09:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0fd076e66b96ddfbafbea982ea30f728 - mirror 1 checksum: 7459dc8b585e384ffa255a93c0eedeff - mirror 2 checksum: 7459dc8b585e384ffa255a93c0eedeff - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 27 02:09:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 27 02:09:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:09:20 (1772176160) Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 02:09:25 (1772176165) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server Failover mds1 to oleg420-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 02:09:47 (1772176187) targets are mounted 02:09:47 (1772176187) facet_failover done oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022949 s, 45.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:10:00 (1772176200) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:10:03 (1772176203) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:10:06 (1772176206) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0114195 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:10:12 (1772176212) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:10:15 (1772176215) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:10:18 (1772176218) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:10:37 (1772176237) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:11:05 (1772176265) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:11:28 (1772176288) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:11:52 (1772176312) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 02:11:59 (1772176319) mirror create pid 52301 mirror split pid 52302 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 52301 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 52302 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 02:13:06 (1772176386) Seed set to 6387 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1772176388.003121 write 0x013e3a thru 0x01d75d (0x9924 bytes) truncating to largest ever: 0x2f1d9 000003 1772176388.009693 trunc from 0x01d75e to 0x02f1d8 (0x11a7b bytes) 000005 1772176388.027340 trunc from 0x02f1d9 to 0x02d29f (0x1f39 bytes) 000006 1772176388.043469 write 0x00ab76 thru 0x00c9b2 (0x1e3d bytes) 000007 1772176388.053669 read 0x0168dc thru 0x018c35 (0x235a bytes) 000008 1772176388.054741 mapwrite 0x01f27a thru 0x0256c6 (0x644d bytes) 000009 1772176388.108139 read 0x01c9a6 thru 0x0213ae (0x4a09 bytes) 000010 1772176388.109529 mapwrite 0x025013 thru 0x0274db (0x24c9 bytes) 000012 1772176388.145274 mapread 0x0218ef thru 0x0260f7 (0x4809 bytes) 000013 1772176388.148482 mapwrite 0x013876 thru 0x01fedd (0xc668 bytes) 000014 1772176388.207870 read 0x00e1a9 thru 0x01db25 (0xf97d bytes) 000015 1772176388.235919 write 0x00f960 thru 0x016197 (0x6838 bytes) 000017 1772176388.237516 write 0x00d71b thru 0x01c1cb (0xeab1 bytes) truncating to largest ever: 0x30494 000021 1772176388.249805 trunc from 0x02d2a0 to 0x030493 (0x31f4 bytes) 000023 1772176388.269160 mapwrite 0x0026db thru 0x006015 (0x393b bytes) 000024 1772176388.329134 mapread 0x020f3f thru 0x0254e3 (0x45a5 bytes) 000025 1772176388.331290 read 0x017336 thru 0x01b583 (0x424e bytes) 000026 1772176388.333418 write 0x020f47 thru 0x028d55 (0x7e0f bytes) 000027 1772176388.335959 write 0x002905 thru 0x0064ed (0x3be9 bytes) 000028 1772176388.337432 mapread 0x01e94a thru 0x02287d (0x3f34 bytes) 000030 1772176388.340272 read 0x01d9ed thru 0x02a7cd (0xcde1 bytes) 000033 1772176388.343345 mapread 0x014100 thru 0x01ffba (0xbebb bytes) 000034 1772176388.345298 write 0x003596 thru 0x0038d8 (0x343 bytes) 000036 1772176388.347282 write 0x015748 thru 0x024da6 (0xf65f bytes) 000037 1772176388.354906 MIRROR_EXTEND to 2 mirrors 000039 1772176388.858342 mapread 0x023564 thru 0x026b75 (0x3612 bytes) 000040 1772176388.882796 write 0x020722 thru 0x02e216 (0xdaf5 bytes) 000044 1772176388.964330 write 0x019fef thru 0x01aaa1 (0xab3 bytes) 000045 1772176388.970981 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) 000046 1772176389.141845 mapwrite 0x02e8de thru 0x0375a7 (0x8cca bytes) 000047 1772176389.357494 read 0x016d99 thru 0x01cc2f (0x5e97 bytes) 000048 1772176389.368960 mapwrite 0x0290d7 thru 0x0333fb (0xa325 bytes) 000050 1772176389.451466 mapwrite 0x02b480 thru 0x0331c9 (0x7d4a bytes) 000051 1772176389.503949 mapread 0x0188ea thru 0x023d99 (0xb4b0 bytes) 000053 1772176389.517383 mapwrite 0x038e9f thru 0x03fef0 (0x7052 bytes) 000054 1772176389.617266 mapwrite 0x03aa4d thru 0x03ffff (0x55b3 bytes) 000055 1772176389.686535 trunc from 0x040000 to 0x01dc5e (0x223a1 bytes) 000057 1772176389.716660 read 0x01815f thru 0x01dc5e (0x5b00 bytes) 000059 1772176389.722843 trunc from 0x01dc5f to 0x00a0d4 (0x13b8a bytes) 000060 1772176389.748780 read 0x00777f thru 0x00a0d4 (0x2956 bytes) 000062 1772176389.762795 mapread 0x003911 thru 0x009ceb (0x63db bytes) 000064 1772176389.767873 mapwrite 0x016a7f thru 0x01c67c (0x5bfe bytes) 000065 1772176389.858840 write 0x03fdcd thru 0x03ffff (0x233 bytes) 000066 1772176389.860256 write 0x02bb5e thru 0x0312b4 (0x5757 bytes) 000067 1772176389.870969 mapwrite 0x030fc3 thru 0x03bfa5 (0xafe3 bytes) 000070 1772176389.935402 mapread 0x002dd7 thru 0x00d844 (0xaa6e bytes) 000071 1772176389.943185 write 0x0382a9 thru 0x03ffff (0x7d57 bytes) 000073 1772176389.945889 mapread 0x0087dd thru 0x017709 (0xef2d bytes) 000074 1772176389.948127 write 0x03255c thru 0x03e3f6 (0xbe9b bytes) 000076 1772176389.951818 read 0x0337a3 thru 0x037cce (0x452c bytes) 000077 1772176389.952923 mapread 0x011809 thru 0x0161c7 (0x49bf bytes) 000078 1772176389.954612 mapread 0x03751f thru 0x03fb88 (0x866a bytes) 000079 1772176389.956834 write 0x026c18 thru 0x02da3e (0x6e27 bytes) 000080 1772176389.958820 write 0x01a70f thru 0x01b1c9 (0xabb bytes) 000083 1772176389.961944 mapread 0x0332b3 thru 0x03ce45 (0x9b93 bytes) 000085 1772176389.964954 write 0x005468 thru 0x0124f1 (0xd08a bytes) 000087 1772176389.969411 write 0x004bf7 thru 0x00adaa (0x61b4 bytes) 000090 1772176389.975246 trunc from 0x040000 to 0x00faf5 (0x3050a bytes) 000092 1772176390.020110 read 0x006c2c thru 0x00dd67 (0x713c bytes) 000093 1772176390.026063 trunc from 0x00faf6 to 0x00dd63 (0x1d92 bytes) 000094 1772176390.070182 mapwrite 0x02d26a thru 0x0375ac (0xa343 bytes) 000096 1772176390.210835 mapwrite 0x00d8db thru 0x01365e (0x5d84 bytes) 000097 1772176390.253178 mapread 0x00b09b thru 0x01a0e2 (0xf048 bytes) 000098 1772176390.254203 trunc from 0x0375ad to 0x009c2c (0x2d980 bytes) 000099 1772176390.269116 read 0x006e78 thru 0x009c2c (0x2db5 bytes) 000100 1772176390.273171 write 0x00e202 thru 0x015c70 (0x7a6f bytes) 000102 1772176390.275837 mapwrite 0x0259a3 thru 0x02b297 (0x58f5 bytes) 000104 1772176390.349814 trunc from 0x02b298 to 0x01407c (0x1721b bytes) 000105 1772176390.369862 trunc from 0x01407d to 0x00d286 (0x6df6 bytes) 000106 1772176390.389687 trunc from 0x00d287 to 0x01fd9a (0x12b14 bytes) 000107 1772176390.412173 trunc from 0x01fd9b to 0x0048d4 (0x1b4c6 bytes) 000110 1772176390.434466 MIRROR_EXTEND to 2 mirrors 000113 1772176390.940659 write 0x021ff8 thru 0x02c5a0 (0xa5a9 bytes) 000114 1772176391.044471 mapread 0x0041b7 thru 0x00fff2 (0xbe3c bytes) 000116 1772176391.069754 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c5a1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c5a1): Mirror 2: 0x23a4b1b9 (stale) CRC-32 checksum value for chunk [0, 0x2c5a1): Mirror 1: 0x44568fce Mirror 3: 0x44568fce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c5a1: skipped 000119 1772176391.883177 write 0x03d743 thru 0x03ffff (0x28bd bytes) 000121 1772176391.985953 read 0x01c0bf thru 0x01cce4 (0xc26 bytes) 000122 1772176392.000748 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc91d439 (stale) Mirror 3: 0xd7bc37ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x24256547 Mirror 4: 0x24256547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1772176392.913595 trunc from 0x040000 to 0x01a9cb (0x25634 bytes) 000126 1772176393.037199 write 0x034949 thru 0x03d2c0 (0x8978 bytes) 000128 1772176393.040817 read 0x01e87c thru 0x024431 (0x5bb6 bytes) 000129 1772176393.050770 read 0x036e0d thru 0x03a965 (0x3b59 bytes) 000132 1772176393.053561 mapwrite 0x007ac3 thru 0x00c1a3 (0x46e1 bytes) 000139 1772176393.176728 read 0x021de0 thru 0x024d24 (0x2f45 bytes) 000140 1772176393.177621 read 0x02f6b4 thru 0x030720 (0x106d bytes) 000141 1772176393.179096 mapread 0x018ca1 thru 0x01cbb7 (0x3f17 bytes) truncating to largest ever: 0x3223e 000142 1772176393.181130 trunc from 0x03d2c1 to 0x03223d (0xb083 bytes) 000143 1772176393.222437 MIRROR_RESYNC 4 mirrors 000144 1772176393.963143 mapwrite 0x00b867 thru 0x0194ec (0xdc86 bytes) 000145 1772176394.103681 read 0x0296ee thru 0x03223d (0x8b50 bytes) 000146 1772176394.109176 read 0x01a1ad thru 0x0207db (0x662f bytes) 000147 1772176394.125550 mapread 0x0208fa thru 0x0219db (0x10e2 bytes) 000148 1772176394.146396 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3223e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3223e): Mirror 2: 0x9c0f5404 (stale) Mirror 3: 0x9c0f5404 (stale) Mirror 4: 0x9c0f5404 (stale) CRC-32 checksum value for chunk [0, 0x3223e): Mirror 1: 0x5bc44418 Mirror 5: 0x5bc44418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3223e: skipped 000150 1772176394.859091 write 0x0134fc thru 0x018e90 (0x5995 bytes) 000151 1772176394.941030 MIRROR_RESYNC 5 mirrors 000152 1772176395.696747 mapwrite 0x002457 thru 0x005037 (0x2be1 bytes) 000153 1772176395.836803 trunc from 0x03223e to 0x01911b (0x19122 bytes) 000154 1772176395.857659 mapread 0x00a298 thru 0x00f10f (0x4e78 bytes) 000155 1772176395.858854 mapread 0x008897 thru 0x00fc78 (0x73e2 bytes) 000156 1772176395.860967 trunc from 0x01911c to 0x002b81 (0x1659a bytes) 000160 1772176395.882701 mapread 0x002287 thru 0x002b81 (0x8fb bytes) 000161 1772176395.883806 trunc from 0x002b82 to 0x015f97 (0x13416 bytes) 000162 1772176395.912319 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x15f98): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f98): Mirror 2: 0xe8d91a6a (stale) Mirror 3: 0xe8d91a6a (stale) Mirror 4: 0xe8d91a6a (stale) Mirror 5: 0xe8d91a6a (stale) CRC-32 checksum value for chunk [0, 0x15f98): Mirror 1: 0x1a89ced Mirror 6: 0x1a89ced lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f98: skipped 000164 1772176396.616095 mapread 0x00f05f thru 0x015f97 (0x6f39 bytes) 000165 1772176396.644570 mapwrite 0x039510 thru 0x03ffff (0x6af0 bytes) 000166 1772176396.789284 mapread 0x0362ed thru 0x03ffff (0x9d13 bytes) 000168 1772176396.792876 mapread 0x03fdec thru 0x03ffff (0x214 bytes) 000169 1772176396.794310 read 0x027f6d thru 0x031d66 (0x9dfa bytes) 000170 1772176396.814660 trunc from 0x040000 to 0x025e0b (0x1a1f4 bytes) 000172 1772176396.841714 trunc from 0x025e0c to 0x026ce8 (0xedd bytes) 000174 1772176396.857598 write 0x023337 thru 0x025198 (0x1e62 bytes) 000176 1772176396.873086 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26ce9): Mirror 2: 0xa40628c4 (stale) Mirror 3: 0xa40628c4 (stale) Mirror 5: 0xa40628c4 (stale) Mirror 6: 0xb6ddd915 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ce9): Mirror 2: 0xa40628c4 (stale) Mirror 3: 0xa40628c4 (stale) Mirror 5: 0xa40628c4 (stale) Mirror 6: 0xb6ddd915 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ce9: skipped 000178 1772176397.199359 write 0x01446c thru 0x018eae (0x4a43 bytes) 000179 1772176397.213684 mapwrite 0x01d92c thru 0x01eb09 (0x11de bytes) 000181 1772176397.300191 trunc from 0x026ce9 to 0x0157f7 (0x114f1 bytes) 000182 1772176397.319593 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x157f8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x157f8): Mirror 2: 0x5ac3c66d (stale) Mirror 3: 0x5ac3c66d (stale) Mirror 5: 0x5ac3c66d (stale) Mirror 6: 0xe43eb428 (stale) CRC-32 checksum value for chunk [0, 0x157f8): Mirror 1: 0xf58aa060 Mirror 7: 0xf58aa060 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x157f8: skipped 000183 1772176398.339819 write 0x00d342 thru 0x016989 (0x9648 bytes) 000184 1772176398.432467 read 0x00f44e thru 0x016989 (0x753c bytes) 000185 1772176398.434337 mapwrite 0x03a1de thru 0x03d4d3 (0x32f6 bytes) 000186 1772176398.535025 mapwrite 0x01e5dc thru 0x01e82d (0x252 bytes) 000187 1772176398.595546 read 0x00331c thru 0x01212b (0xee10 bytes) 000188 1772176398.607966 mapread 0x01a04b thru 0x01c22a (0x21e0 bytes) 000189 1772176398.609620 mapwrite 0x03c98e thru 0x03ffff (0x3672 bytes) 000190 1772176398.705415 write 0x00f10f thru 0x012291 (0x3183 bytes) 000191 1772176398.706623 read 0x0053b6 thru 0x010dd6 (0xba21 bytes) 000193 1772176398.722545 MIRROR_RESYNC 6 mirrors 000195 1772176399.846540 trunc from 0x040000 to 0x00a251 (0x35dae bytes) 000196 1772176399.923414 write 0x00b58c thru 0x013051 (0x7ac6 bytes) 000197 1772176399.924842 write 0x00e52c thru 0x01815c (0x9c31 bytes) 000199 1772176399.928681 trunc from 0x01815d to 0x001d6a (0x163f2 bytes) 000200 1772176399.955644 mapwrite 0x012747 thru 0x01e428 (0xbce2 bytes) 000201 1772176400.077304 mapread 0x001d63 thru 0x00e497 (0xc735 bytes) 000202 1772176400.091126 read 0x0100ad thru 0x014f16 (0x4e6a bytes) 000204 1772176400.102037 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e429): Mirror 3: 0x71f0d5f5 (stale) Mirror 5: 0x71f0d5f5 (stale) Mirror 6: 0x71f0d5f5 (stale) Mirror 7: 0x71f0d5f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e429): Mirror 3: 0x71f0d5f5 (stale) Mirror 5: 0x71f0d5f5 (stale) Mirror 6: 0x71f0d5f5 (stale) Mirror 7: 0x71f0d5f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e429: skipped 000205 1772176400.488116 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e429): Mirror 3: 0x71f0d5f5 (stale) Mirror 6: 0x71f0d5f5 (stale) Mirror 7: 0x71f0d5f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e429): Mirror 3: 0x71f0d5f5 (stale) Mirror 6: 0x71f0d5f5 (stale) Mirror 7: 0x71f0d5f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e429: skipped 000206 1772176400.988712 mapread 0x00e66d thru 0x00ebe4 (0x578 bytes) 000207 1772176401.013515 write 0x02c7f0 thru 0x038c9c (0xc4ad bytes) 000208 1772176401.015707 write 0x032e90 thru 0x03ffff (0xd170 bytes) truncating to largest ever: 0x363a8 000210 1772176401.020017 trunc from 0x040000 to 0x0363a7 (0x9c58 bytes) 000211 1772176401.073926 mapread 0x000e45 thru 0x00ac42 (0x9dfe bytes) 000212 1772176401.098100 read 0x035521 thru 0x0363a7 (0xe87 bytes) 000216 1772176401.107019 trunc from 0x0363a8 to 0x029f9c (0xc40b bytes) 000217 1772176401.129047 read 0x028b5b thru 0x029f9c (0x1442 bytes) 000219 1772176401.142562 mapread 0x01ac63 thru 0x01d634 (0x29d2 bytes) 000220 1772176401.143676 mapwrite 0x0088d5 thru 0x010485 (0x7bb1 bytes) 000221 1772176401.210460 mapread 0x01b6ed thru 0x0254e4 (0x9df8 bytes) 000222 1772176401.219932 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29f9d): Mirror 3: 0x81cf06b7 (stale) Mirror 7: 0x81cf06b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f9d): Mirror 3: 0x81cf06b7 (stale) Mirror 7: 0x81cf06b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f9d: skipped 000223 1772176401.525571 mapread 0x003dd0 thru 0x00ea0b (0xac3c bytes) 000224 1772176401.549156 mapread 0x01927e thru 0x01ac91 (0x1a14 bytes) 000225 1772176401.561241 trunc from 0x029f9d to 0x020e66 (0x9136 bytes) 000226 1772176401.578483 write 0x03afa7 thru 0x03fc5b (0x4cb5 bytes) 000228 1772176401.582200 mapwrite 0x004401 thru 0x0086e7 (0x42e7 bytes) 000232 1772176401.664028 write 0x0212ab thru 0x030ca7 (0xf9fd bytes) 000234 1772176401.724095 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc5c): Mirror 3: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc5c): Mirror 3: 0xd7faafc3 (stale) Mirror 7: 0xd7faafc3 (stale) CRC-32 checksum value for chunk [0, 0x3fc5c): Mirror 1: 0x6d561af2 Mirror 8: 0x6d561af2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc5c: skipped 000235 1772176402.471895 write 0x00c9a0 thru 0x00dcee (0x134f bytes) 000238 1772176402.565303 mapwrite 0x009ea2 thru 0x014725 (0xa884 bytes) 000239 1772176402.646124 read 0x0279a7 thru 0x02e354 (0x69ae bytes) 000242 1772176402.658188 trunc from 0x03fc5c to 0x009036 (0x36c25 bytes) 000244 1772176402.690043 MIRROR_RESYNC 4 mirrors 000245 1772176403.520355 write 0x01d865 thru 0x028345 (0xaae1 bytes) 000246 1772176403.598000 read 0x00e715 thru 0x01983b (0xb127 bytes) 000248 1772176403.609150 read 0x0045c5 thru 0x00ea48 (0xa484 bytes) 000250 1772176403.623577 read 0x021856 thru 0x028345 (0x6af0 bytes) 000252 1772176403.625489 mapread 0x008c3b thru 0x017607 (0xe9cd bytes) 000253 1772176403.627533 write 0x021928 thru 0x02f2d8 (0xd9b1 bytes) truncating to largest ever: 0x3a962 000254 1772176403.634633 trunc from 0x02f2d9 to 0x03a961 (0xb689 bytes) 000255 1772176403.657180 mapread 0x027030 thru 0x033484 (0xc455 bytes) 000256 1772176403.671457 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a962): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a962): Mirror 3: 0x7ff71737 (stale) Mirror 7: 0x7ff71737 (stale) Mirror 8: 0x7ff71737 (stale) CRC-32 checksum value for chunk [0, 0x3a962): Mirror 1: 0x7e69d70b Mirror 9: 0x7e69d70b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a962: skipped 000257 1772176404.753159 mapread 0x00e162 thru 0x00f4b7 (0x1356 bytes) 000259 1772176404.826571 mapwrite 0x00f458 thru 0x01d965 (0xe50e bytes) 000260 1772176405.180306 mapwrite 0x01d232 thru 0x029161 (0xbf30 bytes) 000261 1772176405.271491 trunc from 0x03a962 to 0x02bf39 (0xea28 bytes) 000262 1772176405.302378 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf3a): Mirror 3: 0x5ff80571 (stale) Mirror 8: 0x5ff80571 (stale) Mirror 9: 0xc02c1ef5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf3a): Mirror 3: 0x5ff80571 (stale) Mirror 8: 0x5ff80571 (stale) Mirror 9: 0xc02c1ef5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf3a: skipped 000264 1772176405.660771 read 0x01c26c thru 0x02328c (0x7021 bytes) 000266 1772176405.676596 write 0x026c10 thru 0x02f655 (0x8a46 bytes) 000269 1772176405.686151 trunc from 0x02f656 to 0x00446f (0x2b1e6 bytes) truncating to largest ever: 0x3ace2 000270 1772176405.710133 trunc from 0x004470 to 0x03ace1 (0x36872 bytes) 000273 1772176405.735105 trunc from 0x03ace2 to 0x01a242 (0x20a9f bytes) 000274 1772176405.756448 write 0x014fe0 thru 0x019899 (0x48ba bytes) 000275 1772176405.776375 mapwrite 0x02c47a thru 0x034cd6 (0x885d bytes) 000276 1772176405.883964 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34cd7): Mirror 8: 0x797f68a (stale) Mirror 9: 0x9273301f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34cd7): Mirror 8: 0x797f68a (stale) Mirror 9: 0x9273301f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34cd7: skipped 000278 1772176406.257525 trunc from 0x034cd7 to 0x033107 (0x1bcf bytes) 000279 1772176406.291384 read 0x009087 thru 0x013ebc (0xae36 bytes) 000280 1772176406.299638 mapwrite 0x02b132 thru 0x02c3f5 (0x12c4 bytes) 000281 1772176406.357717 MIRROR_RESYNC 3 mirrors 000283 1772176407.048503 write 0x007e3e thru 0x0152c4 (0xd487 bytes) 000285 1772176407.123384 mapwrite 0x00566f thru 0x00c430 (0x6dc2 bytes) 000286 1772176407.196364 write 0x00f6fe thru 0x018793 (0x9096 bytes) 000288 1772176407.205596 write 0x035162 thru 0x03c608 (0x74a7 bytes) 000289 1772176407.206920 trunc from 0x03c609 to 0x01f17e (0x1d48a bytes) 000291 1772176407.258909 trunc from 0x01f17f to 0x00e109 (0x11075 bytes) 000292 1772176407.285619 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe10a): Mirror 9: 0x2352642f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe10a): Mirror 9: 0x2352642f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe10a: skipped 000293 1772176407.683242 mapwrite 0x01fdf8 thru 0x02ac0d (0xae16 bytes) 000295 1772176407.847724 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) 000297 1772176408.047021 mapread 0x0095ac thru 0x015cd3 (0xc728 bytes) 000299 1772176408.086558 trunc from 0x02ac0e to 0x031e22 (0x7215 bytes) 000302 1772176408.111998 read 0x021fbf thru 0x02b684 (0x96c6 bytes) 000305 1772176408.127551 write 0x03629f thru 0x036e47 (0xba9 bytes) 000306 1772176408.128993 mapwrite 0x00f996 thru 0x01da0e (0xe079 bytes) 000309 1772176408.192038 trunc from 0x036e48 to 0x02b389 (0xbabe bytes) 000310 1772176408.203522 trunc from 0x02b38a to 0x012fdf (0x183aa bytes) 000311 1772176408.221128 write 0x03eefb thru 0x03ffff (0x1105 bytes) 000312 1772176408.222327 trunc from 0x040000 to 0x0187a7 (0x27858 bytes) 000317 1772176408.248417 write 0x010e82 thru 0x01773d (0x68bc bytes) 000318 1772176408.254514 mapwrite 0x02e30c thru 0x039e4d (0xbb42 bytes) 000319 1772176408.355568 trunc from 0x039e4e to 0x02b11d (0xed30 bytes) 000321 1772176408.379990 mapread 0x019717 thru 0x023320 (0x9c0a bytes) 000322 1772176408.401336 read 0x01039c thru 0x018383 (0x7fe8 bytes) 000323 1772176408.402035 mapwrite 0x02e7ab thru 0x03a966 (0xc1bc bytes) 000324 1772176408.520087 read 0x01c4b8 thru 0x02107b (0x4bc4 bytes) 000325 1772176408.521555 trunc from 0x03a967 to 0x009bb2 (0x30db4 bytes) 000326 1772176408.554907 mapread 0x00487e thru 0x0056ed (0xe70 bytes) 000329 1772176408.560510 write 0x009522 thru 0x00cacc (0x35ab bytes) 000330 1772176408.563597 trunc from 0x00cacd to 0x0386a0 (0x2bbd4 bytes) 000331 1772176408.590498 write 0x006d19 thru 0x00c5f2 (0x58da bytes) 000332 1772176408.592135 write 0x0210b4 thru 0x023511 (0x245e bytes) 000336 1772176408.612578 mapwrite 0x012061 thru 0x012fc5 (0xf65 bytes) 000337 1772176408.669491 write 0x01e260 thru 0x02d4ae (0xf24f bytes) 000340 1772176408.680335 MIRROR_EXTEND to 2 mirrors 000341 1772176409.117547 mapread 0x031ccd thru 0x033d93 (0x20c7 bytes) 000342 1772176409.141060 MIRROR_RESYNC 2 mirrors 000343 1772176409.521132 mapwrite 0x036c05 thru 0x03ffff (0x93fb bytes) 000345 1772176409.696291 mapwrite 0x035fc6 thru 0x03f6dd (0x9718 bytes) 000347 1772176409.752840 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) 000348 1772176409.896832 write 0x0053a5 thru 0x00c05c (0x6cb8 bytes) 000350 1772176409.912426 read 0x026305 thru 0x03157e (0xb27a bytes) 000353 1772176409.922488 trunc from 0x040000 to 0x036d6e (0x9291 bytes) 000358 1772176409.955825 write 0x0132b7 thru 0x016fb7 (0x3d01 bytes) 000359 1772176409.972735 MIRROR_EXTEND to 2 mirrors 000360 1772176410.592179 write 0x0287fa thru 0x02a02e (0x1835 bytes) 000361 1772176410.688422 mapwrite 0x02f42c thru 0x0317bc (0x2391 bytes) 000363 1772176410.763400 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 2: 0xf7d8e0d5 (stale) CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 1: 0xfd6e9f1a Mirror 3: 0xfd6e9f1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d6f: skipped 000364 1772176411.567501 mapread 0x02d734 thru 0x035f42 (0x880f bytes) 000366 1772176411.622643 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 2: 0xf7d8e0d5 (stale) CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 1: 0xfd6e9f1a Mirror 3: 0xfd6e9f1a Mirror 4: 0xfd6e9f1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d6f: skipped 000367 1772176412.691863 trunc from 0x036d6f to 0x0388e0 (0x1b72 bytes) 000372 1772176412.808199 write 0x00071f thru 0x00adbf (0xa6a1 bytes) 000373 1772176412.831205 trunc from 0x0388e1 to 0x02f279 (0x9667 bytes) 000374 1772176412.863557 write 0x0275af thru 0x033eb0 (0xc902 bytes) 000375 1772176412.879105 mapwrite 0x0287f4 thru 0x02e41a (0x5c27 bytes) 000376 1772176412.976182 mapwrite 0x02a739 thru 0x0328ef (0x81b7 bytes) 000377 1772176413.030924 write 0x031446 thru 0x03e08f (0xcc4a bytes) 000378 1772176413.040675 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e090): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e090): Mirror 2: 0x21dfd94f (stale) Mirror 3: 0x469657b9 (stale) Mirror 4: 0x469657b9 (stale) CRC-32 checksum value for chunk [0, 0x3e090): Mirror 1: 0x255ddf2c Mirror 5: 0x255ddf2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e090: skipped 000380 1772176413.981693 write 0x03d3db thru 0x03ffff (0x2c25 bytes) 000381 1772176414.078602 trunc from 0x040000 to 0x02cb57 (0x134a8 bytes) 000382 1772176414.125577 mapread 0x009b11 thru 0x014c6f (0xb15f bytes) 000386 1772176414.182034 trunc from 0x02cb58 to 0x005f95 (0x26bc2 bytes) 000387 1772176414.213591 mapread 0x0051bc thru 0x005f95 (0xdda bytes) 000388 1772176414.216854 write 0x0133b5 thru 0x01e470 (0xb0bc bytes) 000390 1772176414.224254 mapwrite 0x00b112 thru 0x00de8b (0x2d7a bytes) 000391 1772176414.334153 read 0x00cb65 thru 0x010eee (0x438a bytes) 000392 1772176414.348531 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e471): Mirror 3: 0x3a4be5e4 (stale) Mirror 4: 0x3a4be5e4 (stale) Mirror 5: 0xc682ffc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e471): Mirror 3: 0x3a4be5e4 (stale) Mirror 4: 0x3a4be5e4 (stale) Mirror 5: 0xc682ffc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e471: skipped 000393 1772176414.860858 trunc from 0x01e471 to 0x01b94f (0x2b21 bytes) 000395 1772176414.887854 read 0x01949e thru 0x01acf5 (0x1858 bytes) 000396 1772176414.896778 mapread 0x01033b thru 0x011f0f (0x1bd5 bytes) 000397 1772176414.909755 trunc from 0x01b950 to 0x002c48 (0x18d07 bytes) 000399 1772176414.934752 read 0x000e16 thru 0x002c48 (0x1e33 bytes) 000400 1772176414.944873 mapread 0x000053 thru 0x002c48 (0x2bf6 bytes) 000401 1772176414.953417 MIRROR_RESYNC 4 mirrors 000403 1772176416.088416 write 0x03832e thru 0x03ffff (0x7cd2 bytes) 000405 1772176416.243198 write 0x0351b1 thru 0x03a1ee (0x503e bytes) 000406 1772176416.255316 write 0x01fe08 thru 0x024a96 (0x4c8f bytes) 000407 1772176416.292097 MIRROR_RESYNC 4 mirrors 000409 1772176417.801652 trunc from 0x040000 to 0x0356da (0xa925 bytes) 000410 1772176417.901830 read 0x02bf05 thru 0x02c9c5 (0xac1 bytes) 000414 1772176417.911829 read 0x0231b9 thru 0x02fedd (0xcd25 bytes) 000415 1772176417.925495 mapread 0x00ebb7 thru 0x0174b1 (0x88fb bytes) 000417 1772176417.928493 mapread 0x0216ac thru 0x021f10 (0x865 bytes) 000418 1772176417.931150 write 0x03f5b3 thru 0x03ffff (0xa4d bytes) 000421 1772176417.936338 write 0x029628 thru 0x03292b (0x9304 bytes) 000422 1772176417.939397 mapread 0x009506 thru 0x016110 (0xcc0b bytes) 000423 1772176417.953410 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0c4e93 (stale) Mirror 5: 0xb0c4e93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0c4e93 (stale) Mirror 5: 0xb0c4e93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000424 1772176418.703685 read 0x00a2e8 thru 0x018566 (0xe27f bytes) 000425 1772176418.711621 trunc from 0x040000 to 0x00533a (0x3acc5 bytes) 000426 1772176418.726717 write 0x000473 thru 0x00b476 (0xb004 bytes) 000429 1772176418.735687 write 0x032370 thru 0x036ad8 (0x4769 bytes) 000430 1772176418.738965 mapwrite 0x013810 thru 0x015aa0 (0x2291 bytes) 000431 1772176418.818120 trunc from 0x036ad9 to 0x03a86e (0x3d96 bytes) 000433 1772176418.861281 read 0x01eec8 thru 0x0216fa (0x2833 bytes) 000437 1772176418.880746 write 0x0028c8 thru 0x00e326 (0xba5f bytes) 000438 1772176418.889165 trunc from 0x03a86f to 0x02e880 (0xbfee bytes) 000439 1772176418.927698 mapwrite 0x003b09 thru 0x011712 (0xdc0a bytes) 000440 1772176419.032924 MIRROR_RESYNC 3 mirrors 000441 1772176420.117566 read 0x019176 thru 0x022f8f (0x9e1a bytes) 000443 1772176420.133771 mapread 0x00f264 thru 0x01e10f (0xeeac bytes) 000444 1772176420.168726 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) 000445 1772176420.657706 read 0x0234ce thru 0x024901 (0x1434 bytes) 000447 1772176420.670794 read 0x005fc8 thru 0x011d85 (0xbdbe bytes) 000448 1772176420.700255 MIRROR_RESYNC 2 mirrors 000449 1772176421.185863 write 0x0187e2 thru 0x023e87 (0xb6a6 bytes) 000450 1772176421.316994 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e881): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e881): Mirror 3: 0xe1bd1e07 (stale) CRC-32 checksum value for chunk [0, 0x2e881): Mirror 1: 0x1da82445 Mirror 4: 0x1da82445 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e881: skipped 000453 1772176422.462966 write 0x02db4f thru 0x03386d (0x5d1f bytes) 000455 1772176422.557475 mapwrite 0x037a83 thru 0x03f221 (0x779f bytes) 000456 1772176422.687135 write 0x00a2d5 thru 0x0125a3 (0x82cf bytes) 000458 1772176422.702793 write 0x03923c thru 0x03e171 (0x4f36 bytes) 000459 1772176422.704711 trunc from 0x03f222 to 0x02edc3 (0x1045e bytes) 000461 1772176422.727424 mapread 0x01e0f9 thru 0x022c41 (0x4b49 bytes) 000464 1772176422.745735 mapwrite 0x014154 thru 0x0177ab (0x3658 bytes) 000465 1772176422.835081 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2edc4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edc4): Mirror 3: 0xc0bf4619 (stale) Mirror 4: 0x2e2ff173 (stale) CRC-32 checksum value for chunk [0, 0x2edc4): Mirror 1: 0x152ca139 Mirror 5: 0x152ca139 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edc4: skipped 000467 1772176423.664258 read 0x00abdc thru 0x0102e6 (0x570b bytes) 000468 1772176423.684656 MIRROR_RESYNC 4 mirrors 000469 1772176424.491629 write 0x00e04e thru 0x012896 (0x4849 bytes) 000471 1772176424.578962 mapwrite 0x006235 thru 0x00f014 (0x8de0 bytes) 000472 1772176424.647695 trunc from 0x02edc4 to 0x010f44 (0x1de7f bytes) 000473 1772176424.668422 trunc from 0x010f45 to 0x021193 (0x1024f bytes) 000474 1772176424.687086 write 0x03dba3 thru 0x03ffff (0x245d bytes) 000476 1772176424.690649 read 0x022299 thru 0x02b2ae (0x9016 bytes) 000477 1772176424.698199 read 0x0059fe thru 0x00ee18 (0x941b bytes) 000479 1772176424.699465 write 0x031e98 thru 0x037e77 (0x5fe0 bytes) 000480 1772176424.711060 mapwrite 0x039855 thru 0x03ffff (0x67ab bytes) 000481 1772176424.755459 write 0x022ecf thru 0x02b8fa (0x8a2c bytes) 000483 1772176424.757420 mapread 0x0235e5 thru 0x02874f (0x516b bytes) 000484 1772176424.758929 mapwrite 0x004f16 thru 0x00b16a (0x6255 bytes) 000486 1772176424.839273 mapread 0x034807 thru 0x035cf9 (0x14f3 bytes) 000488 1772176424.846820 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3d478d85 (stale) Mirror 4: 0x3d478d85 (stale) Mirror 5: 0x3d478d85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x47c35e4b Mirror 6: 0x47c35e4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1772176425.725734 read 0x003752 thru 0x010779 (0xd028 bytes) 000492 1772176425.752100 write 0x03a3e7 thru 0x03ffff (0x5c19 bytes) 000493 1772176425.829366 trunc from 0x040000 to 0x00e086 (0x31f79 bytes) 000494 1772176425.860542 read 0x004192 thru 0x00e086 (0x9ef5 bytes) 000496 1772176425.876936 mapwrite 0x0193ab thru 0x02012b (0x6d81 bytes) 000497 1772176425.959805 read 0x00c1bd thru 0x014377 (0x81bb bytes) 000499 1772176425.961964 trunc from 0x02012c to 0x033971 (0x13846 bytes) 000500 1772176425.985648 read 0x007c51 thru 0x012637 (0xa9e7 bytes) 000501 1772176425.986754 mapread 0x029121 thru 0x031783 (0x8663 bytes) 000502 1772176426.003391 write 0x019c12 thru 0x020729 (0x6b18 bytes) 000503 1772176426.006751 mapwrite 0x002ca6 thru 0x00d12f (0xa48a bytes) 000505 1772176426.056561 trunc from 0x033972 to 0x00c7d1 (0x271a0 bytes) 000507 1772176426.076166 mapwrite 0x03e93f thru 0x03ffff (0x16c1 bytes) 000508 1772176426.152150 MIRROR_RESYNC 5 mirrors 000509 1772176427.205545 write 0x00a85b thru 0x019500 (0xeca6 bytes) 000512 1772176427.315440 read 0x030e83 thru 0x03b316 (0xa494 bytes) 000515 1772176427.323737 write 0x038a80 thru 0x03b3ec (0x296d bytes) 000516 1772176427.343349 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x42fdd0b (stale) Mirror 4: 0x42fdd0b (stale) Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe83befdd Mirror 7: 0xe83befdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000517 1772176428.178980 mapwrite 0x02d760 thru 0x03607a (0x891b bytes) 000521 1772176428.363661 mapwrite 0x033452 thru 0x038333 (0x4ee2 bytes) 000522 1772176428.404875 write 0x0314d5 thru 0x03640e (0x4f3a bytes) 000523 1772176428.406206 mapread 0x02ac2e thru 0x02ff4a (0x531d bytes) 000525 1772176428.407479 write 0x02cedc thru 0x038ab9 (0xbbde bytes) 000527 1772176428.408738 write 0x00d398 thru 0x01484a (0x74b3 bytes) 000530 1772176428.422779 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x42fdd0b (stale) Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) Mirror 7: 0xe83befdd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x42fdd0b (stale) Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) Mirror 7: 0xe83befdd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1772176428.819890 write 0x00a7dc thru 0x01259e (0x7dc3 bytes) 000535 1772176428.840656 write 0x02377c thru 0x02ac02 (0x7487 bytes) 000537 1772176428.858569 write 0x00fc47 thru 0x017a5e (0x7e18 bytes) 000538 1772176428.864145 mapwrite 0x03b439 thru 0x03ffff (0x4bc7 bytes) 000542 1772176428.924179 mapwrite 0x020294 thru 0x02b4c5 (0xb232 bytes) 000543 1772176428.977421 read 0x001fb8 thru 0x00ab2c (0x8b75 bytes) 000544 1772176428.988577 trunc from 0x040000 to 0x02d491 (0x12b6e bytes) 000547 1772176429.036593 write 0x00d142 thru 0x0135fd (0x64bc bytes) 000548 1772176429.045407 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d492): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d492): Mirror 3: 0x709f18b9 (stale) Mirror 5: 0x709f18b9 (stale) Mirror 6: 0x709f18b9 (stale) Mirror 7: 0xedd62595 (stale) CRC-32 checksum value for chunk [0, 0x2d492): Mirror 1: 0x8ca24996 Mirror 8: 0x8ca24996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d492: skipped 000549 1772176429.907722 trunc from 0x02d492 to 0x020b32 (0xc95f bytes) 000550 1772176430.006874 mapread 0x01234d thru 0x0162e3 (0x3f97 bytes) 000551 1772176430.032873 mapread 0x020145 thru 0x020b32 (0x9ee bytes) truncating to largest ever: 0x3c079 000554 1772176430.042658 trunc from 0x020b33 to 0x03c078 (0x1b546 bytes) 000555 1772176430.062690 write 0x03d059 thru 0x03fb8b (0x2b33 bytes) 000556 1772176430.063879 write 0x03ad20 thru 0x03ffff (0x52e0 bytes) 000561 1772176430.072408 read 0x008516 thru 0x01261c (0xa107 bytes) 000564 1772176430.081088 read 0x01dc1e thru 0x027a4d (0x9e30 bytes) 000567 1772176430.098393 mapwrite 0x007889 thru 0x01286d (0xafe5 bytes) 000571 1772176430.168047 write 0x015d6c thru 0x0257a9 (0xfa3e bytes) 000573 1772176430.170273 mapwrite 0x0096ae thru 0x0117c3 (0x8116 bytes) 000574 1772176430.217432 mapread 0x00faee thru 0x019977 (0x9e8a bytes) 000575 1772176430.218931 mapwrite 0x0133db thru 0x01ffee (0xcc14 bytes) 000580 1772176430.254194 write 0x00a49c thru 0x012436 (0x7f9b bytes) 000581 1772176430.256992 mapread 0x034a89 thru 0x038723 (0x3c9b bytes) 000583 1772176430.259631 trunc from 0x040000 to 0x00c58c (0x33a73 bytes) 000584 1772176430.280396 write 0x035fb0 thru 0x03ffff (0xa050 bytes) 000585 1772176430.282342 trunc from 0x040000 to 0x01beb8 (0x24147 bytes) 000586 1772176430.300261 mapwrite 0x03f59b thru 0x03ffff (0xa65 bytes) 000588 1772176430.390826 read 0x0348b6 thru 0x03acb8 (0x6403 bytes) 000589 1772176430.405562 read 0x00dfde thru 0x016c57 (0x8c7a bytes) 000591 1772176430.412297 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x42fdd0b (stale) Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) Mirror 7: 0xe83befdd (stale) Mirror 8: 0xb0a4b0ab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18eb2bc2 Mirror 9: 0x18eb2bc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000592 1772176431.195634 read 0x01d667 thru 0x01ecb3 (0x164d bytes) 000594 1772176431.211659 mapwrite 0x02d96d thru 0x03a88d (0xcf21 bytes) 000597 1772176431.450548 mapread 0x019f69 thru 0x01a7be (0x856 bytes) 000598 1772176431.499817 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) Mirror 7: 0xe83befdd (stale) Mirror 8: 0xb0a4b0ab (stale) Mirror 9: 0x18eb2bc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x42fdd0b (stale) Mirror 6: 0x42fdd0b (stale) Mirror 7: 0xe83befdd (stale) Mirror 8: 0xb0a4b0ab (stale) Mirror 9: 0x18eb2bc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000599 1772176432.107136 write 0x01f926 thru 0x024ad1 (0x51ac bytes) 000600 1772176432.120807 mapread 0x03dc78 thru 0x03ffad (0x2336 bytes) 000602 1772176432.130976 read 0x027c22 thru 0x02adfc (0x31db bytes) 000603 1772176432.136442 mapwrite 0x033216 thru 0x03c8e7 (0x96d2 bytes) 000605 1772176432.198503 write 0x03151d thru 0x039c22 (0x8706 bytes) 000612 1772176432.206615 trunc from 0x040000 to 0x030cf3 (0xf30c bytes) 000614 1772176432.249459 trunc from 0x030cf4 to 0x033b72 (0x2e7f bytes) 000615 1772176432.263611 mapread 0x003107 thru 0x00930d (0x6207 bytes) 000617 1772176432.276946 write 0x00b3ef thru 0x0177ca (0xc3dc bytes) 000618 1772176432.283901 read 0x027760 thru 0x02f144 (0x79e5 bytes) 000619 1772176432.293990 mapwrite 0x023006 thru 0x024242 (0x123d bytes) 000623 1772176432.338814 MIRROR_RESYNC 6 mirrors 000625 1772176433.523584 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) 000627 1772176433.873157 mapwrite 0x03cb3f thru 0x03ffff (0x34c1 bytes) 000628 1772176434.023932 write 0x01ea1b thru 0x02e088 (0xf66e bytes) 000629 1772176434.040541 trunc from 0x040000 to 0x0345a9 (0xba56 bytes) 000630 1772176434.074087 write 0x00e684 thru 0x01bfec (0xd969 bytes) 000632 1772176434.092044 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x345aa): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345aa): Mirror 6: 0x52d20752 (stale) Mirror 7: 0x52d20752 (stale) Mirror 8: 0x52d20752 (stale) Mirror 9: 0x52d20752 (stale) CRC-32 checksum value for chunk [0, 0x345aa): Mirror 1: 0x1e222997 Mirror 10: 0x1e222997 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345aa: skipped 000634 1772176435.092378 MIRROR_RESYNC 6 mirrors 000636 1772176436.044151 read 0x0271ed thru 0x027d21 (0xb35 bytes) 000638 1772176436.051566 write 0x00c584 thru 0x0147e4 (0x8261 bytes) 000639 1772176436.148375 mapread 0x00b6de thru 0x019cbe (0xe5e1 bytes) 000640 1772176436.165187 write 0x019426 thru 0x01fe62 (0x6a3d bytes) 000642 1772176436.182911 read 0x0133ea thru 0x021890 (0xe4a7 bytes) 000643 1772176436.206537 mapwrite 0x021472 thru 0x02cf5b (0xbaea bytes) 000644 1772176436.309266 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x345aa): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345aa): Mirror 6: 0x1e222997 (stale) Mirror 7: 0x1e222997 (stale) Mirror 8: 0x1e222997 (stale) Mirror 9: 0x1e222997 (stale) Mirror 10: 0x1e222997 (stale) CRC-32 checksum value for chunk [0, 0x345aa): Mirror 1: 0x15c40421 Mirror 11: 0x15c40421 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345aa: skipped 000647 1772176437.319899 read 0x00f168 thru 0x012847 (0x36e0 bytes) 000649 1772176437.334471 mapread 0x02040c thru 0x02d4f1 (0xd0e6 bytes) 000650 1772176437.366462 write 0x037576 thru 0x03e164 (0x6bef bytes) 000651 1772176437.443065 trunc from 0x03e165 to 0x030c15 (0xd54f bytes) 000653 1772176437.487788 write 0x019780 thru 0x01b69f (0x1f20 bytes) 000656 1772176437.501083 write 0x02c1f8 thru 0x0371f7 (0xb000 bytes) 000658 1772176437.514182 read 0x010940 thru 0x012a4d (0x210e bytes) 000659 1772176437.529689 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x371f8): Mirror 6: 0xdfa21018 (stale) Mirror 8: 0xdfa21018 (stale) Mirror 9: 0xdfa21018 (stale) Mirror 10: 0xdfa21018 (stale) Mirror 11: 0x91c2a808 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x371f8): Mirror 6: 0xdfa21018 (stale) Mirror 8: 0xdfa21018 (stale) Mirror 9: 0xdfa21018 (stale) Mirror 10: 0xdfa21018 (stale) Mirror 11: 0x91c2a808 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x371f8: skipped 000661 1772176437.951293 write 0x01a6d4 thru 0x027a93 (0xd3c0 bytes) 000662 1772176437.969137 MIRROR_RESYNC 6 mirrors 000663 1772176439.238138 write 0x026940 thru 0x02d73d (0x6dfe bytes) 000668 1772176439.363009 read 0x01a1d0 thru 0x0292fb (0xf12c bytes) 000670 1772176439.386411 trunc from 0x0371f8 to 0x034cbd (0x253a bytes) 000671 1772176439.417947 trunc from 0x034cbe to 0x001b48 (0x33175 bytes) 000672 1772176439.442884 mapwrite 0x01a536 thru 0x01d5f8 (0x30c3 bytes) 000673 1772176439.584359 read 0x01c9ac thru 0x01d5f8 (0xc4d bytes) 000674 1772176439.585005 mapwrite 0x038b82 thru 0x03ff93 (0x7412 bytes) 000675 1772176439.762882 read 0x01931d thru 0x020ec2 (0x7ba6 bytes) 000679 1772176439.768504 write 0x026f77 thru 0x03510c (0xe196 bytes) 000681 1772176439.774918 mapwrite 0x005a94 thru 0x009e62 (0x43cf bytes) 000682 1772176439.841575 read 0x02c94f thru 0x038293 (0xb945 bytes) 000683 1772176439.842392 write 0x01ded0 thru 0x01ff8b (0x20bc bytes) 000685 1772176439.853081 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff94): Mirror 6: 0x2dcb1b34 (stale) Mirror 8: 0x2dcb1b34 (stale) Mirror 9: 0x2dcb1b34 (stale) Mirror 11: 0x2dcb1b34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff94): Mirror 6: 0x2dcb1b34 (stale) Mirror 8: 0x2dcb1b34 (stale) Mirror 9: 0x2dcb1b34 (stale) Mirror 11: 0x2dcb1b34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff94: skipped 000686 1772176440.333464 trunc from 0x03ff94 to 0x01d615 (0x2297e bytes) 000687 1772176440.369427 mapwrite 0x000470 thru 0x00d9ef (0xd580 bytes) 000688 1772176440.464290 mapread 0x000d72 thru 0x010283 (0xf512 bytes) 000689 1772176440.467124 trunc from 0x01d616 to 0x03481e (0x17209 bytes) 000690 1772176440.487976 trunc from 0x03481f to 0x01a876 (0x19fa8 bytes) 000691 1772176440.511368 write 0x038472 thru 0x038f44 (0xad3 bytes) 000695 1772176440.517324 read 0x01483d thru 0x01cc10 (0x83d4 bytes) 000696 1772176440.533936 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38f45): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f45): Mirror 6: 0x6e35e3a6 (stale) Mirror 8: 0x6e35e3a6 (stale) Mirror 9: 0x6e35e3a6 (stale) Mirror 11: 0x6e35e3a6 (stale) CRC-32 checksum value for chunk [0, 0x38f45): Mirror 1: 0x6d2e08b9 Mirror 12: 0x6d2e08b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f45: skipped 000697 1772176441.407918 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38f45): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f45): Mirror 6: 0x6e35e3a6 (stale) Mirror 8: 0x6e35e3a6 (stale) Mirror 9: 0x6e35e3a6 (stale) Mirror 11: 0x6e35e3a6 (stale) CRC-32 checksum value for chunk [0, 0x38f45): Mirror 1: 0x6d2e08b9 Mirror 12: 0x6d2e08b9 Mirror 13: 0x6d2e08b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f45: skipped 000699 1772176442.258529 read 0x0307c7 thru 0x038f44 (0x877e bytes) 000701 1772176442.281856 mapread 0x033138 thru 0x038f44 (0x5e0d bytes) 000703 1772176442.284795 write 0x0130bc thru 0x01b495 (0x83da bytes) 000704 1772176442.390647 mapwrite 0x006335 thru 0x015e47 (0xfb13 bytes) 000705 1772176442.499867 write 0x00c5b5 thru 0x01734e (0xad9a bytes) 000706 1772176442.529813 MIRROR_RESYNC 7 mirrors 000709 1772176443.627933 mapwrite 0x018867 thru 0x023410 (0xabaa bytes) 000713 1772176443.838498 trunc from 0x038f45 to 0x01c2ba (0x1cc8a bytes) 000714 1772176443.868287 mapread 0x012410 thru 0x01832b (0x5f1c bytes) 000716 1772176443.869863 mapread 0x001c00 thru 0x00d505 (0xb906 bytes) 000717 1772176443.880853 mapwrite 0x031f00 thru 0x037570 (0x5671 bytes) 000718 1772176443.980078 MIRROR_RESYNC 7 mirrors 000719 1772176445.335277 mapread 0x03002d thru 0x037570 (0x7544 bytes) 000722 1772176445.354158 read 0x02aef1 thru 0x031462 (0x6572 bytes) 000724 1772176445.362734 read 0x01294f thru 0x01fd27 (0xd3d9 bytes) 000726 1772176445.374299 trunc from 0x037571 to 0x039cda (0x276a bytes) 000727 1772176445.466048 read 0x01e66a thru 0x0240f6 (0x5a8d bytes) 000729 1772176445.487620 trunc from 0x039cdb to 0x02a8b6 (0xf424 bytes) 000730 1772176445.523433 mapread 0x020a81 thru 0x02a8b6 (0x9e36 bytes) 000732 1772176445.534801 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2a8b7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8b7): Mirror 6: 0x479aed6 (stale) Mirror 8: 0x479aed6 (stale) Mirror 9: 0x479aed6 (stale) Mirror 11: 0x479aed6 (stale) Mirror 12: 0x479aed6 (stale) Mirror 13: 0x479aed6 (stale) CRC-32 checksum value for chunk [0, 0x2a8b7): Mirror 1: 0x479aed6 Mirror 14: 0x479aed6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8b7: skipped truncating to largest ever: 0x3c14e 000733 1772176446.506128 trunc from 0x02a8b7 to 0x03c14d (0x11897 bytes) 000734 1772176446.706271 trunc from 0x03c14e to 0x0382bf (0x3e8e bytes) 000735 1772176446.773331 write 0x026b47 thru 0x02ac20 (0x40da bytes) 000736 1772176446.791100 read 0x031029 thru 0x0382bf (0x7297 bytes) 000737 1772176446.819364 mapread 0x00bc21 thru 0x019d11 (0xe0f1 bytes) 000738 1772176446.848406 read 0x027385 thru 0x032df4 (0xba70 bytes) 000739 1772176446.865456 write 0x0358b2 thru 0x037d61 (0x24b0 bytes) 000740 1772176446.872369 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x382c0): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382c0): Mirror 6: 0xc921fb1b (stale) Mirror 8: 0xc921fb1b (stale) Mirror 9: 0xc921fb1b (stale) Mirror 11: 0xc921fb1b (stale) Mirror 12: 0xc921fb1b (stale) Mirror 13: 0xc921fb1b (stale) Mirror 14: 0x609ffba8 (stale) CRC-32 checksum value for chunk [0, 0x382c0): Mirror 1: 0x27fb816c Mirror 15: 0x27fb816c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382c0: skipped 000744 1772176447.898027 mapwrite 0x01d437 thru 0x0221d8 (0x4da2 bytes) 000746 1772176448.098666 write 0x011602 thru 0x019260 (0x7c5f bytes) 000748 1772176448.112989 mapread 0x033186 thru 0x033265 (0xe0 bytes) 000751 1772176448.122766 read 0x02fef6 thru 0x031072 (0x117d bytes) 000752 1772176448.123496 mapwrite 0x001557 thru 0x0066e0 (0x518a bytes) 000753 1772176448.162442 mapread 0x00c9cd thru 0x00d9cf (0x1003 bytes) 000754 1772176448.163486 write 0x0053c3 thru 0x012b85 (0xd7c3 bytes) 000756 1772176448.167150 write 0x03a896 thru 0x03ffff (0x576a bytes) 000757 1772176448.169061 trunc from 0x040000 to 0x02529d (0x1ad62 bytes) 000758 1772176448.192751 mapread 0x0159c4 thru 0x025135 (0xf772 bytes) 000759 1772176448.194190 mapwrite 0x001eba thru 0x00547e (0x35c5 bytes) 000765 1772176448.245154 read 0x017e20 thru 0x018f6b (0x114c bytes) 000767 1772176448.247333 mapwrite 0x00f784 thru 0x010c7a (0x14f7 bytes) 000769 1772176448.284555 write 0x03f334 thru 0x03ffff (0xccc bytes) 000771 1772176448.286279 write 0x01ecba thru 0x02d1c9 (0xe510 bytes) 000772 1772176448.296310 write 0x01f89f thru 0x022472 (0x2bd4 bytes) 000773 1772176448.301791 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1b952461 (stale) Mirror 9: 0x1b952461 (stale) Mirror 11: 0x1b952461 (stale) Mirror 12: 0x1b952461 (stale) Mirror 13: 0x1b952461 (stale) Mirror 14: 0xa80ff305 (stale) Mirror 15: 0x536e59ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1b952461 (stale) Mirror 9: 0x1b952461 (stale) Mirror 11: 0x1b952461 (stale) Mirror 12: 0x1b952461 (stale) Mirror 13: 0x1b952461 (stale) Mirror 14: 0xa80ff305 (stale) Mirror 15: 0x536e59ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1772176448.794711 mapwrite 0x02b16e thru 0x03a1a9 (0xf03c bytes) 000776 1772176448.878523 read 0x001f3b thru 0x011951 (0xfa17 bytes) 000777 1772176448.892169 mapread 0x0198bf thru 0x01b1eb (0x192d bytes) 000778 1772176448.900467 mapread 0x027a5a thru 0x029b2b (0x20d2 bytes) 000780 1772176448.902293 trunc from 0x040000 to 0x03ad37 (0x52c8 bytes) 000782 1772176448.920995 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ad38): Mirror 6: 0x7cd85e6a (stale) Mirror 9: 0x7cd85e6a (stale) Mirror 11: 0x7cd85e6a (stale) Mirror 12: 0x7cd85e6a (stale) Mirror 13: 0x7cd85e6a (stale) Mirror 15: 0xf0aad7d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad38): Mirror 6: 0x7cd85e6a (stale) Mirror 9: 0x7cd85e6a (stale) Mirror 11: 0x7cd85e6a (stale) Mirror 12: 0x7cd85e6a (stale) Mirror 13: 0x7cd85e6a (stale) Mirror 15: 0xf0aad7d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad38: skipped 000785 1772176449.499374 read 0x0344ed thru 0x036e33 (0x2947 bytes) 000787 1772176449.507525 read 0x01bd24 thru 0x027819 (0xbaf6 bytes) 000788 1772176449.531152 trunc from 0x03ad38 to 0x034657 (0x66e0 bytes) 000789 1772176449.561188 write 0x0195aa thru 0x01ac7f (0x16d6 bytes) 000790 1772176449.563328 write 0x02fffd thru 0x03a60a (0xa60e bytes) 000791 1772176449.565396 read 0x000b47 thru 0x00c6b6 (0xbb70 bytes) 000792 1772176449.573954 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a60b): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a60b): Mirror 6: 0x8df1ddc8 (stale) Mirror 9: 0x8df1ddc8 (stale) Mirror 11: 0x8df1ddc8 (stale) Mirror 12: 0x8df1ddc8 (stale) Mirror 13: 0x8df1ddc8 (stale) Mirror 15: 0x9349f4fe (stale) CRC-32 checksum value for chunk [0, 0x3a60b): Mirror 1: 0x694dac6b Mirror 16: 0x694dac6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a60b: skipped 000795 1772176450.543169 read 0x01bb69 thru 0x02288c (0x6d24 bytes) 000796 1772176450.556240 read 0x01f597 thru 0x02dd38 (0xe7a2 bytes) 000800 1772176450.613601 mapwrite 0x0155ce thru 0x0230d3 (0xdb06 bytes) 000802 1772176450.884467 MIRROR_RESYNC 8 mirrors 000803 1772176452.408073 mapread 0x01e263 thru 0x028511 (0xa2af bytes) truncating to largest ever: 0x3fde7 000805 1772176452.464327 trunc from 0x03a60b to 0x03fde6 (0x57dc bytes) 000808 1772176452.554263 mapread 0x00ea99 thru 0x018b68 (0xa0d0 bytes) 000809 1772176452.605123 trunc from 0x03fde7 to 0x02e112 (0x11cd4 bytes) 000810 1772176452.640507 read 0x006d8b thru 0x01264c (0xb8c2 bytes) 000811 1772176452.653638 read 0x00e21e thru 0x00e452 (0x235 bytes) 000813 1772176452.668172 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e113): Mirror 6: 0xd4a5701b (stale) Mirror 9: 0xd4a5701b (stale) Mirror 11: 0xd4a5701b (stale) Mirror 12: 0xd4a5701b (stale) Mirror 15: 0xd4a5701b (stale) Mirror 16: 0xd4a5701b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e113): Mirror 6: 0xd4a5701b (stale) Mirror 9: 0xd4a5701b (stale) Mirror 11: 0xd4a5701b (stale) Mirror 12: 0xd4a5701b (stale) Mirror 15: 0xd4a5701b (stale) Mirror 16: 0xd4a5701b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e113: skipped 000814 1772176453.210492 read 0x002e1d thru 0x012192 (0xf376 bytes) 000817 1772176453.222033 mapread 0x011f83 thru 0x01f6e7 (0xd765 bytes) 000818 1772176453.270590 mapread 0x01404d thru 0x014d27 (0xcdb bytes) 000819 1772176453.272647 write 0x01f362 thru 0x02cee7 (0xdb86 bytes) 000822 1772176453.281110 mapwrite 0x018441 thru 0x020b18 (0x86d8 bytes) 000824 1772176453.356938 mapread 0x017f2f thru 0x0184ee (0x5c0 bytes) 000827 1772176453.366261 MIRROR_RESYNC 7 mirrors 000830 1772176454.882704 read 0x007597 thru 0x00a9ad (0x3417 bytes) 000831 1772176454.895040 trunc from 0x02e113 to 0x0083ab (0x25d67 bytes) 000835 1772176455.040023 write 0x038557 thru 0x03ffff (0x7aa9 bytes) 000836 1772176455.041812 write 0x0346c2 thru 0x03ffff (0xb93e bytes) 000837 1772176455.052796 MIRROR_RESYNC 7 mirrors 000839 1772176456.276274 trunc from 0x040000 to 0x01e13a (0x21ec5 bytes) 000842 1772176456.347645 mapread 0x011d01 thru 0x011d9a (0x9a bytes) 000844 1772176456.354411 mapread 0x005dda thru 0x0073b2 (0x15d9 bytes) 000846 1772176456.375618 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1e13b): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e13b): Mirror 6: 0xff2ea2ea (stale) Mirror 9: 0xff2ea2ea (stale) Mirror 11: 0xff2ea2ea (stale) Mirror 12: 0xff2ea2ea (stale) Mirror 15: 0xff2ea2ea (stale) Mirror 16: 0xff2ea2ea (stale) CRC-32 checksum value for chunk [0, 0x1e13b): Mirror 1: 0xff2ea2ea Mirror 17: 0xff2ea2ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e13b: skipped 000848 1772176457.168395 trunc from 0x01e13b to 0x02b4d8 (0xd39e bytes) 000849 1772176457.285627 read 0x016f50 thru 0x01f746 (0x87f7 bytes) 000852 1772176457.295937 read 0x013934 thru 0x016db8 (0x3485 bytes) 000853 1772176457.310444 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2b4d9): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4d9): Mirror 6: 0x135f80e0 (stale) Mirror 9: 0x135f80e0 (stale) Mirror 11: 0x135f80e0 (stale) Mirror 12: 0x135f80e0 (stale) Mirror 15: 0x135f80e0 (stale) Mirror 16: 0x135f80e0 (stale) Mirror 17: 0x135f80e0 (stale) CRC-32 checksum value for chunk [0, 0x2b4d9): Mirror 1: 0x135f80e0 Mirror 18: 0x135f80e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4d9: skipped 000854 1772176458.003577 mapread 0x027f7c thru 0x02b4d8 (0x355d bytes) 000855 1772176458.043744 trunc from 0x02b4d9 to 0x03cd62 (0x1188a bytes) 000856 1772176458.204171 mapread 0x0321f9 thru 0x039a84 (0x788c bytes) 000857 1772176458.216922 mapwrite 0x023e89 thru 0x02ac69 (0x6de1 bytes) 000859 1772176458.291829 mapread 0x01b90c thru 0x01f717 (0x3e0c bytes) 000861 1772176458.312890 mapread 0x018cd8 thru 0x027114 (0xe43d bytes) 000863 1772176458.316147 read 0x020f9e thru 0x0308a3 (0xf906 bytes) 000864 1772176458.316931 mapwrite 0x02ece0 thru 0x03a97c (0xbc9d bytes) 000866 1772176458.402453 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 6: 0xfee9a1e7 (stale) Mirror 9: 0xfee9a1e7 (stale) Mirror 11: 0xfee9a1e7 (stale) Mirror 12: 0xfee9a1e7 (stale) Mirror 15: 0xfee9a1e7 (stale) Mirror 16: 0xfee9a1e7 (stale) Mirror 17: 0xfa8b8b58 (stale) Mirror 18: 0xfa8b8b58 (stale) CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 1: 0xc3048ef0 Mirror 19: 0xc3048ef0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd63: skipped 000867 1772176459.308011 mapread 0x01c57f thru 0x01d280 (0xd02 bytes) 000868 1772176459.325509 write 0x02d2c2 thru 0x0328a3 (0x55e2 bytes) 000869 1772176459.405891 read 0x006b7a thru 0x008ded (0x2274 bytes) 000870 1772176459.411526 read 0x022b6e thru 0x02fb2f (0xcfc2 bytes) 000871 1772176459.419798 write 0x02d7d9 thru 0x0361f2 (0x8a1a bytes) 000872 1772176459.425807 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 6: 0xfee9a1e7 (stale) Mirror 9: 0xfee9a1e7 (stale) Mirror 11: 0xfee9a1e7 (stale) Mirror 12: 0xfee9a1e7 (stale) Mirror 15: 0xfee9a1e7 (stale) Mirror 16: 0xfee9a1e7 (stale) Mirror 17: 0xfa8b8b58 (stale) Mirror 18: 0xfa8b8b58 (stale) Mirror 19: 0xc3048ef0 (stale) CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 1: 0x2c8ef0b1 Mirror 20: 0x2c8ef0b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd63: skipped 000875 1772176460.557530 read 0x01ce49 thru 0x01e56f (0x1727 bytes) 000878 1772176460.572947 mapwrite 0x033c41 thru 0x03c1b0 (0x8570 bytes) 000879 1772176460.777842 trunc from 0x03cd63 to 0x01adac (0x21fb6 bytes) 000880 1772176460.832334 write 0x023b54 thru 0x02f06c (0xb519 bytes) 000881 1772176460.840736 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2f06d): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f06d): Mirror 6: 0x44db5d0a (stale) Mirror 9: 0x44db5d0a (stale) Mirror 11: 0x44db5d0a (stale) Mirror 12: 0x44db5d0a (stale) Mirror 15: 0x44db5d0a (stale) Mirror 16: 0x44db5d0a (stale) Mirror 17: 0x44db5d0a (stale) Mirror 18: 0x44db5d0a (stale) Mirror 19: 0x5099eddc (stale) Mirror 20: 0x3385d4f8 (stale) CRC-32 checksum value for chunk [0, 0x2f06d): Mirror 1: 0x3bb104ad Mirror 21: 0x3bb104ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f06d: skipped 000882 1772176461.829196 trunc from 0x02f06d to 0x021e4f (0xd21d bytes) 000883 1772176461.934093 write 0x00f2f8 thru 0x01b340 (0xc049 bytes) 000884 1772176461.951056 read 0x01b926 thru 0x01ccb0 (0x138b bytes) 000886 1772176461.972105 write 0x02c966 thru 0x035956 (0x8ff1 bytes) 000887 1772176461.981055 MIRROR_RESYNC 12 mirrors 000888 1772176463.878920 mapwrite 0x00f151 thru 0x018f0d (0x9dbd bytes) 000889 1772176464.081519 read 0x0112f0 thru 0x0148a3 (0x35b4 bytes) 000890 1772176464.082272 write 0x03db2d thru 0x03ffff (0x24d3 bytes) 000891 1772176464.089704 MIRROR_RESYNC 12 mirrors 000893 1772176465.573143 write 0x027d0a thru 0x036993 (0xec8a bytes) 000895 1772176465.675417 write 0x031447 thru 0x03ffff (0xebb9 bytes) 000896 1772176465.677756 mapread 0x025204 thru 0x033a74 (0xe871 bytes) 000897 1772176465.689891 mapread 0x00e193 thru 0x01c7fb (0xe669 bytes) 000900 1772176465.720762 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1d35b154 (stale) Mirror 9: 0x1d35b154 (stale) Mirror 11: 0x1d35b154 (stale) Mirror 12: 0x1d35b154 (stale) Mirror 15: 0x1d35b154 (stale) Mirror 16: 0x1d35b154 (stale) Mirror 17: 0x1d35b154 (stale) Mirror 18: 0x1d35b154 (stale) Mirror 19: 0x1d35b154 (stale) Mirror 20: 0x1d35b154 (stale) Mirror 21: 0x1d35b154 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92c38f6 Mirror 22: 0x92c38f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000901 1772176466.713345 trunc from 0x040000 to 0x013346 (0x2ccb9 bytes) 000902 1772176466.802803 mapwrite 0x00a257 thru 0x015661 (0xb40b bytes) 000904 1772176466.908905 trunc from 0x015662 to 0x00aafe (0xab63 bytes) 000905 1772176466.941757 mapread 0x003ea2 thru 0x00aafe (0x6c5d bytes) 000908 1772176466.957796 MIRROR_RESYNC 13 mirrors 000911 1772176468.504412 mapread 0x002f29 thru 0x009c14 (0x6cec bytes) 000913 1772176468.531507 trunc from 0x00aaff to 0x031aa2 (0x26fa4 bytes) 000915 1772176468.641294 trunc from 0x031aa3 to 0x025558 (0xc54a bytes) 000916 1772176468.663839 write 0x03e0a7 thru 0x03ffff (0x1f59 bytes) 000917 1772176468.665530 write 0x02e04c thru 0x035920 (0x78d5 bytes) 000918 1772176468.676714 read 0x0314fe thru 0x03f658 (0xe15b bytes) 000919 1772176468.683040 write 0x0180bf thru 0x01f229 (0x716b bytes) 000922 1772176468.704158 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x77b17f9b (stale) Mirror 9: 0x77b17f9b (stale) Mirror 11: 0x77b17f9b (stale) Mirror 12: 0x77b17f9b (stale) Mirror 15: 0x77b17f9b (stale) Mirror 16: 0x77b17f9b (stale) Mirror 17: 0x77b17f9b (stale) Mirror 18: 0x77b17f9b (stale) Mirror 19: 0x77b17f9b (stale) Mirror 20: 0x77b17f9b (stale) Mirror 21: 0x77b17f9b (stale) Mirror 22: 0x77b17f9b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x52ad86df Mirror 23: 0x52ad86df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1772176469.682917 trunc from 0x040000 to 0x012297 (0x2dd68 bytes) 000924 1772176469.805723 MIRROR_RESYNC 14 mirrors 000925 1772176471.603402 read 0x004d85 thru 0x00deb5 (0x9131 bytes) 000927 1772176471.613663 trunc from 0x012298 to 0x03b919 (0x29682 bytes) 000929 1772176471.723999 mapwrite 0x0114ae thru 0x0150a5 (0x3bf8 bytes) 000932 1772176471.812840 mapread 0x00b148 thru 0x011b4f (0x6a08 bytes) 000933 1772176471.822171 read 0x020525 thru 0x02b986 (0xb462 bytes) 000934 1772176471.836290 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3b91a): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b91a): Mirror 6: 0x25af6544 (stale) Mirror 9: 0x25af6544 (stale) Mirror 11: 0x25af6544 (stale) Mirror 12: 0x25af6544 (stale) Mirror 15: 0x25af6544 (stale) Mirror 16: 0x25af6544 (stale) Mirror 17: 0x25af6544 (stale) Mirror 18: 0x25af6544 (stale) Mirror 19: 0x25af6544 (stale) Mirror 20: 0x25af6544 (stale) Mirror 21: 0x25af6544 (stale) Mirror 22: 0x25af6544 (stale) Mirror 23: 0x25af6544 (stale) CRC-32 checksum value for chunk [0, 0x3b91a): Mirror 1: 0x943dcd7 Mirror 24: 0x943dcd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b91a: skipped 000936 1772176472.721040 mapwrite 0x01ec54 thru 0x023c87 (0x5034 bytes) 000937 1772176472.912497 write 0x031b87 thru 0x031c29 (0xa3 bytes) 000938 1772176472.929343 read 0x01fd57 thru 0x02d7fb (0xdaa5 bytes) 000939 1772176472.937868 mapwrite 0x021c58 thru 0x02dd30 (0xc0d9 bytes) 000940 1772176472.991690 trunc from 0x03b91a to 0x0307fd (0xb11c bytes) 000941 1772176473.014260 mapread 0x019134 thru 0x025f8f (0xce5c bytes) 000942 1772176473.016521 mapread 0x0127b8 thru 0x01e734 (0xbf7d bytes) 000943 1772176473.029642 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x307fe): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307fe): Mirror 6: 0xa75f8d0a (stale) Mirror 9: 0xa75f8d0a (stale) Mirror 11: 0xa75f8d0a (stale) Mirror 12: 0xa75f8d0a (stale) Mirror 15: 0xa75f8d0a (stale) Mirror 16: 0xa75f8d0a (stale) Mirror 17: 0xa75f8d0a (stale) Mirror 18: 0xa75f8d0a (stale) Mirror 19: 0xa75f8d0a (stale) Mirror 20: 0xa75f8d0a (stale) Mirror 21: 0xa75f8d0a (stale) Mirror 22: 0xa75f8d0a (stale) Mirror 23: 0xa75f8d0a (stale) Mirror 24: 0x21c0ae63 (stale) CRC-32 checksum value for chunk [0, 0x307fe): Mirror 1: 0x65f3ee6a Mirror 25: 0x65f3ee6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307fe: skipped 000947 1772176474.089819 mapwrite 0x007d72 thru 0x008801 (0xa90 bytes) 000954 1772176474.246389 write 0x024423 thru 0x034005 (0xfbe3 bytes) 000955 1772176474.258825 trunc from 0x034006 to 0x013524 (0x20ae1 bytes) 000959 1772176474.280104 trunc from 0x013525 to 0x00aca5 (0x887f bytes) 000961 1772176474.295155 read 0x0032bc thru 0x009cdc (0x6a21 bytes) 000962 1772176474.303111 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaca6): Mirror 6: 0xb0f66b0d (stale) Mirror 9: 0xb0f66b0d (stale) Mirror 12: 0xb0f66b0d (stale) Mirror 15: 0xb0f66b0d (stale) Mirror 16: 0xb0f66b0d (stale) Mirror 17: 0xb0f66b0d (stale) Mirror 18: 0xb0f66b0d (stale) Mirror 19: 0xb0f66b0d (stale) Mirror 20: 0xb0f66b0d (stale) Mirror 21: 0xb0f66b0d (stale) Mirror 22: 0xb0f66b0d (stale) Mirror 23: 0xb0f66b0d (stale) Mirror 24: 0xb0f66b0d (stale) Mirror 25: 0xb0f66b0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaca6): Mirror 6: 0xb0f66b0d (stale) Mirror 9: 0xb0f66b0d (stale) Mirror 12: 0xb0f66b0d (stale) Mirror 15: 0xb0f66b0d (stale) Mirror 16: 0xb0f66b0d (stale) Mirror 17: 0xb0f66b0d (stale) Mirror 18: 0xb0f66b0d (stale) Mirror 19: 0xb0f66b0d (stale) Mirror 20: 0xb0f66b0d (stale) Mirror 21: 0xb0f66b0d (stale) Mirror 22: 0xb0f66b0d (stale) Mirror 23: 0xb0f66b0d (stale) Mirror 24: 0xb0f66b0d (stale) Mirror 25: 0xb0f66b0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaca6: skipped 000964 1772176474.914701 mapwrite 0x013866 thru 0x01d932 (0xa0cd bytes) 000965 1772176475.007651 trunc from 0x01d933 to 0x01c232 (0x1700 bytes) 000967 1772176475.025614 read 0x014da4 thru 0x018570 (0x37cd bytes) 000970 1772176475.035099 mapwrite 0x023e42 thru 0x031898 (0xda57 bytes) 000971 1772176475.170434 read 0x014d8e thru 0x02129c (0xc50f bytes) 000972 1772176475.171111 write 0x01d651 thru 0x02d337 (0xfce7 bytes) 000973 1772176475.172607 mapwrite 0x023724 thru 0x02594a (0x2227 bytes) 000974 1772176475.205931 mapread 0x00f0b5 thru 0x01b73d (0xc689 bytes) 000977 1772176475.207656 mapread 0x026191 thru 0x030dd8 (0xac48 bytes) 000978 1772176475.216783 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x31899): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31899): Mirror 6: 0x8be3fd41 (stale) Mirror 9: 0x8be3fd41 (stale) Mirror 12: 0x8be3fd41 (stale) Mirror 15: 0x8be3fd41 (stale) Mirror 16: 0x8be3fd41 (stale) Mirror 17: 0x8be3fd41 (stale) Mirror 18: 0x8be3fd41 (stale) Mirror 19: 0x8be3fd41 (stale) Mirror 20: 0x8be3fd41 (stale) Mirror 21: 0x8be3fd41 (stale) Mirror 22: 0x8be3fd41 (stale) Mirror 23: 0x8be3fd41 (stale) Mirror 24: 0x7e4cedb0 (stale) Mirror 25: 0xe812f7b4 (stale) CRC-32 checksum value for chunk [0, 0x31899): Mirror 1: 0xe2f2c209 Mirror 26: 0xe2f2c209 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31899: skipped 000980 1772176476.145158 trunc from 0x031899 to 0x03eead (0xd615 bytes) 000981 1772176476.247629 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eeae): Mirror 6: 0x10b0eeed (stale) Mirror 9: 0x10b0eeed (stale) Mirror 12: 0x10b0eeed (stale) Mirror 15: 0x10b0eeed (stale) Mirror 16: 0x10b0eeed (stale) Mirror 17: 0x10b0eeed (stale) Mirror 18: 0x10b0eeed (stale) Mirror 19: 0x10b0eeed (stale) Mirror 21: 0x10b0eeed (stale) Mirror 22: 0x10b0eeed (stale) Mirror 23: 0x10b0eeed (stale) Mirror 24: 0x111687df (stale) Mirror 25: 0xc881018a (stale) Mirror 26: 0xc3adb57a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eeae): Mirror 6: 0x10b0eeed (stale) Mirror 9: 0x10b0eeed (stale) Mirror 12: 0x10b0eeed (stale) Mirror 15: 0x10b0eeed (stale) Mirror 16: 0x10b0eeed (stale) Mirror 17: 0x10b0eeed (stale) Mirror 18: 0x10b0eeed (stale) Mirror 19: 0x10b0eeed (stale) Mirror 21: 0x10b0eeed (stale) Mirror 22: 0x10b0eeed (stale) Mirror 23: 0x10b0eeed (stale) Mirror 24: 0x111687df (stale) Mirror 25: 0xc881018a (stale) Mirror 26: 0xc3adb57a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eeae: skipped 000983 1772176476.868486 read 0x024165 thru 0x032d9e (0xec3a bytes) 000984 1772176476.883491 read 0x02a252 thru 0x02a8b3 (0x662 bytes) 000985 1772176476.884178 mapread 0x0040fb thru 0x00f4b9 (0xb3bf bytes) 000986 1772176476.892198 trunc from 0x03eeae to 0x035ca1 (0x920c bytes) 000987 1772176476.903662 mapwrite 0x026600 thru 0x03272e (0xc12f bytes) 000988 1772176476.950180 read 0x033f57 thru 0x035ca1 (0x1d4b bytes) 000989 1772176476.950933 read 0x01e860 thru 0x025966 (0x7107 bytes) 000990 1772176476.951657 write 0x039755 thru 0x03ffff (0x68ab bytes) 000991 1772176476.952952 trunc from 0x040000 to 0x01fb62 (0x2049d bytes) 000993 1772176476.972202 trunc from 0x01fb63 to 0x00e6b4 (0x114ae bytes) 000994 1772176476.996449 write 0x02c076 thru 0x03b43d (0xf3c8 bytes) 000995 1772176476.998978 mapwrite 0x0285cc thru 0x031dcc (0x9801 bytes) 000996 1772176477.063165 mapwrite 0x00f362 thru 0x016adb (0x777a bytes) 000999 1772176477.118654 write 0x002ae5 thru 0x00df03 (0xb41f bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 02:14:41 (1772176481) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 02:14:44 (1772176484) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32121 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124949 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 02:14:51 (1772176491) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6114592 .. Extending file size to 5626784 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7711552 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9097696 .. Extending file size to 9124704 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8524448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9208064 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8713888 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9217024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9161664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9261120 .. Waiting 55573 55575 55576 55579 55581 failed 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:16:08 (1772176568) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7565984 .. Extending file size to 3486432 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7520256 .. Extending file size to 9355648 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8235296 .. Extending file size to 8270528 .. Extending file size to 8444480 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8446880 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8655648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8901312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57551 57552 57553 57556 57557 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:17:24 (1772176644) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10166 s, 20.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273195 s, 7.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xff1b79ca (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:17:43 (1772176663) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0804802 s, 26.1 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 02:17:49 (1772176669) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x226:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:17:56 (1772176676) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195402 s, 10.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:18:03 (1772176683) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204575 s, 10.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:18:11 (1772176691) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182158 s, 11.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:18:43 (1772176723) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779053 s, 13.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:19:37 (1772176777) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175756 s, 11.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:20:33 (1772176833) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. Creating new pool test_204f oleg420-server: Pool lustre.test_204f created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154301 s, 13.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg420-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:21:45 (1772176905) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:21:52 (1772176912) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:21:58 (1772176918) oleg420-server: Pool lustre.test_206 created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg420-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:22:21 (1772176941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551901 s, 19.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073792 s, 14.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc26f7c40 Mirror 3: 0xc26f7c40 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:22:30 (1772176950) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:22:32 (1772176952) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:22:35 (1772176955) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.125881 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1410000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1990000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12c0000 Read this (4 bytes): BBBB 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 209a (29s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:23:04 (1772176984) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0213422 s, 384 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:23:10 (1772176990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484718 s, 21.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24b lmm_fid: [0x200000404:0x24b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24b lmm_fid: [0x200000404:0x24b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834507 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-27 02:23:11.000000000 -0500 Modify: 2026-02-27 02:23:11.000000000 -0500 Change: 2026-02-27 02:23:12.000000000 -0500 Birth: 2026-02-27 02:23:11.000000000 -0500 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:23:16 (1772176996) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:23:18 (1772176998) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.391067 s, 105 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0762284 s, 53.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 27 02:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x24d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 27 02:23 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:23:25 (1772177005) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:23:31 (1772177011) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1540 sec =========== 02:23:38 (1772177018) === sanity-flr: start cleanup 02:23:39 (1772177019) === === sanity-flr: finish cleanup 02:23:44 (1772177024) ===