-----============= acceptance-small: sanity-flr ============----- Tue Jun 9 04:31:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg425-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:31:46 (1780993906) === oleg425-client.virtnet: executing check_config_client /mnt/lustre oleg425-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg425-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f7a09ce4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f7a09ce4800.idle_timeout=debug disable quota as required oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:32:29 (1780993949) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7554] == sanity-flr test 0a: lfs mirror create with -N option == 04:32:33 (1780993953) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:32:53 (1780993973) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:32:57 (1780993977) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:33:01 (1780993981) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:33:29 (1780994009) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:33:33 (1780994013) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:33:37 (1780994017) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f7a09ce4800.stats=clear osc.lustre-OST0001-osc-ffff9f7a09ce4800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:33:48 (1780994028) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:34:00 (1780994040) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:34:10 (1780994050) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:34:14 (1780994054) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:34:22 (1780994062) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 04:34:33 (1780994073) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:34:41 (1780994081) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:34:44 (1780994084) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:34:51 (1780994091) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:35:01 (1780994101) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:35:07 (1780994107) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.52939 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198758 s, 5.3 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:35:21 (1780994121) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.259833 s, 40.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:35:29 (1780994129) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:35:40 (1780994140) Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:36:51 (1780994211) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184738 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:37:39 (1780994259) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:39:07 (1780994347) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.61408 s, 6.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 9 04:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.637s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.451s user 0m0.000s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.252s user 0m0.000s sys 0m0.201s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f7a09ce4800.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:40:46 (1780994446) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:40:50 (1780994450) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:40:53 (1780994453) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 04:40:57 (1780994457) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 04:41:05 (1780994465) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.756058 s, 16.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:41:20 (1780994480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394511 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202719 s, 51.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:41:27 (1780994487) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:41:38 (1780994498) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00278178 s, 368 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303428 s, 337 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000952778 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000977678 s, 1.0 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 04:41:50 (1780994510) create test files with size 15743191 .. 02d6f0f00fd92b1396b2786a1a0ff655 - c92cd6e1ee251596697821fb64f2dd50 - d37c5b976e35b90181b533844fd8eb99 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 04:42:14 (1780994534) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:42:32 (1780994552) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:42:39 (1780994559) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176859 s, 5.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442193 s, 23.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:42:48 (1780994568) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115073 s, 18.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.441505 s, 9.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.297226 s, 7.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 04:43:05 (1780994585) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:43:09 (1780994589) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:43:12 (1780994592) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390391 s, 26.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 04:43:21 (1780994601) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:43:24 (1780994604) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:43:35 (1780994615) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.374628 s, 28.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:43:47 (1780994627) llite.lustre-ffff9f7a1f7ab800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.238822 s, 43.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f7a104cd000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:44:01 (1780994641) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:44:40 (1780994680) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:45:19 (1780994719) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:45:23 (1780994723) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.240808 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.234814 s, 13.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8ecb2e79 (nosync,stale) Mirror 2: 0x8ecb2e79 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d42c011bc49b28e179be003d5d9c0e57 - mirror 1 checksum: a65761750211773dffa8d0071e1ea321 - mirror 2 checksum: d42c011bc49b28e179be003d5d9c0e57 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-09T04:45:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-09T04:45:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d42c011bc49b28e179be003d5d9c0e57 - mirror 1 checksum: a65761750211773dffa8d0071e1ea321 - mirror 2 checksum: a65761750211773dffa8d0071e1ea321 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-09T04:45:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-09T04:45:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:45:38 (1780994738) Failing mds1 on oleg425-server Stopping /mnt/lustre-mds1 (opts:) on oleg425-server 04:45:45 (1780994745) shut down facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server Failover mds1 to oleg425-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 04:46:19 (1780994779) targets are mounted 04:46:19 (1780994779) facet_failover done oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592817 s, 17.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:46:35 (1780994795) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:46:39 (1780994799) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:46:42 (1780994802) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00972194 s, 1.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:46:49 (1780994809) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:46:52 (1780994812) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:46:55 (1780994815) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:47:19 (1780994839) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:47:49 (1780994869) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:48:13 (1780994893) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:48:39 (1780994919) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 04:48:47 (1780994927) mirror create pid 53490 mirror split pid 53491 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5001: 53490 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5001: 53491 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 04:49:54 (1780994994) Seed set to 4996 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3d937 000004 1780994996.811214 trunc from 00000000 to 0x03d936 (0x3d937 bytes) 000007 1780994996.832214 mapread 0x017574 thru 0x01fcef (0x877c bytes) 000008 1780994996.858051 mapread 0x01b512 thru 0x01dbe3 (0x26d2 bytes) 000010 1780994996.868540 read 0x021d84 thru 0x026b3f (0x4dbc bytes) 000011 1780994996.881159 mapwrite 0x024785 thru 0x02d7a1 (0x901d bytes) 000012 1780994997.058848 write 0x03726d thru 0x03cdaf (0x5b43 bytes) 000013 1780994997.085511 write 0x022a54 thru 0x03147b (0xea28 bytes) 000014 1780994997.105047 MIRROR_EXTEND to 2 mirrors 000015 1780994998.054320 MIRROR_EXTEND to 3 mirrors 000017 1780994999.083691 write 0x02c9dc thru 0x032244 (0x5869 bytes) 000018 1780994999.197922 read 0x0044cc thru 0x00acf6 (0x682b bytes) 000019 1780994999.213384 write 0x027de7 thru 0x02fce9 (0x7f03 bytes) 000022 1780994999.227579 mapwrite 0x01021d thru 0x016fb7 (0x6d9b bytes) 000024 1780994999.386465 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d937): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d937): Mirror 2: 0xfbc06b57 (stale) Mirror 3: 0xfbc06b57 (stale) CRC-32 checksum value for chunk [0, 0x3d937): Mirror 1: 0xd6fa530c Mirror 4: 0xd6fa530c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d937: skipped 000025 1780995000.412398 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d937): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d937): Mirror 2: 0xfbc06b57 (stale) Mirror 3: 0xfbc06b57 (stale) CRC-32 checksum value for chunk [0, 0x3d937): Mirror 1: 0xd6fa530c Mirror 4: 0xd6fa530c Mirror 5: 0xd6fa530c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d937: skipped 000026 1780995001.504875 mapread 0x022721 thru 0x02f03d (0xc91d bytes) 000027 1780995001.555726 mapread 0x014a65 thru 0x021cb3 (0xd24f bytes) 000028 1780995001.569598 read 0x020ab3 thru 0x02933f (0x888d bytes) truncating to largest ever: 0x3f911 000030 1780995001.571159 trunc from 0x03d937 to 0x03f910 (0x1fda bytes) 000031 1780995001.744136 mapread 0x00b7ae thru 0x01445e (0x8cb1 bytes) 000032 1780995001.797900 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f911): Mirror 3: 0x1bc81c92 (stale) Mirror 4: 0xff267220 (stale) Mirror 5: 0xff267220 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f911): Mirror 3: 0x1bc81c92 (stale) Mirror 4: 0xff267220 (stale) Mirror 5: 0xff267220 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f911: skipped 000034 1780995002.396123 read 0x02c156 thru 0x02c213 (0xbe bytes) 000038 1780995002.421439 MIRROR_RESYNC 4 mirrors 000039 1780995003.761979 mapread 0x01f3ec thru 0x021571 (0x2186 bytes) 000040 1780995003.790289 mapwrite 0x000dfc thru 0x003988 (0x2b8d bytes) 000042 1780995004.003056 read 0x01ecb2 thru 0x0245dd (0x592c bytes) 000043 1780995004.018370 write 0x01cb4f thru 0x02c8f4 (0xfda6 bytes) 000045 1780995004.041310 write 0x03a481 thru 0x03d6a0 (0x3220 bytes) 000046 1780995004.064770 trunc from 0x03f911 to 0x03d58a (0x2386 bytes) 000048 1780995004.114368 trunc from 0x03d58b to 0x03c94a (0xc40 bytes) 000049 1780995004.179515 MIRROR_RESYNC 4 mirrors 000050 1780995005.238272 trunc from 0x03c94b to 0x02f61f (0xd32b bytes) 000054 1780995005.380166 MIRROR_RESYNC 4 mirrors 000056 1780995006.442463 read 0x02087f thru 0x025753 (0x4ed5 bytes) 000057 1780995006.467550 write 0x034a94 thru 0x03a142 (0x56af bytes) 000058 1780995006.626037 write 0x02111f thru 0x02c3a4 (0xb286 bytes) 000059 1780995006.640340 write 0x0068f5 thru 0x007c11 (0x131d bytes) 000061 1780995006.666565 trunc from 0x03a143 to 0x03190b (0x8837 bytes) 000063 1780995006.710158 write 0x003cb8 thru 0x010c34 (0xcf7d bytes) 000064 1780995006.734339 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3190c): Mirror 3: 0x68a615de (stale) Mirror 5: 0x68a615de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3190c): Mirror 3: 0x68a615de (stale) Mirror 5: 0x68a615de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3190c: skipped 000065 1780995007.272402 read 0x00aa40 thru 0x00efe2 (0x45a3 bytes) 000066 1780995007.287112 mapwrite 0x022a9c thru 0x026271 (0x37d6 bytes) 000068 1780995007.490201 mapwrite 0x03e6db thru 0x03ffff (0x1925 bytes) 000070 1780995007.591024 trunc from 0x040000 to 0x02db8e (0x12471 bytes) 000073 1780995007.619201 write 0x024451 thru 0x030b21 (0xc6d1 bytes) 000076 1780995007.627866 mapread 0x0136b4 thru 0x0226fd (0xf04a bytes) 000077 1780995007.653440 trunc from 0x030b22 to 0x00556c (0x2b5b5 bytes) 000078 1780995007.682928 mapread 0x000c5f thru 0x0054cd (0x486f bytes) 000080 1780995007.698950 mapwrite 0x010f2d thru 0x0137f2 (0x28c6 bytes) 000082 1780995007.800013 mapwrite 0x0244f1 thru 0x02cf56 (0x8a66 bytes) 000083 1780995007.869718 write 0x002611 thru 0x00e9e4 (0xc3d4 bytes) 000084 1780995007.871840 mapread 0x00b259 thru 0x0163ef (0xb197 bytes) 000085 1780995007.884630 trunc from 0x02cf57 to 0x00988a (0x236cc bytes) 000092 1780995007.916455 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x988b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x988b): Mirror 3: 0x475c383 (stale) Mirror 5: 0x475c383 (stale) CRC-32 checksum value for chunk [0, 0x988b): Mirror 1: 0x827a4a8c Mirror 6: 0x827a4a8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x988b: skipped 000093 1780995009.136414 write 0x038af8 thru 0x03ffff (0x7508 bytes) 000094 1780995009.277312 trunc from 0x040000 to 0x030fe7 (0xf018 bytes) 000095 1780995009.329441 write 0x008d4f thru 0x00b826 (0x2ad8 bytes) 000097 1780995009.341702 mapwrite 0x005296 thru 0x007f4c (0x2cb7 bytes) 000098 1780995009.449523 read 0x01fc6a thru 0x023d6f (0x4106 bytes) 000100 1780995009.492053 read 0x0042d5 thru 0x00ca54 (0x8780 bytes) 000101 1780995009.497044 mapwrite 0x00134e thru 0x00d965 (0xc618 bytes) 000103 1780995009.616894 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30fe8): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fe8): Mirror 3: 0x66ec3f19 (stale) Mirror 5: 0x66ec3f19 (stale) Mirror 6: 0x4b2c7407 (stale) CRC-32 checksum value for chunk [0, 0x30fe8): Mirror 1: 0x85857158 Mirror 7: 0x85857158 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fe8: skipped 000105 1780995011.489822 trunc from 0x030fe8 to 0x035950 (0x4969 bytes) 000106 1780995011.584863 mapwrite 0x0289ef thru 0x0307e8 (0x7dfa bytes) 000107 1780995011.695470 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35951): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35951): Mirror 3: 0x7664d371 (stale) Mirror 5: 0x7664d371 (stale) Mirror 6: 0xfd2a2f50 (stale) Mirror 7: 0x198248f1 (stale) CRC-32 checksum value for chunk [0, 0x35951): Mirror 1: 0xe0a20c2a Mirror 8: 0xe0a20c2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35951: skipped 000111 1780995012.941084 mapwrite 0x0381c8 thru 0x03ffff (0x7e38 bytes) 000112 1780995013.122580 mapwrite 0x0065cb thru 0x00f6ec (0x9122 bytes) 000113 1780995013.187228 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5e118765 (stale) Mirror 6: 0xc6edceaf (stale) Mirror 7: 0xf59d135b (stale) Mirror 8: 0x801d0fb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5e118765 (stale) Mirror 6: 0xc6edceaf (stale) Mirror 7: 0xf59d135b (stale) Mirror 8: 0x801d0fb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1780995013.743020 write 0x00ab2e thru 0x00c90d (0x1de0 bytes) 000115 1780995013.767165 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5e118765 (stale) Mirror 7: 0xf59d135b (stale) Mirror 8: 0x801d0fb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5e118765 (stale) Mirror 7: 0xf59d135b (stale) Mirror 8: 0x801d0fb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000116 1780995014.145973 read 0x03ba80 thru 0x03fde9 (0x436a bytes) 000117 1780995014.171357 trunc from 0x040000 to 0x038762 (0x789d bytes) 000121 1780995014.222995 MIRROR_RESYNC 4 mirrors 000122 1780995015.698726 write 0x023dae thru 0x02b712 (0x7965 bytes) 000123 1780995015.985352 write 0x008511 thru 0x017ce5 (0xf7d5 bytes) 000124 1780995016.017208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38763): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38763): Mirror 3: 0x3708f31e (stale) Mirror 7: 0x3708f31e (stale) Mirror 8: 0x3708f31e (stale) CRC-32 checksum value for chunk [0, 0x38763): Mirror 1: 0x9a7c59a8 Mirror 9: 0x9a7c59a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38763: skipped 000126 1780995017.863632 mapwrite 0x038f79 thru 0x03ffff (0x7087 bytes) 000127 1780995018.152157 trunc from 0x040000 to 0x01b6a8 (0x24957 bytes) 000131 1780995018.185551 mapwrite 0x02f00b thru 0x03c142 (0xd138 bytes) 000132 1780995018.313249 write 0x00f587 thru 0x014911 (0x538b bytes) 000134 1780995018.329390 write 0x0020b6 thru 0x007259 (0x51a4 bytes) 000135 1780995018.351278 trunc from 0x03c143 to 0x03d85a (0x1718 bytes) 000137 1780995018.377238 read 0x01e62a thru 0x0213c7 (0x2d9e bytes) 000138 1780995018.390479 write 0x01443e thru 0x01c949 (0x850c bytes) 000140 1780995018.418060 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d85b): Mirror 7: 0x83fd5a77 (stale) Mirror 8: 0x83fd5a77 (stale) Mirror 9: 0xd765fba9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d85b): Mirror 7: 0x83fd5a77 (stale) Mirror 8: 0x83fd5a77 (stale) Mirror 9: 0xd765fba9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d85b: skipped 000141 1780995019.141223 mapwrite 0x01910c thru 0x024ebf (0xbdb4 bytes) 000142 1780995019.278507 write 0x01417e thru 0x01d8b0 (0x9733 bytes) 000144 1780995019.282018 write 0x02b7e7 thru 0x02d159 (0x1973 bytes) 000146 1780995019.306908 trunc from 0x03d85b to 0x01c80f (0x2104b bytes) 000147 1780995019.349668 trunc from 0x01c810 to 0x00cb1e (0xfcf1 bytes) 000148 1780995019.402943 read 0x003aa5 thru 0x00cb1e (0x907a bytes) 000151 1780995019.440480 write 0x03bce2 thru 0x03f9fe (0x3d1d bytes) 000152 1780995019.485564 MIRROR_RESYNC 4 mirrors 000153 1780995021.020601 MIRROR_EXTEND to 5 mirrors 000154 1780995022.286935 read 0x018dc0 thru 0x01d3fc (0x463d bytes) 000156 1780995022.302583 mapread 0x003308 thru 0x00e2a0 (0xaf99 bytes) 000157 1780995022.331360 write 0x01d90e thru 0x025a21 (0x8114 bytes) 000158 1780995022.449692 write 0x026e97 thru 0x027682 (0x7ec bytes) 000159 1780995022.463985 read 0x005c51 thru 0x0085d0 (0x2980 bytes) 000161 1780995022.517291 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f9ff): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9ff): Mirror 7: 0x91c3a2b9 (stale) Mirror 8: 0x91c3a2b9 (stale) Mirror 9: 0x91c3a2b9 (stale) Mirror 10: 0x91c3a2b9 (stale) CRC-32 checksum value for chunk [0, 0x3f9ff): Mirror 1: 0x1fe10029 Mirror 11: 0x1fe10029 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9ff: skipped 000162 1780995023.514418 trunc from 0x03f9ff to 0x02511d (0x1a8e1 bytes) 000163 1780995023.648555 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2511e): Mirror 7: 0xd196424f (stale) Mirror 8: 0xd196424f (stale) Mirror 9: 0xd196424f (stale) Mirror 11: 0x69860143 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2511e): Mirror 7: 0xd196424f (stale) Mirror 8: 0xd196424f (stale) Mirror 9: 0xd196424f (stale) Mirror 11: 0x69860143 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2511e: skipped 000169 1780995024.182124 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2511e): Mirror 8: 0xd196424f (stale) Mirror 9: 0xd196424f (stale) Mirror 11: 0x69860143 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2511e): Mirror 8: 0xd196424f (stale) Mirror 9: 0xd196424f (stale) Mirror 11: 0x69860143 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2511e: skipped 000170 1780995024.554583 trunc from 0x02511e to 0x02210c (0x3011 bytes) 000171 1780995024.575903 MIRROR_RESYNC 4 mirrors 000172 1780995025.679931 MIRROR_EXTEND to 5 mirrors 000178 1780995026.737828 read 0x01beba thru 0x01f996 (0x3add bytes) 000179 1780995026.761076 mapread 0x0081eb thru 0x0164f8 (0xe30e bytes) 000180 1780995026.820079 read 0x010671 thru 0x017663 (0x6ff3 bytes) 000183 1780995026.821404 read 0x01bc5e thru 0x01f365 (0x3708 bytes) 000184 1780995026.826342 mapwrite 0x001c70 thru 0x00ac94 (0x9025 bytes) 000187 1780995027.474267 read 0x01ad0a thru 0x01deb5 (0x31ac bytes) 000188 1780995027.563608 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2210d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2210d): Mirror 8: 0xe90778f (stale) Mirror 9: 0xe90778f (stale) Mirror 11: 0xe90778f (stale) Mirror 12: 0xe90778f (stale) CRC-32 checksum value for chunk [0, 0x2210d): Mirror 1: 0x6b6b8026 Mirror 13: 0x6b6b8026 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2210d: skipped 000189 1780995029.675798 read 0x004aaf thru 0x00f58b (0xaadd bytes) 000191 1780995029.715226 read 0x009563 thru 0x00ad21 (0x17bf bytes) 000192 1780995029.728619 read 0x0175a9 thru 0x02210c (0xab64 bytes) 000193 1780995029.764074 trunc from 0x02210d to 0x024e6b (0x2d5f bytes) 000194 1780995030.009468 read 0x024478 thru 0x024e6b (0x9f4 bytes) 000195 1780995030.027308 mapwrite 0x01a2ab thru 0x020268 (0x5fbe bytes) 000196 1780995030.120449 trunc from 0x024e6c to 0x034892 (0xfa27 bytes) 000198 1780995030.153787 trunc from 0x034893 to 0x0118ee (0x22fa4 bytes) 000199 1780995030.186549 read 0x00fc58 thru 0x0118ee (0x1c97 bytes) 000200 1780995030.187402 mapread 0x00feb6 thru 0x0118ee (0x1a39 bytes) 000204 1780995030.206108 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x118ef): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x118ef): Mirror 8: 0x9b1c9ee7 (stale) Mirror 9: 0x9b1c9ee7 (stale) Mirror 11: 0x9b1c9ee7 (stale) Mirror 12: 0x9b1c9ee7 (stale) Mirror 13: 0x27ea72c8 (stale) CRC-32 checksum value for chunk [0, 0x118ef): Mirror 1: 0x27ea72c8 Mirror 14: 0x27ea72c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x118ef: skipped 000205 1780995031.914991 read 0x004615 thru 0x0082c4 (0x3cb0 bytes) 000206 1780995031.952786 mapwrite 0x02c8d9 thru 0x03865b (0xbd83 bytes) 000207 1780995032.439068 mapread 0x029eae thru 0x02d16c (0x32bf bytes) 000209 1780995032.456033 mapwrite 0x030b69 thru 0x03b4d1 (0xa969 bytes) 000210 1780995032.781884 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 8: 0x8625275d (stale) Mirror 9: 0x8625275d (stale) Mirror 11: 0x8625275d (stale) Mirror 12: 0x8625275d (stale) Mirror 13: 0xc4a39bec (stale) Mirror 14: 0xaff91a93 (stale) CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 1: 0x43f55df3 Mirror 15: 0x43f55df3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4d2: skipped 000211 1780995035.666321 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 8: 0x8625275d (stale) Mirror 11: 0x8625275d (stale) Mirror 12: 0x8625275d (stale) Mirror 13: 0xc4a39bec (stale) Mirror 14: 0xaff91a93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 8: 0x8625275d (stale) Mirror 11: 0x8625275d (stale) Mirror 12: 0x8625275d (stale) Mirror 13: 0xc4a39bec (stale) Mirror 14: 0xaff91a93 (stale) CRC-32 checksum value for chunk [0, 0x3b4d2): Mirror 1: 0x43f55df3 Mirror 15: 0x43f55df3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4d2: skipped 000212 1780995036.975183 trunc from 0x03b4d2 to 0x004f1b (0x365b6 bytes) 000213 1780995037.226816 mapwrite 0x0371ab thru 0x03ffff (0x8e55 bytes) 000216 1780995037.459557 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x78ae62f0 (stale) Mirror 11: 0x78ae62f0 (stale) Mirror 12: 0x78ae62f0 (stale) Mirror 13: 0x29f09d9f (stale) Mirror 14: 0xb6abb62d (stale) Mirror 15: 0x16b89d98 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f0e45e2 Mirror 16: 0x1f0e45e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000217 1780995040.068021 write 0x03feea thru 0x03ffff (0x116 bytes) 000218 1780995040.196775 mapread 0x011d33 thru 0x01d1dc (0xb4aa bytes) 000219 1780995040.272791 read 0x03be8f thru 0x03ffff (0x4171 bytes) 000223 1780995040.286237 write 0x020cba thru 0x025bda (0x4f21 bytes) 000226 1780995040.309848 mapwrite 0x002a95 thru 0x00b5ec (0x8b58 bytes) 000229 1780995040.491179 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x78ae62f0 (stale) Mirror 11: 0x78ae62f0 (stale) Mirror 12: 0x78ae62f0 (stale) Mirror 13: 0x29f09d9f (stale) Mirror 14: 0xb6abb62d (stale) Mirror 15: 0x16b89d98 (stale) Mirror 16: 0x1f0e45e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcf461023 Mirror 17: 0xcf461023 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1780995042.373181 write 0x0091ca thru 0x0148ee (0xb725 bytes) 000231 1780995042.600131 read 0x01d472 thru 0x020785 (0x3314 bytes) 000235 1780995042.626285 trunc from 0x040000 to 0x0216eb (0x1e914 bytes) 000237 1780995042.664543 mapwrite 0x0163aa thru 0x01c30c (0x5f63 bytes) 000238 1780995042.847013 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x216ec): Mirror 11: 0x4580c194 (stale) Mirror 12: 0x4580c194 (stale) Mirror 13: 0x52aa60e (stale) Mirror 14: 0x5d86743d (stale) Mirror 15: 0x5d86743d (stale) Mirror 16: 0x4570c30f (stale) Mirror 17: 0x1c06b888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x216ec): Mirror 11: 0x4580c194 (stale) Mirror 12: 0x4580c194 (stale) Mirror 13: 0x52aa60e (stale) Mirror 14: 0x5d86743d (stale) Mirror 15: 0x5d86743d (stale) Mirror 16: 0x4570c30f (stale) Mirror 17: 0x1c06b888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x216ec: skipped 000239 1780995044.009520 trunc from 0x0216ec to 0x0251fc (0x3b11 bytes) 000242 1780995044.086603 write 0x01315c thru 0x0214d4 (0xe379 bytes) 000244 1780995044.140310 read 0x016a95 thru 0x019895 (0x2e01 bytes) 000245 1780995044.149100 mapread 0x01085e thru 0x019038 (0x87db bytes) 000249 1780995044.225504 write 0x02bb42 thru 0x0395d1 (0xda90 bytes) 000250 1780995044.239209 mapread 0x02d9fe thru 0x032541 (0x4b44 bytes) 000252 1780995044.260277 mapread 0x0199bc thru 0x01ef82 (0x55c7 bytes) 000253 1780995044.261485 mapread 0x00f74d thru 0x01a171 (0xaa25 bytes) 000254 1780995044.262773 read 0x010857 thru 0x01fe82 (0xf62c bytes) 000255 1780995044.263592 mapread 0x012055 thru 0x01ac2b (0x8bd7 bytes) 000256 1780995044.264924 write 0x03fd8e thru 0x03ffff (0x272 bytes) 000257 1780995044.274275 MIRROR_RESYNC 8 mirrors 000259 1780995047.019973 mapwrite 0x005a86 thru 0x00b8ec (0x5e67 bytes) 000260 1780995047.347574 trunc from 0x040000 to 0x03e7dc (0x1823 bytes) 000261 1780995047.386330 mapwrite 0x027e09 thru 0x03202d (0xa225 bytes) 000262 1780995047.579194 mapwrite 0x0226ac thru 0x030c1a (0xe56f bytes) 000263 1780995047.792739 mapread 0x01c213 thru 0x026553 (0xa341 bytes) 000264 1780995047.851263 mapread 0x00319e thru 0x0106be (0xd521 bytes) 000266 1780995047.857939 trunc from 0x03e7dd to 0x011db4 (0x2ca28 bytes) 000267 1780995047.917609 mapread 0x004512 thru 0x009434 (0x4f23 bytes) 000269 1780995047.919383 trunc from 0x011db5 to 0x0140a9 (0x22f5 bytes) 000273 1780995047.958509 write 0x00e8da thru 0x01d1f2 (0xe919 bytes) 000274 1780995047.966385 mapread 0x017d83 thru 0x01d1f2 (0x5470 bytes) 000276 1780995047.982304 mapread 0x017cbf thru 0x01a599 (0x28db bytes) 000277 1780995047.983403 mapwrite 0x0229d9 thru 0x02abbe (0x81e6 bytes) 000278 1780995048.248543 mapread 0x026601 thru 0x02abbe (0x45be bytes) 000279 1780995048.256597 write 0x003eab thru 0x00f46c (0xb5c2 bytes) 000280 1780995048.261382 write 0x01ee5c thru 0x022580 (0x3725 bytes) 000283 1780995048.268048 read 0x0238ff thru 0x02abbe (0x72c0 bytes) 000284 1780995048.269261 write 0x01fba7 thru 0x02f0fd (0xf557 bytes) 000285 1780995048.274753 write 0x018a39 thru 0x028380 (0xf948 bytes) 000286 1780995048.279223 mapread 0x01ec7b thru 0x02ad8e (0xc114 bytes) 000288 1780995048.292695 mapread 0x021aa1 thru 0x023395 (0x18f5 bytes) 000289 1780995048.314824 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f0fe): Mirror 12: 0xceb386d3 (stale) Mirror 13: 0xceb386d3 (stale) Mirror 14: 0xceb386d3 (stale) Mirror 15: 0xceb386d3 (stale) Mirror 16: 0xceb386d3 (stale) Mirror 17: 0xceb386d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f0fe): Mirror 12: 0xceb386d3 (stale) Mirror 13: 0xceb386d3 (stale) Mirror 14: 0xceb386d3 (stale) Mirror 15: 0xceb386d3 (stale) Mirror 16: 0xceb386d3 (stale) Mirror 17: 0xceb386d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f0fe: skipped 000291 1780995049.119242 mapread 0x001926 thru 0x0105c5 (0xeca0 bytes) 000292 1780995049.165360 trunc from 0x02f0fe to 0x032aeb (0x39ee bytes) 000296 1780995049.195066 trunc from 0x032aec to 0x037455 (0x496a bytes) 000297 1780995049.237701 write 0x0259bd thru 0x02872b (0x2d6f bytes) 000298 1780995049.256959 mapwrite 0x017697 thru 0x01b739 (0x40a3 bytes) 000300 1780995049.362881 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37456): Mirror 12: 0x4bf8e952 (stale) Mirror 13: 0x4bf8e952 (stale) Mirror 14: 0x4bf8e952 (stale) Mirror 15: 0x4bf8e952 (stale) Mirror 17: 0x4bf8e952 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37456): Mirror 12: 0x4bf8e952 (stale) Mirror 13: 0x4bf8e952 (stale) Mirror 14: 0x4bf8e952 (stale) Mirror 15: 0x4bf8e952 (stale) Mirror 17: 0x4bf8e952 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37456: skipped 000301 1780995049.906290 trunc from 0x037456 to 0x02b4f6 (0xbf5f bytes) 000303 1780995049.973937 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b4f7): Mirror 12: 0x6dd9a97c (stale) Mirror 13: 0x6dd9a97c (stale) Mirror 14: 0x6dd9a97c (stale) Mirror 17: 0x6dd9a97c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4f7): Mirror 12: 0x6dd9a97c (stale) Mirror 13: 0x6dd9a97c (stale) Mirror 14: 0x6dd9a97c (stale) Mirror 17: 0x6dd9a97c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4f7: skipped 000304 1780995050.804722 read 0x02b0e0 thru 0x02b4f6 (0x417 bytes) 000305 1780995050.810021 write 0x013e00 thru 0x01fcfd (0xbefe bytes) 000307 1780995050.829608 trunc from 0x02b4f7 to 0x01b39d (0x10159 bytes) 000308 1780995050.859772 read 0x002879 thru 0x008713 (0x5e9b bytes) 000310 1780995050.876912 write 0x016673 thru 0x017a33 (0x13c1 bytes) 000311 1780995050.880473 read 0x00c0a4 thru 0x0198d7 (0xd834 bytes) 000314 1780995050.882915 read 0x004e54 thru 0x00d738 (0x88e5 bytes) 000315 1780995050.884588 mapread 0x017ba5 thru 0x01a164 (0x25c0 bytes) 000316 1780995050.892504 MIRROR_RESYNC 5 mirrors 000318 1780995052.528940 read 0x016457 thru 0x01b39d (0x4f47 bytes) 000319 1780995052.561139 mapwrite 0x011287 thru 0x011ee4 (0xc5e bytes) 000320 1780995052.752499 read 0x000246 thru 0x00fe4a (0xfc05 bytes) 000323 1780995052.782390 read 0x00fdc0 thru 0x012a03 (0x2c44 bytes) 000325 1780995052.800215 read 0x01096d thru 0x017312 (0x69a6 bytes) 000327 1780995052.802337 trunc from 0x01b39e to 0x026b06 (0xb769 bytes) 000328 1780995052.890209 mapwrite 0x00bdd4 thru 0x00d7cb (0x19f8 bytes) 000329 1780995053.007626 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26b07): Mirror 13: 0x8e02960e (stale) Mirror 14: 0x8e02960e (stale) Mirror 17: 0x8e02960e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b07): Mirror 13: 0x8e02960e (stale) Mirror 14: 0x8e02960e (stale) Mirror 17: 0x8e02960e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b07: skipped 000332 1780995053.498111 mapread 0x00f0d1 thru 0x010d7b (0x1cab bytes) 000340 1780995053.530025 read 0x020d69 thru 0x026b06 (0x5d9e bytes) 000342 1780995053.544882 mapread 0x01d7a7 thru 0x026b06 (0x9360 bytes) 000343 1780995053.557413 read 0x00f6ab thru 0x01ad1e (0xb674 bytes) 000346 1780995053.567845 mapwrite 0x007f5d thru 0x011e7c (0x9f20 bytes) 000347 1780995053.679289 trunc from 0x026b07 to 0x016dd5 (0xfd31 bytes) 000351 1780995053.727498 read 0x000356 thru 0x003bda (0x3885 bytes) 000353 1780995053.730372 read 0x009078 thru 0x016dd5 (0xdd5e bytes) 000354 1780995053.732523 write 0x0377c2 thru 0x037a72 (0x2b1 bytes) 000355 1780995053.738599 mapread 0x001236 thru 0x00b949 (0xa714 bytes) 000360 1780995053.761526 MIRROR_RESYNC 4 mirrors 000361 1780995055.132126 trunc from 0x037a73 to 0x0022c3 (0x357af bytes) 000362 1780995055.249639 trunc from 0x0022c4 to 0x019663 (0x173a0 bytes) 000363 1780995055.279872 trunc from 0x019664 to 0x033f66 (0x1a903 bytes) 000364 1780995055.303850 mapwrite 0x01cc20 thru 0x027e14 (0xb1f5 bytes) 000366 1780995055.401987 read 0x02de0c thru 0x033f66 (0x615b bytes) 000369 1780995055.410481 trunc from 0x033f67 to 0x015c09 (0x1e35d bytes) 000370 1780995055.431456 write 0x01e63f thru 0x02a882 (0xc244 bytes) 000373 1780995055.437443 mapwrite 0x00a573 thru 0x014eb4 (0xa942 bytes) 000375 1780995055.513978 mapwrite 0x031167 thru 0x034281 (0x311b bytes) 000379 1780995055.599244 write 0x0015be thru 0x00e3a4 (0xcde7 bytes) 000381 1780995055.609105 write 0x01bdbe thru 0x02482e (0x8a71 bytes) 000383 1780995055.612399 write 0x010724 thru 0x01ce9c (0xc779 bytes) 000384 1780995055.619706 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34282): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34282): Mirror 13: 0x4276e834 (stale) Mirror 14: 0x4276e834 (stale) Mirror 17: 0x4276e834 (stale) CRC-32 checksum value for chunk [0, 0x34282): Mirror 1: 0x5a0465ed Mirror 18: 0x5a0465ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34282: skipped 000385 1780995056.541806 mapwrite 0x02338e thru 0x0330dd (0xfd50 bytes) 000386 1780995056.781727 mapwrite 0x031be9 thru 0x03263a (0xa52 bytes) 000387 1780995056.831462 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34282): Mirror 13: 0x4276e834 (stale) Mirror 14: 0x4276e834 (stale) Mirror 18: 0x5a0465ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34282): Mirror 13: 0x4276e834 (stale) Mirror 14: 0x4276e834 (stale) Mirror 18: 0x5a0465ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34282: skipped 000388 1780995057.315581 read 0x01eff6 thru 0x0252db (0x62e6 bytes) 000389 1780995057.327964 write 0x0219b9 thru 0x0262d0 (0x4918 bytes) 000390 1780995057.336788 read 0x01abc6 thru 0x0276b9 (0xcaf4 bytes) 000391 1780995057.350594 trunc from 0x034282 to 0x00a255 (0x2a02c bytes) 000393 1780995057.400685 read 0x008878 thru 0x00a255 (0x19de bytes) 000394 1780995057.402274 mapwrite 0x034d8c thru 0x03ffff (0xb274 bytes) 000396 1780995057.502400 write 0x02e1b5 thru 0x0347e1 (0x662d bytes) 000398 1780995057.506569 write 0x001ca3 thru 0x00c415 (0xa773 bytes) 000401 1780995057.514273 write 0x01b48f thru 0x025bd5 (0xa747 bytes) 000402 1780995057.516412 mapwrite 0x021e12 thru 0x02bc63 (0x9e52 bytes) 000403 1780995057.576739 trunc from 0x040000 to 0x0286ff (0x17900 bytes) 000405 1780995057.603108 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28700): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28700): Mirror 13: 0x19ac8f17 (stale) Mirror 14: 0x19ac8f17 (stale) Mirror 18: 0xd13dd4c9 (stale) CRC-32 checksum value for chunk [0, 0x28700): Mirror 1: 0xac9a1edc Mirror 19: 0xac9a1edc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28700: skipped 000406 1780995058.937038 trunc from 0x028700 to 0x00c18b (0x1c574 bytes) 000407 1780995059.072237 mapwrite 0x0181cf thru 0x01c192 (0x3fc4 bytes) 000410 1780995059.221816 mapread 0x00e8f0 thru 0x0136ba (0x4dcb bytes) 000414 1780995059.272835 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c193): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c193): Mirror 13: 0x5dca33fa (stale) Mirror 14: 0x5dca33fa (stale) Mirror 18: 0xb058bea6 (stale) Mirror 19: 0xc82f160f (stale) CRC-32 checksum value for chunk [0, 0x1c193): Mirror 1: 0x8a8b7fb6 Mirror 20: 0x8a8b7fb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c193: skipped 000416 1780995060.398600 trunc from 0x01c193 to 0x0275ae (0xb41c bytes) 000417 1780995060.616325 read 0x00e2fd thru 0x013910 (0x5614 bytes) 000420 1780995060.666561 mapwrite 0x027aea thru 0x02a61a (0x2b31 bytes) 000421 1780995060.898109 mapread 0x00d4ae thru 0x0166f0 (0x9243 bytes) 000423 1780995060.925461 trunc from 0x02a61b to 0x021016 (0x9604 bytes) 000424 1780995060.959231 read 0x0007ea thru 0x0069c9 (0x61e0 bytes) 000425 1780995060.979246 mapread 0x01fe0e thru 0x021016 (0x1209 bytes) 000426 1780995060.983156 read 0x01979b thru 0x021016 (0x787c bytes) 000427 1780995060.984203 write 0x03c2ac thru 0x03dd80 (0x1ad5 bytes) 000431 1780995060.989029 mapread 0x002c2a thru 0x010381 (0xd758 bytes) 000432 1780995060.991413 mapread 0x003751 thru 0x003af2 (0x3a2 bytes) 000434 1780995060.993572 mapread 0x00c4b9 thru 0x0167ba (0xa302 bytes) 000436 1780995061.001193 MIRROR_RESYNC 6 mirrors 000437 1780995062.793870 trunc from 0x03dd81 to 0x024837 (0x19549 bytes) 000438 1780995062.875128 mapread 0x004c72 thru 0x01011e (0xb4ad bytes) 000439 1780995062.897787 mapwrite 0x01b5b6 thru 0x0281e5 (0xcc30 bytes) 000442 1780995063.032358 trunc from 0x0281e6 to 0x032386 (0xa1a1 bytes) 000443 1780995063.055626 mapread 0x011b7c thru 0x0195be (0x7a43 bytes) 000446 1780995063.069237 trunc from 0x032387 to 0x033191 (0xe0b bytes) 000448 1780995063.112292 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33192): Mirror 13: 0xa2c47cc8 (stale) Mirror 14: 0xa2c47cc8 (stale) Mirror 18: 0xa2c47cc8 (stale) Mirror 20: 0xa2c47cc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33192): Mirror 13: 0xa2c47cc8 (stale) Mirror 14: 0xa2c47cc8 (stale) Mirror 18: 0xa2c47cc8 (stale) Mirror 20: 0xa2c47cc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33192: skipped 000449 1780995063.755254 MIRROR_RESYNC 5 mirrors 000450 1780995065.040313 mapwrite 0x03d733 thru 0x03ffff (0x28cd bytes) 000451 1780995065.269299 mapwrite 0x01117f thru 0x01c3fa (0xb27c bytes) 000456 1780995065.404304 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x30bbca2f (stale) Mirror 14: 0x30bbca2f (stale) Mirror 18: 0x30bbca2f (stale) Mirror 20: 0x30bbca2f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa5500c82 Mirror 21: 0xa5500c82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000457 1780995066.506220 mapwrite 0x007f0b thru 0x00c7f5 (0x48eb bytes) 000458 1780995066.802720 trunc from 0x040000 to 0x0297a5 (0x1685a bytes) 000459 1780995066.855847 mapwrite 0x019739 thru 0x0290bf (0xf987 bytes) 000462 1780995066.941979 write 0x013d7c thru 0x022665 (0xe8ea bytes) 000463 1780995066.944592 trunc from 0x0297a6 to 0x021c5e (0x7b47 bytes) 000466 1780995066.964525 mapwrite 0x031d52 thru 0x03ca5c (0xad0b bytes) 000467 1780995067.076446 write 0x012813 thru 0x018db2 (0x65a0 bytes) 000468 1780995067.085993 mapread 0x004b3b thru 0x00d9c6 (0x8e8c bytes) 000470 1780995067.097482 MIRROR_RESYNC 6 mirrors 000471 1780995068.512205 mapread 0x03c9ee thru 0x03ca5c (0x6f bytes) 000473 1780995068.522702 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000474 1780995068.997107 mapread 0x00c345 thru 0x012d9c (0x6a58 bytes) 000475 1780995069.025282 mapread 0x0039a2 thru 0x006bc4 (0x3223 bytes) 000476 1780995069.034430 mapwrite 0x02218e thru 0x029ed8 (0x7d4b bytes) 000477 1780995069.314489 mapread 0x01db2c thru 0x0297af (0xbc84 bytes) 000478 1780995069.319435 trunc from 0x03ca5d to 0x00ddb1 (0x2ecab bytes) 000479 1780995069.389511 write 0x0163fc thru 0x020cf9 (0xa8fe bytes) 000480 1780995069.394877 mapread 0x00caba thru 0x01b5c3 (0xeb0a bytes) 000481 1780995069.424199 write 0x0357b4 thru 0x03cba7 (0x73f4 bytes) 000484 1780995069.429766 read 0x0017f4 thru 0x00b204 (0x9a11 bytes) 000485 1780995069.446173 write 0x001fb4 thru 0x00c8b7 (0xa904 bytes) 000487 1780995069.452070 read 0x02347c thru 0x02a046 (0x6bcb bytes) 000488 1780995069.465325 mapwrite 0x034c4c thru 0x037902 (0x2cb7 bytes) 000490 1780995069.514142 mapwrite 0x00b325 thru 0x015608 (0xa2e4 bytes) 000491 1780995069.567166 mapwrite 0x0372a2 thru 0x03c11a (0x4e79 bytes) 000492 1780995069.613520 MIRROR_RESYNC 5 mirrors 000495 1780995070.730294 mapwrite 0x021be1 thru 0x03151c (0xf93c bytes) 000497 1780995070.969359 read 0x01247c thru 0x012846 (0x3cb bytes) 000498 1780995070.977219 trunc from 0x03cba8 to 0x03a73c (0x246b bytes) 000499 1780995071.026582 read 0x005d94 thru 0x013422 (0xd68f bytes) 000502 1780995071.042004 read 0x017436 thru 0x01da6f (0x663a bytes) 000504 1780995071.048681 mapwrite 0x009bb7 thru 0x0130e1 (0x952b bytes) 000506 1780995071.143745 mapread 0x00b78a thru 0x0166f4 (0xaf6b bytes) 000509 1780995071.162002 write 0x03294f thru 0x03a6ba (0x7d6c bytes) 000510 1780995071.173638 mapread 0x01e197 thru 0x024312 (0x617c bytes) 000512 1780995071.175761 mapread 0x0399fd thru 0x03a73c (0xd40 bytes) 000514 1780995071.189982 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a73d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a73d): Mirror 13: 0x45aee791 (stale) Mirror 14: 0x45aee791 (stale) Mirror 18: 0x45aee791 (stale) Mirror 20: 0x45aee791 (stale) CRC-32 checksum value for chunk [0, 0x3a73d): Mirror 1: 0x10d310d6 Mirror 21: 0x10d310d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a73d: skipped 000515 1780995072.300826 write 0x038ca6 thru 0x03ffff (0x735a bytes) 000518 1780995072.391073 mapread 0x02ee53 thru 0x0385ae (0x975c bytes) 000519 1780995072.423421 mapread 0x021053 thru 0x0225a3 (0x1551 bytes) 000521 1780995072.448599 trunc from 0x040000 to 0x029dc9 (0x16236 bytes) 000522 1780995072.504151 mapread 0x029231 thru 0x029dc9 (0xb99 bytes) 000523 1780995072.505499 mapread 0x00c393 thru 0x012b77 (0x67e5 bytes) 000524 1780995072.515700 mapwrite 0x0008b8 thru 0x008bc2 (0x830b bytes) 000525 1780995072.578486 trunc from 0x029dca to 0x03ce55 (0x1308c bytes) 000526 1780995072.593737 mapread 0x03c9be thru 0x03ce55 (0x498 bytes) 000527 1780995072.604720 mapwrite 0x00a098 thru 0x011fd1 (0x7f3a bytes) 000529 1780995072.646250 mapwrite 0x013fc5 thru 0x020c74 (0xccb0 bytes) 000531 1780995072.704250 mapwrite 0x028f4f thru 0x02efac (0x605e bytes) 000532 1780995072.761649 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ce56): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce56): Mirror 13: 0x78ad9014 (stale) Mirror 14: 0x78ad9014 (stale) Mirror 18: 0x78ad9014 (stale) Mirror 20: 0x78ad9014 (stale) Mirror 21: 0x72c70bc8 (stale) CRC-32 checksum value for chunk [0, 0x3ce56): Mirror 1: 0xcd15a8d Mirror 22: 0xcd15a8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce56: skipped 000533 1780995074.073297 read 0x01d330 thru 0x029e7e (0xcb4f bytes) 000534 1780995074.094826 mapwrite 0x0146ae thru 0x01b3e1 (0x6d34 bytes) 000536 1780995074.285976 read 0x016902 thru 0x01a192 (0x3891 bytes) 000537 1780995074.286809 mapread 0x0383f9 thru 0x03c016 (0x3c1e bytes) 000538 1780995074.322487 read 0x01dcdf thru 0x0288e2 (0xac04 bytes) 000539 1780995074.332118 mapread 0x0378f0 thru 0x03ce55 (0x5566 bytes) 000546 1780995074.350990 trunc from 0x03ce56 to 0x03cf41 (0xec bytes) 000547 1780995074.392983 mapread 0x03b837 thru 0x03cf41 (0x170b bytes) 000551 1780995074.408110 mapwrite 0x02cc02 thru 0x031dd1 (0x51d0 bytes) 000552 1780995074.474356 mapread 0x031fa8 thru 0x03cf41 (0xaf9a bytes) 000556 1780995074.487940 mapwrite 0x025bc3 thru 0x03224b (0xc689 bytes) 000557 1780995074.576074 read 0x00c948 thru 0x013f36 (0x75ef bytes) 000558 1780995074.579187 mapwrite 0x0376bf thru 0x03a4d2 (0x2e14 bytes) 000559 1780995074.629251 read 0x03ba94 thru 0x03cf41 (0x14ae bytes) 000560 1780995074.630576 mapwrite 0x03b6fc thru 0x03ffff (0x4904 bytes) 000561 1780995074.745349 read 0x038d3f thru 0x03ffff (0x72c1 bytes) 000562 1780995074.748247 mapread 0x02d019 thru 0x02f95d (0x2945 bytes) 000563 1780995074.749846 mapread 0x034ff4 thru 0x03ffff (0xb00c bytes) 000564 1780995074.754897 read 0x00f2e8 thru 0x01936b (0xa084 bytes) 000565 1780995074.755765 mapread 0x0021f0 thru 0x00a9df (0x87f0 bytes) 000566 1780995074.758231 mapread 0x031bf6 thru 0x03ffff (0xe40a bytes) 000567 1780995074.769620 MIRROR_RESYNC 7 mirrors 000569 1780995076.209830 trunc from 0x040000 to 0x003d09 (0x3c2f6 bytes) 000570 1780995076.348191 trunc from 0x003d0a to 0x02830d (0x24604 bytes) 000571 1780995076.374508 mapwrite 0x033ccf thru 0x0359f6 (0x1d28 bytes) 000572 1780995076.492064 MIRROR_RESYNC 7 mirrors 000574 1780995077.748596 write 0x030bc7 thru 0x03c0aa (0xb4e4 bytes) 000575 1780995077.846231 mapread 0x026d63 thru 0x0369c7 (0xfc65 bytes) 000578 1780995077.870084 mapwrite 0x029b1c thru 0x02fc65 (0x614a bytes) 000579 1780995077.978568 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c0ab): Mirror 14: 0x21edf9d5 (stale) Mirror 18: 0x21edf9d5 (stale) Mirror 20: 0x21edf9d5 (stale) Mirror 21: 0x21edf9d5 (stale) Mirror 22: 0x21edf9d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0ab): Mirror 14: 0x21edf9d5 (stale) Mirror 18: 0x21edf9d5 (stale) Mirror 20: 0x21edf9d5 (stale) Mirror 21: 0x21edf9d5 (stale) Mirror 22: 0x21edf9d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0ab: skipped 000581 1780995078.542549 write 0x009d13 thru 0x015106 (0xb3f4 bytes) 000582 1780995078.562221 mapwrite 0x01a7a0 thru 0x01fb10 (0x5371 bytes) 000584 1780995078.694973 mapread 0x033e80 thru 0x03c0aa (0x822b bytes) 000585 1780995078.718447 write 0x013465 thru 0x013fd9 (0xb75 bytes) 000586 1780995078.725165 mapread 0x0002de thru 0x009c30 (0x9953 bytes) 000588 1780995078.746520 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c0ab): Mirror 14: 0x21edf9d5 (stale) Mirror 20: 0x21edf9d5 (stale) Mirror 21: 0x21edf9d5 (stale) Mirror 22: 0x21edf9d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0ab): Mirror 14: 0x21edf9d5 (stale) Mirror 20: 0x21edf9d5 (stale) Mirror 21: 0x21edf9d5 (stale) Mirror 22: 0x21edf9d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0ab: skipped 000593 1780995079.293253 mapwrite 0x02c32e thru 0x02e396 (0x2069 bytes) 000595 1780995079.388133 mapread 0x038a1f thru 0x03a59b (0x1b7d bytes) 000597 1780995079.421292 read 0x017b3a thru 0x01e87a (0x6d41 bytes) 000598 1780995079.430215 trunc from 0x03c0ab to 0x018d7d (0x2332d bytes) 000600 1780995079.461457 mapwrite 0x035229 thru 0x036c2a (0x1a02 bytes) 000601 1780995079.598595 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36c2b): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c2b): Mirror 14: 0x78c52a28 (stale) Mirror 20: 0x78c52a28 (stale) Mirror 21: 0x78c52a28 (stale) Mirror 22: 0x78c52a28 (stale) CRC-32 checksum value for chunk [0, 0x36c2b): Mirror 1: 0x120b117b Mirror 23: 0x120b117b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c2b: skipped 000603 1780995081.019464 trunc from 0x036c2b to 0x037e8d (0x1263 bytes) 000605 1780995081.199240 mapread 0x005eec thru 0x010fc6 (0xb0db bytes) 000609 1780995081.260983 trunc from 0x037e8e to 0x00c6e3 (0x2b7aa bytes) 000610 1780995081.305857 mapread 0x001410 thru 0x0054da (0x40cb bytes) 000611 1780995081.307024 trunc from 0x00c6e4 to 0x032033 (0x25950 bytes) 000612 1780995081.330309 write 0x03c153 thru 0x03f435 (0x32e3 bytes) 000615 1780995081.336056 read 0x01d7ae thru 0x028c7d (0xb4d0 bytes) 000617 1780995081.356486 mapread 0x01a869 thru 0x01c381 (0x1b19 bytes) 000619 1780995081.369459 mapwrite 0x02a4e4 thru 0x035c7d (0xb79a bytes) 000620 1780995081.520069 read 0x03b67e thru 0x03f435 (0x3db8 bytes) 000623 1780995081.524238 read 0x01ac58 thru 0x01f1d8 (0x4581 bytes) 000624 1780995081.531443 MIRROR_RESYNC 6 mirrors 000626 1780995082.645703 trunc from 0x03f436 to 0x03adc4 (0x4671 bytes) 000630 1780995082.726201 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3adc5): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adc5): Mirror 14: 0x49c1fd4d (stale) Mirror 20: 0x49c1fd4d (stale) Mirror 21: 0x49c1fd4d (stale) Mirror 22: 0x49c1fd4d (stale) Mirror 23: 0x49c1fd4d (stale) CRC-32 checksum value for chunk [0, 0x3adc5): Mirror 1: 0x49c1fd4d Mirror 24: 0x49c1fd4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adc5: skipped 000632 1780995084.294423 write 0x033d8a thru 0x03fb38 (0xbdaf bytes) 000633 1780995084.460247 mapwrite 0x01b1d0 thru 0x025907 (0xa738 bytes) 000635 1780995084.643235 write 0x01d1b8 thru 0x01eb6d (0x19b6 bytes) 000636 1780995084.645605 mapwrite 0x03eb4b thru 0x03ffff (0x14b5 bytes) 000637 1780995084.799373 mapread 0x00355d thru 0x006d86 (0x382a bytes) 000638 1780995084.811185 write 0x028c0f thru 0x02b2fc (0x26ee bytes) 000639 1780995084.831607 write 0x029465 thru 0x02d700 (0x429c bytes) 000640 1780995084.842079 trunc from 0x040000 to 0x012d3b (0x2d2c4 bytes) 000643 1780995084.884500 read 0x00f72b thru 0x012d3b (0x3611 bytes) 000644 1780995084.900529 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12d3c): Mirror 14: 0xcd07c6c9 (stale) Mirror 20: 0xcd07c6c9 (stale) Mirror 22: 0xcd07c6c9 (stale) Mirror 23: 0xcd07c6c9 (stale) Mirror 24: 0xcd07c6c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d3c): Mirror 14: 0xcd07c6c9 (stale) Mirror 20: 0xcd07c6c9 (stale) Mirror 22: 0xcd07c6c9 (stale) Mirror 23: 0xcd07c6c9 (stale) Mirror 24: 0xcd07c6c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d3c: skipped 000646 1780995085.347423 mapwrite 0x0175b2 thru 0x021c0c (0xa65b bytes) 000648 1780995085.444057 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21c0d): Mirror 20: 0x2d57a435 (stale) Mirror 22: 0x2d57a435 (stale) Mirror 23: 0x2d57a435 (stale) Mirror 24: 0x2d57a435 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c0d): Mirror 20: 0x2d57a435 (stale) Mirror 22: 0x2d57a435 (stale) Mirror 23: 0x2d57a435 (stale) Mirror 24: 0x2d57a435 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c0d: skipped 000650 1780995085.930020 trunc from 0x021c0d to 0x0090da (0x18b32 bytes) 000651 1780995085.966976 mapread 0x003b78 thru 0x004503 (0x98c bytes) 000654 1780995085.993338 mapwrite 0x026fe5 thru 0x035355 (0xe371 bytes) 000656 1780995086.133090 write 0x026c56 thru 0x035897 (0xec42 bytes) 000660 1780995086.139028 read 0x02288a thru 0x032420 (0xfb97 bytes) 000661 1780995086.142994 mapread 0x00131d thru 0x010b1c (0xf800 bytes) 000662 1780995086.180338 mapread 0x00b21a thru 0x015850 (0xa637 bytes) 000663 1780995086.181555 mapwrite 0x00fc7a thru 0x011b83 (0x1f0a bytes) 000668 1780995086.293497 mapread 0x0029c2 thru 0x009734 (0x6d73 bytes) 000671 1780995086.299196 write 0x021984 thru 0x025dd4 (0x4451 bytes) 000673 1780995086.302307 mapread 0x008b49 thru 0x00a1ea (0x16a2 bytes) 000674 1780995086.304116 trunc from 0x035898 to 0x022576 (0x13321 bytes) 000675 1780995086.327405 mapread 0x01b6d3 thru 0x022576 (0x6ea4 bytes) 000676 1780995086.340397 mapwrite 0x035a5a thru 0x03b564 (0x5b0b bytes) 000680 1780995086.443756 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b565): Mirror 22: 0x8dfcfa4b (stale) Mirror 23: 0x8dfcfa4b (stale) Mirror 24: 0x8dfcfa4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b565): Mirror 22: 0x8dfcfa4b (stale) Mirror 23: 0x8dfcfa4b (stale) Mirror 24: 0x8dfcfa4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b565: skipped 000681 1780995086.944176 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b565): Mirror 23: 0x8dfcfa4b (stale) Mirror 24: 0x8dfcfa4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b565): Mirror 23: 0x8dfcfa4b (stale) Mirror 24: 0x8dfcfa4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b565: skipped 000682 1780995087.383097 trunc from 0x03b565 to 0x011eed (0x29677 bytes) 000683 1780995087.417194 write 0x03bcc5 thru 0x03ffff (0x433b bytes) 000685 1780995087.419870 write 0x033983 thru 0x0394d1 (0x5b4f bytes) 000686 1780995087.453857 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xc22f07af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xc22f07af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1780995087.829159 write 0x026a1e thru 0x031d92 (0xb375 bytes) 000688 1780995087.843236 trunc from 0x040000 to 0x0066c8 (0x39937 bytes) 000690 1780995087.870798 MIRROR_RESYNC 2 mirrors 000692 1780995088.526820 write 0x000b15 thru 0x0035be (0x2aaa bytes) 000694 1780995088.612941 mapread 0x002b8b thru 0x0066c8 (0x3b3e bytes) 000695 1780995088.628237 mapwrite 0x00476f thru 0x0108ca (0xc15c bytes) 000697 1780995088.737448 write 0x032aff thru 0x03f03e (0xc540 bytes) 000704 1780995088.747049 read 0x037edd thru 0x03f03e (0x7162 bytes) 000705 1780995088.748273 write 0x03df53 thru 0x03f21c (0x12ca bytes) 000706 1780995088.749532 mapread 0x01e12e thru 0x02a13b (0xc00e bytes) 000707 1780995088.765529 read 0x02239a thru 0x02b37c (0x8fe3 bytes) 000709 1780995088.772086 MIRROR_RESYNC 2 mirrors 000710 1780995089.594853 trunc from 0x03f21d to 0x00005c (0x3f1c0 bytes) 000712 1780995089.716035 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000713 1780995089.911491 mapwrite 0x01624b thru 0x01bad3 (0x5889 bytes) 000714 1780995090.118616 trunc from 0x01bad4 to 0x03ea1d (0x22f4a bytes) 000716 1780995090.138306 trunc from 0x03ea1e to 0x02936d (0x156b0 bytes) 000718 1780995090.181948 mapwrite 0x02e4c5 thru 0x03393a (0x5476 bytes) 000720 1780995090.363086 mapread 0x020be9 thru 0x02547a (0x4892 bytes) 000722 1780995090.402378 trunc from 0x03393b to 0x00a46b (0x294cf bytes) 000724 1780995090.428940 mapwrite 0x02ee30 thru 0x0323d0 (0x35a1 bytes) 000727 1780995090.544138 mapwrite 0x008293 thru 0x00ba8b (0x37f9 bytes) 000728 1780995090.615434 trunc from 0x0323d1 to 0x0227dc (0xfbf4 bytes) 000735 1780995090.641448 write 0x02ab84 thru 0x036b2d (0xbfaa bytes) 000736 1780995090.644359 read 0x01fdbb thru 0x024edc (0x5122 bytes) 000738 1780995090.657879 write 0x03717c thru 0x03ffff (0x8e84 bytes) 000740 1780995090.660678 read 0x0187d0 thru 0x026937 (0xe168 bytes) 000741 1780995090.681106 write 0x00dbf7 thru 0x0135d9 (0x59e3 bytes) 000742 1780995090.684512 write 0x039d65 thru 0x03ffff (0x629b bytes) 000743 1780995090.686890 mapwrite 0x03d998 thru 0x03ffff (0x2668 bytes) 000744 1780995090.772174 trunc from 0x040000 to 0x01117b (0x2ee84 bytes) 000745 1780995090.817880 write 0x0071b3 thru 0x008d5d (0x1bab bytes) 000747 1780995090.822456 mapwrite 0x0336d4 thru 0x039217 (0x5b44 bytes) 000751 1780995090.921897 read 0x01e023 thru 0x025a99 (0x7a77 bytes) 000753 1780995090.934053 read 0x0274d3 thru 0x02e61d (0x714b bytes) 000755 1780995090.936077 mapread 0x023b47 thru 0x02e7fe (0xacb8 bytes) 000756 1780995090.937906 write 0x00ba75 thru 0x011eaa (0x6436 bytes) 000757 1780995090.943995 MIRROR_EXTEND to 2 mirrors 000758 1780995091.766024 read 0x02c35a thru 0x0321fb (0x5ea2 bytes) 000759 1780995091.782387 MIRROR_RESYNC 2 mirrors 000760 1780995092.140137 MIRROR_RESYNC 2 mirrors 000765 1780995092.559869 trunc from 0x039218 to 0x01f29a (0x19f7d bytes) 000767 1780995092.686716 mapread 0x00dcdd thru 0x017e5e (0xa182 bytes) 000768 1780995092.727475 read 0x007fc5 thru 0x009c22 (0x1c5e bytes) 000772 1780995092.754090 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000774 1780995092.961916 mapread 0x01c625 thru 0x01f29a (0x2c76 bytes) 000775 1780995092.994454 trunc from 0x01f29b to 0x039c4d (0x1a9b3 bytes) 000776 1780995093.014259 write 0x01e3e6 thru 0x02b4a8 (0xd0c3 bytes) 000779 1780995093.025048 trunc from 0x039c4e to 0x00a66e (0x2f5df bytes) 000781 1780995093.048544 write 0x03ab86 thru 0x03e549 (0x39c4 bytes) 000785 1780995093.053230 read 0x0130a3 thru 0x018ff7 (0x5f55 bytes) 000786 1780995093.062550 write 0x011f89 thru 0x02088e (0xe906 bytes) 000787 1780995093.074979 trunc from 0x03e54a to 0x03a1af (0x439a bytes) 000789 1780995093.097521 trunc from 0x03a1b0 to 0x036d50 (0x345f bytes) 000790 1780995093.114513 mapwrite 0x0394fa thru 0x03ffff (0x6b06 bytes) 000793 1780995093.224030 trunc from 0x040000 to 0x00e484 (0x31b7b bytes) 000795 1780995093.278538 read 0x006ccd thru 0x00e484 (0x77b8 bytes) 000796 1780995093.301403 trunc from 0x00e485 to 0x0154f2 (0x706e bytes) 000797 1780995093.329410 MIRROR_EXTEND to 2 mirrors 000798 1780995094.106107 write 0x0327e6 thru 0x03968f (0x6eaa bytes) 000799 1780995094.179082 mapwrite 0x02ba0d thru 0x02c862 (0xe56 bytes) 000801 1780995094.251862 write 0x00165a thru 0x010fb8 (0xf95f bytes) 000802 1780995094.284730 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39690): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39690): Mirror 2: 0x90586a10 (stale) CRC-32 checksum value for chunk [0, 0x39690): Mirror 1: 0x1b589bd5 Mirror 3: 0x1b589bd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39690: skipped 000803 1780995095.274136 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39690): Mirror 2: 0x90586a10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39690): Mirror 2: 0x90586a10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39690: skipped 000804 1780995095.673374 MIRROR_RESYNC 2 mirrors 000806 1780995096.576173 trunc from 0x039690 to 0x01f5b9 (0x1a0d6 bytes) 000808 1780995096.748127 read 0x00896a thru 0x0187a4 (0xfe3b bytes) 000810 1780995096.771880 mapwrite 0x03ed21 thru 0x03ffff (0x12df bytes) 000812 1780995096.880058 trunc from 0x040000 to 0x001d7f (0x3e280 bytes) 000813 1780995096.906421 mapread 0x00021b thru 0x001d7f (0x1b65 bytes) 000814 1780995096.914450 write 0x022a03 thru 0x02cba8 (0xa1a6 bytes) 000815 1780995096.917220 mapread 0x023814 thru 0x02cba8 (0x9395 bytes) 000817 1780995096.923372 write 0x0271bb thru 0x02b5d4 (0x441a bytes) 000819 1780995096.926535 mapwrite 0x03d7ab thru 0x03ffff (0x2855 bytes) 000821 1780995097.011237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb5b36e06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ea8ed32 Mirror 3: 0x7ea8ed32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000822 1780995097.930514 read 0x03f879 thru 0x03ffff (0x787 bytes) 000824 1780995097.977481 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb5b36e06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ea8ed32 Mirror 3: 0x7ea8ed32 Mirror 4: 0x7ea8ed32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1780995098.929041 trunc from 0x040000 to 0x03efd5 (0x102a bytes) 000826 1780995099.037268 MIRROR_RESYNC 4 mirrors 000828 1780995100.033034 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) 000829 1780995100.501379 write 0x00deda thru 0x012bcf (0x4cf6 bytes) 000830 1780995100.605197 read 0x0397fd thru 0x03efd5 (0x57d9 bytes) 000831 1780995100.617602 trunc from 0x03efd6 to 0x03f3d0 (0x3fb bytes) 000832 1780995100.649307 mapread 0x02d3df thru 0x036de6 (0x9a08 bytes) 000834 1780995100.680956 mapwrite 0x035979 thru 0x03e85a (0x8ee2 bytes) 000837 1780995100.806402 mapread 0x001f74 thru 0x00e81c (0xc8a9 bytes) 000838 1780995100.820877 read 0x030eaa thru 0x0352c0 (0x4417 bytes) 000839 1780995100.821893 write 0x039c01 thru 0x03ffff (0x63ff bytes) 000840 1780995100.823119 write 0x00df98 thru 0x01523b (0x72a4 bytes) 000841 1780995100.839608 read 0x031f6f thru 0x035557 (0x35e9 bytes) 000842 1780995100.849402 MIRROR_RESYNC 3 mirrors 000845 1780995101.664627 write 0x01c5ee thru 0x028807 (0xc21a bytes) 000848 1780995101.782622 write 0x01fdb3 thru 0x025145 (0x5393 bytes) 000849 1780995101.784455 write 0x001e3a thru 0x00cb3e (0xad05 bytes) 000851 1780995101.800338 mapread 0x01ecb2 thru 0x027212 (0x8561 bytes) 000854 1780995101.803526 mapread 0x01be25 thru 0x01d52d (0x1709 bytes) 000855 1780995101.816332 trunc from 0x040000 to 0x0031b3 (0x3ce4c bytes) 000857 1780995101.845115 trunc from 0x0031b4 to 0x000304 (0x2eaf bytes) 000858 1780995101.871101 read 0x000230 thru 0x000304 (0xd5 bytes) 000861 1780995101.888137 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x305): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x305): Mirror 3: 0xd7952687 (stale) Mirror 4: 0xd7952687 (stale) CRC-32 checksum value for chunk [0, 0x305): Mirror 1: 0xd7952687 Mirror 5: 0xd7952687 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x305: skipped 000862 1780995102.775624 MIRROR_RESYNC 4 mirrors 000863 1780995103.516268 trunc from 0x000305 to 0x00c887 (0xc583 bytes) 000864 1780995103.611230 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xc888): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc888): Mirror 3: 0x62ae7390 (stale) Mirror 4: 0x62ae7390 (stale) Mirror 5: 0x62ae7390 (stale) CRC-32 checksum value for chunk [0, 0xc888): Mirror 1: 0x62ae7390 Mirror 6: 0x62ae7390 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc888: skipped 000865 1780995104.554199 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc888): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc888): Mirror 3: 0x62ae7390 (stale) Mirror 4: 0x62ae7390 (stale) Mirror 5: 0x62ae7390 (stale) CRC-32 checksum value for chunk [0, 0xc888): Mirror 1: 0x62ae7390 Mirror 6: 0x62ae7390 Mirror 7: 0x62ae7390 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc888: skipped 000866 1780995105.683899 read 0x008408 thru 0x00c887 (0x4480 bytes) 000867 1780995105.730608 trunc from 0x00c888 to 0x028047 (0x1b7c0 bytes) 000868 1780995105.830285 mapwrite 0x0011d4 thru 0x0069ff (0x582c bytes) 000869 1780995105.939179 read 0x010f28 thru 0x0119bf (0xa98 bytes) 000870 1780995105.957408 trunc from 0x028048 to 0x03dd47 (0x15d00 bytes) 000871 1780995106.001141 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 3: 0xddfba08a (stale) Mirror 4: 0xddfba08a (stale) Mirror 5: 0xddfba08a (stale) Mirror 6: 0xddfba08a (stale) Mirror 7: 0xddfba08a (stale) CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 1: 0x8c4d783b Mirror 8: 0x8c4d783b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd48: skipped 000872 1780995107.269274 mapread 0x0369e7 thru 0x03dd47 (0x7361 bytes) 000873 1780995107.370084 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 3: 0xddfba08a (stale) Mirror 4: 0xddfba08a (stale) Mirror 6: 0xddfba08a (stale) Mirror 7: 0xddfba08a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 3: 0xddfba08a (stale) Mirror 4: 0xddfba08a (stale) Mirror 6: 0xddfba08a (stale) Mirror 7: 0xddfba08a (stale) CRC-32 checksum value for chunk [0, 0x3dd48): Mirror 1: 0x8c4d783b Mirror 8: 0x8c4d783b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd48: skipped 000874 1780995108.501391 MIRROR_RESYNC 6 mirrors 000876 1780995109.748885 write 0x026476 thru 0x031e45 (0xb9d0 bytes) 000877 1780995109.906774 write 0x037d80 thru 0x03ffff (0x8280 bytes) 000878 1780995109.922825 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3117d38a (stale) Mirror 4: 0x3117d38a (stale) Mirror 6: 0x3117d38a (stale) Mirror 8: 0x3117d38a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3117d38a (stale) Mirror 4: 0x3117d38a (stale) Mirror 6: 0x3117d38a (stale) Mirror 8: 0x3117d38a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000879 1780995110.504494 mapwrite 0x006bb1 thru 0x014090 (0xd4e0 bytes) 000880 1780995110.635520 trunc from 0x040000 to 0x014477 (0x2bb88 bytes) 000881 1780995110.657969 read 0x0104f1 thru 0x0123a7 (0x1eb7 bytes) 000883 1780995110.666585 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14478): Mirror 4: 0x6a570f29 (stale) Mirror 6: 0x6a570f29 (stale) Mirror 8: 0x6a570f29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14478): Mirror 4: 0x6a570f29 (stale) Mirror 6: 0x6a570f29 (stale) Mirror 8: 0x6a570f29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14478: skipped 000885 1780995111.081181 write 0x033459 thru 0x036cdd (0x3885 bytes) 000886 1780995111.082639 mapwrite 0x019b11 thru 0x024505 (0xa9f5 bytes) 000889 1780995111.206926 read 0x01871a thru 0x01f744 (0x702b bytes) 000890 1780995111.208829 mapread 0x017667 thru 0x027176 (0xfb10 bytes) 000892 1780995111.212094 mapread 0x001908 thru 0x008266 (0x695f bytes) 000893 1780995111.229158 read 0x0142f3 thru 0x020f0e (0xcc1c bytes) 000894 1780995111.236136 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36cde): Mirror 6: 0xbbe39d13 (stale) Mirror 8: 0xbbe39d13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cde): Mirror 6: 0xbbe39d13 (stale) Mirror 8: 0xbbe39d13 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cde: skipped 000897 1780995111.708910 trunc from 0x036cde to 0x0288a1 (0xe43c bytes) 000899 1780995111.756437 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x288a2): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288a2): Mirror 6: 0x9b568405 (stale) Mirror 8: 0x9b568405 (stale) CRC-32 checksum value for chunk [0, 0x288a2): Mirror 1: 0x461f04f8 Mirror 9: 0x461f04f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288a2: skipped 000900 1780995112.861749 mapread 0x01fc56 thru 0x0253ff (0x57aa bytes) 000903 1780995112.936269 trunc from 0x0288a2 to 0x02f5c6 (0x6d25 bytes) 000905 1780995113.092958 read 0x02734d thru 0x02f5c6 (0x827a bytes) 000906 1780995113.106395 read 0x004881 thru 0x005db5 (0x1535 bytes) 000911 1780995113.156317 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f5c7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5c7): Mirror 6: 0x7b9942a4 (stale) Mirror 8: 0x7b9942a4 (stale) Mirror 9: 0xe8488461 (stale) CRC-32 checksum value for chunk [0, 0x2f5c7): Mirror 1: 0xe8488461 Mirror 10: 0xe8488461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5c7: skipped 000912 1780995114.260994 mapread 0x015e81 thru 0x0224ed (0xc66d bytes) 000913 1780995114.322595 mapread 0x01c83a thru 0x02a4b7 (0xdc7e bytes) 000914 1780995114.358280 MIRROR_RESYNC 5 mirrors 000916 1780995115.595522 read 0x0101eb thru 0x018d28 (0x8b3e bytes) 000917 1780995115.610319 read 0x01b820 thru 0x0269c9 (0xb1aa bytes) 000918 1780995115.640098 trunc from 0x02f5c7 to 0x0362b4 (0x6cee bytes) 000919 1780995115.835785 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x362b5): Mirror 6: 0xffa0d79e (stale) Mirror 8: 0xffa0d79e (stale) Mirror 9: 0xffa0d79e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x362b5): Mirror 6: 0xffa0d79e (stale) Mirror 8: 0xffa0d79e (stale) Mirror 9: 0xffa0d79e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x362b5: skipped 000924 1780995116.236048 MIRROR_RESYNC 4 mirrors 000925 1780995117.347730 read 0x031707 thru 0x0362b4 (0x4bae bytes) 000926 1780995117.365398 write 0x011f6c thru 0x01a654 (0x86e9 bytes) 000928 1780995117.521179 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x362b5): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x362b5): Mirror 6: 0xffa0d79e (stale) Mirror 8: 0xffa0d79e (stale) Mirror 9: 0xffa0d79e (stale) CRC-32 checksum value for chunk [0, 0x362b5): Mirror 1: 0x6ffa3459 Mirror 10: 0x6ffa3459 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x362b5: skipped 000929 1780995118.758573 mapwrite 0x011a10 thru 0x020fca (0xf5bb bytes) 000930 1780995118.968017 mapread 0x029d8a thru 0x03480a (0xaa81 bytes) 000931 1780995118.992302 write 0x03636e thru 0x03ab74 (0x4807 bytes) 000933 1780995118.996006 mapwrite 0x01f9a6 thru 0x02b8f7 (0xbf52 bytes) 000934 1780995119.065749 read 0x01cee6 thru 0x0296b4 (0xc7cf bytes) 000937 1780995119.067782 mapwrite 0x02ae14 thru 0x037d4a (0xcf37 bytes) 000938 1780995119.158542 mapread 0x02ed15 thru 0x032e7c (0x4168 bytes) 000939 1780995119.167973 MIRROR_RESYNC 5 mirrors 000943 1780995120.554767 write 0x000290 thru 0x001ad4 (0x1845 bytes) 000944 1780995120.730352 write 0x005cd6 thru 0x0078dc (0x1c07 bytes) 000946 1780995120.754235 write 0x03da00 thru 0x03ffff (0x2600 bytes) 000947 1780995120.755437 write 0x02f2ee thru 0x03e8bb (0xf5ce bytes) 000948 1780995120.794970 mapread 0x0281e1 thru 0x0338e8 (0xb708 bytes) 000949 1780995120.835359 write 0x02163b thru 0x0236c4 (0x208a bytes) 000950 1780995120.857925 trunc from 0x040000 to 0x00d481 (0x32b7e bytes) 000951 1780995120.912394 mapwrite 0x027f6e thru 0x033310 (0xb3a3 bytes) 000955 1780995121.129368 trunc from 0x033311 to 0x033a4c (0x73c bytes) 000956 1780995121.195265 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a4d): Mirror 8: 0xaf1bdcbc (stale) Mirror 9: 0xaf1bdcbc (stale) Mirror 10: 0xaf1bdcbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a4d): Mirror 8: 0xaf1bdcbc (stale) Mirror 9: 0xaf1bdcbc (stale) Mirror 10: 0xaf1bdcbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a4d: skipped 000957 1780995121.782192 mapread 0x01d72d thru 0x020f63 (0x3837 bytes) 000958 1780995121.852439 read 0x013652 thru 0x016f4e (0x38fd bytes) 000959 1780995121.860403 write 0x03fc0b thru 0x03ffff (0x3f5 bytes) 000960 1780995121.862626 read 0x0152d0 thru 0x02526d (0xff9e bytes) 000963 1780995121.881308 read 0x0232c6 thru 0x02dbbb (0xa8f6 bytes) 000965 1780995121.883631 mapwrite 0x026702 thru 0x029fca (0x38c9 bytes) 000970 1780995122.031039 write 0x02b2d3 thru 0x02f858 (0x4586 bytes) 000971 1780995122.041043 mapwrite 0x0187ed thru 0x01dddb (0x55ef bytes) 000972 1780995122.134060 MIRROR_RESYNC 4 mirrors 000973 1780995123.477873 read 0x03b843 thru 0x03ffff (0x47bd bytes) 000974 1780995123.491123 read 0x0174b5 thru 0x01af9b (0x3ae7 bytes) 000975 1780995123.519587 write 0x03eda3 thru 0x03ffff (0x125d bytes) 000977 1780995123.711513 write 0x0268d6 thru 0x02d95f (0x708a bytes) 000978 1780995123.744420 mapread 0x0369c7 thru 0x039f89 (0x35c3 bytes) 000980 1780995123.779098 mapread 0x022481 thru 0x0295be (0x713e bytes) 000987 1780995123.812606 mapwrite 0x01beed thru 0x028299 (0xc3ad bytes) 000988 1780995123.942618 read 0x00b26d thru 0x014bf9 (0x998d bytes) 000989 1780995123.963919 mapread 0x0034f7 thru 0x006a9f (0x35a9 bytes) 000990 1780995123.965563 trunc from 0x040000 to 0x0017b1 (0x3e84e bytes) 000991 1780995124.048839 trunc from 0x0017b2 to 0x039a3d (0x3828c bytes) 000992 1780995124.067927 mapread 0x00ebe4 thru 0x010a1d (0x1e3a bytes) 000996 1780995124.093891 mapwrite 0x019314 thru 0x026731 (0xd41e bytes) 000997 1780995124.266705 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a3e): Mirror 8: 0xe436f695 (stale) Mirror 9: 0xe436f695 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a3e): Mirror 8: 0xe436f695 (stale) Mirror 9: 0xe436f695 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a3e: skipped All operations completed A-OK! PASS 70a (136s) == sanity-flr test 71: check mirror extend parent fid ==== 04:52:10 (1780995130) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 04:52:14 (1780995134) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309217 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226406 s, 4.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 04:52:23 (1780995143) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 640512 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2978080 .. failed Extending file size to 1679744 .. Extending file size to 8754336 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6827232 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8868384 .. Extending file size to 6971040 .. Extending file size to 8072288 .. Extending file size to 8443392 .. Extending file size to 8639872 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56712 56714 56716 56717 56719 done 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) 192.168.204.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:53:51 (1780995231) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 7145216 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5985344 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7794624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8356768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8496544 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8405504 .. Extending file size to 8501056 .. Extending file size to 8727616 .. Extending file size to 8748000 .. Extending file size to 8500992 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9168192 .. Waiting 58129 58131 58133 58135 58136 done 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) 192.168.204.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:55:09 (1780995309) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175596 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.513107 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4628dc8f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:55:31 (1780995331) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201506 s, 10.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 04:55:39 (1780995339) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:55:48 (1780995348) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183155 s, 11.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:55:59 (1780995359) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221537 s, 9.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:56:07 (1780995367) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286959 s, 7.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:56:41 (1780995401) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163545 s, 6.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:57:45 (1780995465) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.465691 s, 4.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:58:46 (1780995526) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. Creating new pool test_204f oleg425-server: Pool lustre.test_204f created Adding targets to pool oleg425-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg425-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.464024 s, 4.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg425-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg425-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg425-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:00:10 (1780995610) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:00:21 (1780995621) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:00:31 (1780995631) oleg425-server: Pool lustre.test_206 created Adding targets to pool oleg425-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg425-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg425-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg425-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg425-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:00:56 (1780995656) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117653 s, 8.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392341 s, 26.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x54300c43 Mirror 3: 0x54300c43 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:01:07 (1780995667) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:01:11 (1780995671) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:01:14 (1780995674) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.109202 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2620000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xed0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xed0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xed0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe10000 Read this (4 bytes): BBBB 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:01:57 (1780995717) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0183721 s, 446 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:02:04 (1780995724) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427524 s, 24.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x219 lmm_fid: [0x200000404:0x219:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834457 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 05:02:06.000000000 -0400 Modify: 2026-06-09 05:02:06.000000000 -0400 Change: 2026-06-09 05:02:08.000000000 -0400 Birth: 2026-06-09 05:02:06.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:02:13 (1780995733) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:02:16 (1780995736) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.687497 s, 59.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.109894 s, 37.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 9 05:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 9 05:02 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:02:27 (1780995747) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:02:35 (1780995755) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1875 sec =========== 05:02:44 (1780995764) === sanity-flr: start cleanup 05:02:45 (1780995765) === === sanity-flr: finish cleanup 05:02:52 (1780995772) ===