-----============= acceptance-small: sanity-flr ============----- Sat Sep 12 22:07:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg457-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:07:24 (1789265244) === oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b7484a2a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b7484a2a000.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:08:03 (1789265283) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanity-flr test 0a: lfs mirror create with -N option == 22:08:06 (1789265286) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:08:25 (1789265305) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:08:28 (1789265308) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:08:31 (1789265311) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:09:01 (1789265341) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:09:05 (1789265345) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:09:09 (1789265349) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b7484a2a000.stats=clear osc.lustre-OST0001-osc-ffff9b7484a2a000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:09:16 (1789265356) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:09:28 (1789265368) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:09:35 (1789265375) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:09:39 (1789265379) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:09:48 (1789265388) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 22:09:57 (1789265397) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:10:05 (1789265405) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:10:08 (1789265408) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:10:15 (1789265415) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:10:25 (1789265425) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:10:31 (1789265431) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.437604 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123451 s, 8.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:10:41 (1789265441) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.439576 s, 40.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:10:50 (1789265450) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:11:02 (1789265462) Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:12:10 (1789265530) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.244888 s, 17.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:12:50 (1789265570) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:14:13 (1789265653) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.35303 s, 9.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 12 22:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.985s user 0m0.001s sys 0m0.202s Stopping /mnt/lustre-ost1 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.408s user 0m0.000s sys 0m0.215s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg457-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg457-server oleg457-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.406s user 0m0.000s sys 0m0.197s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg457-server: oleg457-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg457-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7484a2a000.ost_server_uuid in FULL state after 0 sec PASS 33b (107s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:16:00 (1789265760) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:16:04 (1789265764) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:16:08 (1789265768) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:16:12 (1789265772) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:16:19 (1789265779) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.76438 s, 16.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:16:32 (1789265792) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223908 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031232 s, 33.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:16:40 (1789265800) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:16:49 (1789265809) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134759 s, 760 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00276254 s, 371 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00288161 s, 355 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00246499 s, 415 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 22:17:00 (1789265820) create test files with size 15746607 .. 15842b6e904642c2b73d34b2bb429949 - 60e5e350e7fdd5fa4dfd4ba35f4db496 - 04ac9ebe829be001d4ca7c8daa27e028 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 22:17:29 (1789265849) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:17:46 (1789265866) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:17:53 (1789265873) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967727 s, 10.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028633 s, 36.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 22:18:01 (1789265881) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206263 s, 10.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.367239 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20701 s, 10.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:18:18 (1789265898) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:18:21 (1789265901) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:18:25 (1789265905) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432819 s, 24.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:18:33 (1789265913) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:18:37 (1789265917) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:18:48 (1789265928) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312784 s, 33.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:18:59 (1789265939) llite.lustre-ffff9b7482e1d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.460049 s, 22.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b7492f51800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:19:15 (1789265955) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:19:58 (1789265998) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:20:38 (1789266038) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:20:42 (1789266042) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228726 s, 13.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.117423 s, 26.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa82dd779 (nosync,stale) Mirror 2: 0xa82dd779 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 97403ee676a93c4f0d6aa533b82b6af6 - mirror 1 checksum: 7d6663ddbaf982e89ffd861be04d25d7 - mirror 2 checksum: 97403ee676a93c4f0d6aa533b82b6af6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:20:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:20:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 97403ee676a93c4f0d6aa533b82b6af6 - mirror 1 checksum: 7d6663ddbaf982e89ffd861be04d25d7 - mirror 2 checksum: 7d6663ddbaf982e89ffd861be04d25d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:20:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:20:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:20:55 (1789266055) Failing mds1 on oleg457-server Stopping /mnt/lustre-mds1 (opts:) on oleg457-server 22:21:03 (1789266063) shut down facet: mds1 facet_host: oleg457-server facet_failover_host: oleg457-server Failover mds1 to oleg457-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 22:21:39 (1789266099) targets are mounted 22:21:39 (1789266099) facet_failover done oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319017 s, 32.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:21:59 (1789266119) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:22:02 (1789266122) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:22:06 (1789266126) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00689098 s, 2.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:22:14 (1789266134) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:22:18 (1789266138) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00650694 s, 629 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0695837 s, 58.9 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:22:26 (1789266146) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:22:52 (1789266172) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:23:20 (1789266200) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:23:42 (1789266222) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:24:05 (1789266245) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:24:13 (1789266253) mirror create pid 54340 mirror split pid 54342 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54340 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54342 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 22:25:21 (1789266321) Seed set to 6323 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000003 1789266324.502784 write 0x02356d thru 0x031619 (0xe0ad bytes) 000004 1789266324.524360 write 0x031b4a thru 0x03b6c3 (0x9b7a bytes) 000006 1789266324.618050 read 0x01be75 thru 0x0260d8 (0xa264 bytes) 000007 1789266324.658050 read 0x00a700 thru 0x0108eb (0x61ec bytes) 000008 1789266324.730726 read 0x0043d4 thru 0x00b651 (0x727e bytes) truncating to largest ever: 0x28df2 000010 1789266324.731855 trunc from 0x03b6c4 to 0x028df1 (0x128d2 bytes) 000011 1789266324.790995 write 0x03a6cc thru 0x03ffff (0x5934 bytes) 000012 1789266324.816107 mapread 0x02cc6d thru 0x030f3c (0x42d0 bytes) 000013 1789266324.849133 write 0x01c2ed thru 0x022853 (0x6567 bytes) 000014 1789266324.887368 punch from 0x029c66 to 0x02aeeb (0x1286 bytes) 000017 1789266324.944827 read 0x006a74 thru 0x0141aa (0xd737 bytes) 000018 1789266324.983255 MIRROR_EXTEND to 2 mirrors 000019 1789266326.798490 punch from 0x015d2a to 0x01ea4e (0x8d25 bytes) 000020 1789266326.963531 MIRROR_RESYNC 2 mirrors 000021 1789266328.186727 trunc from 0x040000 to 0x021568 (0x1ea97 bytes) 000024 1789266328.443144 write 0x025584 thru 0x030ca8 (0xb725 bytes) 000025 1789266328.456678 read 0x02efad thru 0x030ca8 (0x1cfc bytes) 000027 1789266328.464575 write 0x0312f0 thru 0x035a66 (0x4777 bytes) 000032 1789266328.501318 punch from 0x016ffb to 0x01bc59 (0x4c5f bytes) 000033 1789266328.534568 trunc from 0x035a67 to 0x011081 (0x249e5 bytes) 000035 1789266328.589620 mapwrite 0x01e119 thru 0x02d60e (0xf4f6 bytes) 000038 1789266328.847847 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: 0x2615b4e2 (stale) CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 1: 0xa1428f47 Mirror 3: 0xa1428f47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60f: skipped 000039 1789266330.520904 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: 0x2615b4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: 0x2615b4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60f: skipped 000040 1789266331.056318 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 2: 0x2615b4e2 (stale) CRC-32 checksum value for chunk [0, 0x2d60f): Mirror 1: 0xa1428f47 Mirror 3: 0xa1428f47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60f: skipped 000042 1789266332.706225 mapwrite 0x029942 thru 0x03352a (0x9be9 bytes) 000044 1789266333.130815 MIRROR_RESYNC 3 mirrors 000045 1789266334.630496 read 0x004f01 thru 0x00e774 (0x9874 bytes) 000046 1789266334.650080 read 0x009130 thru 0x00f9d0 (0x68a1 bytes) 000047 1789266334.676096 write 0x03cd40 thru 0x03ffff (0x32c0 bytes) 000048 1789266334.778327 punch from 0x032d09 to 0x03d6a3 (0xa99b bytes) 000049 1789266334.796024 write 0x028cb6 thru 0x035793 (0xcade bytes) 000050 1789266334.815813 write 0x013c50 thru 0x022737 (0xeae8 bytes) 000052 1789266334.839258 read 0x03970d thru 0x03ffff (0x68f3 bytes) 000054 1789266334.846052 mapread 0x03da9b thru 0x03ffff (0x2565 bytes) 000055 1789266334.850125 trunc from 0x040000 to 0x011e47 (0x2e1b8 bytes) 000058 1789266334.883355 punch from 0x00d8c3 to 0x011e47 (0x4585 bytes) 000059 1789266334.893373 mapread 0x007f8c thru 0x00b456 (0x34cb bytes) 000060 1789266334.909856 mapread 0x002871 thru 0x0091db (0x696b bytes) 000061 1789266334.915148 mapread 0x00c295 thru 0x011e47 (0x5bb3 bytes) 000062 1789266334.918686 trunc from 0x011e48 to 0x01af4a (0x9103 bytes) 000065 1789266334.939806 write 0x01663f thru 0x02514c (0xeb0e bytes) 000068 1789266334.987972 read 0x01e381 thru 0x021ee3 (0x3b63 bytes) 000070 1789266334.989568 read 0x011af6 thru 0x01d2bc (0xb7c7 bytes) 000071 1789266334.994623 punch from 0x00859c to 0x00de36 (0x589b bytes) 000072 1789266335.050890 mapwrite 0x030405 thru 0x0389c8 (0x85c4 bytes) 000074 1789266335.283498 read 0x01875b thru 0x024271 (0xbb17 bytes) 000076 1789266335.285511 mapread 0x009ccb thru 0x00ed62 (0x5098 bytes) 000077 1789266335.290240 write 0x03c2f5 thru 0x03ffff (0x3d0b bytes) 000078 1789266335.294805 punch from 0x0344f8 to 0x037eb3 (0x39bc bytes) truncating to largest ever: 0x2a210 000079 1789266335.339714 trunc from 0x040000 to 0x02a20f (0x15df0 bytes) 000080 1789266335.388028 punch from 0x00fc30 to 0x016da9 (0x717a bytes) 000081 1789266335.396730 punch from 0x024d9a to 0x02a20f (0x5476 bytes) 000082 1789266335.407335 read 0x0124f2 thru 0x020997 (0xe4a6 bytes) 000083 1789266335.418260 write 0x027bc6 thru 0x02825f (0x69a bytes) 000084 1789266335.422029 mapread 0x010b40 thru 0x010d24 (0x1e5 bytes) 000086 1789266335.426038 trunc from 0x02a210 to 0x012399 (0x17e76 bytes) 000087 1789266335.489677 MIRROR_RESYNC 3 mirrors 000088 1789266337.276195 mapread 0x011e87 thru 0x012399 (0x513 bytes) truncating to largest ever: 0x2ae91 000089 1789266337.291644 trunc from 0x01239a to 0x02ae90 (0x18af7 bytes) 000090 1789266337.448066 write 0x01b875 thru 0x026151 (0xa8dd bytes) 000092 1789266337.467697 punch from 0x00c49a to 0x011d21 (0x5888 bytes) 000093 1789266337.492043 mapwrite 0x01a3f9 thru 0x0214b9 (0x70c1 bytes) 000096 1789266337.573009 mapwrite 0x014a0e thru 0x020c60 (0xc253 bytes) 000097 1789266337.667982 punch from 0x01a6ea to 0x01c4cb (0x1de2 bytes) 000098 1789266337.680261 trunc from 0x02ae91 to 0x02a229 (0xc67 bytes) 000099 1789266337.722448 mapread 0x018c3e thru 0x026424 (0xd7e7 bytes) 000100 1789266337.737617 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a22a): Mirror 3: 0xfbec6786 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a22a): Mirror 3: 0xfbec6786 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a22a: skipped 000101 1789266338.230858 write 0x01ba35 thru 0x023741 (0x7d0d bytes) 000102 1789266338.247846 mapwrite 0x00f15b thru 0x01d7c3 (0xe669 bytes) 000104 1789266338.409273 punch from 0x01c3f0 to 0x02a229 (0xde3a bytes) 000105 1789266338.420941 write 0x027072 thru 0x02d3e1 (0x6370 bytes) 000106 1789266338.439493 mapread 0x02badb thru 0x02d3e1 (0x1907 bytes) 000107 1789266338.440963 mapread 0x0099fa thru 0x0155c3 (0xbbca bytes) 000108 1789266338.467705 MIRROR_RESYNC 2 mirrors 000109 1789266339.675603 trunc from 0x02d3e2 to 0x005f5f (0x27482 bytes) 000111 1789266339.786886 write 0x0328e9 thru 0x03bc5d (0x9375 bytes) 000112 1789266339.792796 read 0x002751 thru 0x00bb6d (0x941d bytes) 000113 1789266339.812132 mapwrite 0x024592 thru 0x03324b (0xecba bytes) 000114 1789266339.916372 mapread 0x00fb35 thru 0x010016 (0x4e2 bytes) 000115 1789266339.927917 punch from 0x007973 to 0x0122eb (0xa979 bytes) truncating to largest ever: 0x3690a 000116 1789266339.933940 trunc from 0x03bc5e to 0x036909 (0x5354 bytes) 000117 1789266339.965840 punch from 0x0103c7 to 0x016986 (0x65c0 bytes) 000118 1789266339.973632 write 0x02052d thru 0x0239ad (0x3481 bytes) 000120 1789266339.975373 write 0x025711 thru 0x0340ce (0xe9be bytes) 000121 1789266339.981736 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000122 1789266340.227279 MIRROR_EXTEND to 2 mirrors 000125 1789266341.477118 write 0x004be6 thru 0x0083b6 (0x37d1 bytes) 000126 1789266341.565177 write 0x009483 thru 0x00abe6 (0x1764 bytes) 000127 1789266341.576790 punch from 0x00abcf to 0x0105cc (0x59fe bytes) 000130 1789266341.592715 punch from 0x0109da to 0x01f92d (0xef54 bytes) 000131 1789266341.597331 punch from 0x00f063 to 0x01d9f8 (0xe996 bytes) 000132 1789266341.601828 write 0x03fcff thru 0x03ffff (0x301 bytes) 000133 1789266341.607753 MIRROR_RESYNC 2 mirrors 000134 1789266342.692728 MIRROR_EXTEND to 3 mirrors 000135 1789266343.866726 MIRROR_EXTEND to 4 mirrors 000136 1789266344.713986 read 0x038a7e thru 0x03c423 (0x39a6 bytes) 000138 1789266344.729126 trunc from 0x040000 to 0x030287 (0xfd78 bytes) 000140 1789266344.852811 mapread 0x005796 thru 0x0140cc (0xe937 bytes) 000141 1789266344.881707 punch from 0x001d07 to 0x01046d (0xe767 bytes) 000142 1789266344.889993 punch from 0x02369c to 0x0297a2 (0x6107 bytes) 000143 1789266344.897800 read 0x00965c thru 0x00985c (0x201 bytes) 000144 1789266344.926950 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30288): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30288): Mirror 2: 0xbecf5cad (stale) Mirror 3: 0xbecf5cad (stale) Mirror 4: 0xbecf5cad (stale) CRC-32 checksum value for chunk [0, 0x30288): Mirror 1: 0x3eeb9c11 Mirror 5: 0x3eeb9c11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30288: skipped 000146 1789266346.094347 mapwrite 0x010404 thru 0x01aff8 (0xabf5 bytes) 000147 1789266346.360312 write 0x016512 thru 0x01aa72 (0x4561 bytes) 000148 1789266346.362673 read 0x0013e9 thru 0x001c69 (0x881 bytes) 000150 1789266346.379449 mapwrite 0x000741 thru 0x00bc87 (0xb547 bytes) 000154 1789266346.489166 punch from 0x01867c to 0x01889d (0x222 bytes) 000155 1789266346.499045 write 0x0365b9 thru 0x03ffff (0x9a47 bytes) 000156 1789266346.518672 write 0x0175c8 thru 0x0227f5 (0xb22e bytes) truncating to largest ever: 0x3c3a4 000157 1789266346.523012 trunc from 0x040000 to 0x03c3a3 (0x3c5c bytes) 000158 1789266346.555527 trunc from 0x03c3a4 to 0x02fe5e (0xc545 bytes) 000159 1789266346.585571 punch from 0x01c216 to 0x0210de (0x4ec9 bytes) 000161 1789266346.615090 read 0x00cc7f thru 0x00ccc8 (0x4a bytes) 000162 1789266346.616021 read 0x01f24f thru 0x025047 (0x5df9 bytes) 000163 1789266346.629822 write 0x01e2db thru 0x02ae79 (0xcb9f bytes) 000164 1789266346.636573 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fe5f): Mirror 3: 0xbd1bee8d (stale) Mirror 4: 0xbd1bee8d (stale) Mirror 5: 0xe8cbe5d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fe5f): Mirror 3: 0xbd1bee8d (stale) Mirror 4: 0xbd1bee8d (stale) Mirror 5: 0xe8cbe5d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fe5f: skipped 000166 1789266347.098355 read 0x0144dd thru 0x01bf4b (0x7a6f bytes) 000167 1789266347.110905 trunc from 0x02fe5f to 0x028aa0 (0x73be bytes) 000169 1789266347.143102 MIRROR_RESYNC 4 mirrors 000170 1789266348.074641 punch from 0x01296e to 0x018e40 (0x64d3 bytes) 000171 1789266348.153947 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28aa1): Mirror 3: 0xbf2b55da (stale) Mirror 4: 0xbf2b55da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28aa1): Mirror 3: 0xbf2b55da (stale) Mirror 4: 0xbf2b55da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28aa1: skipped 000172 1789266348.570571 read 0x01344e thru 0x014c30 (0x17e3 bytes) 000174 1789266348.589238 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28aa1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28aa1): Mirror 3: 0xbf2b55da (stale) Mirror 4: 0xbf2b55da (stale) CRC-32 checksum value for chunk [0, 0x28aa1): Mirror 1: 0x2d578725 Mirror 5: 0x2d578725 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28aa1: skipped 000175 1789266349.628699 write 0x00b039 thru 0x00c7db (0x17a3 bytes) 000176 1789266349.716248 trunc from 0x028aa1 to 0x036934 (0xde94 bytes) 000177 1789266349.742308 read 0x028bbb thru 0x030118 (0x755e bytes) 000178 1789266349.760328 mapread 0x028d35 thru 0x0332fb (0xa5c7 bytes) 000179 1789266349.802731 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36935): Mirror 4: 0x6e287b9c (stale) Mirror 5: 0x4b1ec1aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36935): Mirror 4: 0x6e287b9c (stale) Mirror 5: 0x4b1ec1aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36935: skipped 000180 1789266350.231689 punch from 0x00c34f to 0x01b16d (0xee1f bytes) 000181 1789266350.236024 write 0x0069ae thru 0x013182 (0xc7d5 bytes) 000182 1789266350.248356 write 0x03645c thru 0x03ffff (0x9ba4 bytes) 000183 1789266350.265901 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9e2f473e (stale) Mirror 5: 0x9771af11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85f94b8 Mirror 6: 0x85f94b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000187 1789266351.409845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9e2f473e (stale) Mirror 5: 0x9771af11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85f94b8 Mirror 6: 0x85f94b8 Mirror 7: 0x85f94b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000190 1789266352.493327 read 0x00947d thru 0x00da6f (0x45f3 bytes) 000192 1789266352.531823 punch from 0x02e15a to 0x0332a4 (0x514b bytes) 000193 1789266352.675597 punch from 0x0141c5 to 0x01a9cf (0x680b bytes) 000194 1789266352.681884 read 0x009ed3 thru 0x017326 (0xd454 bytes) truncating to largest ever: 0x3fb2e 000195 1789266352.695437 trunc from 0x040000 to 0x03fb2d (0x4d2 bytes) 000196 1789266352.720216 punch from 0x007450 to 0x00f590 (0x8141 bytes) 000197 1789266352.730219 mapread 0x0206cf thru 0x027f22 (0x7854 bytes) 000198 1789266352.732202 mapwrite 0x026728 thru 0x028883 (0x215c bytes) 000201 1789266352.833173 read 0x01cde2 thru 0x02b499 (0xe6b8 bytes) 000203 1789266352.835615 mapwrite 0x0010b8 thru 0x00455a (0x34a3 bytes) 000204 1789266352.907155 punch from 0x00897c to 0x015cea (0xd36f bytes) 000205 1789266352.915783 mapread 0x008fa2 thru 0x013244 (0xa2a3 bytes) 000206 1789266352.938861 punch from 0x016fa9 to 0x024b6c (0xdbc4 bytes) 000207 1789266352.948252 read 0x01d8d1 thru 0x022224 (0x4954 bytes) 000208 1789266352.964192 mapread 0x0046fa thru 0x00adea (0x66f1 bytes) 000210 1789266352.971952 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fb2e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb2e): Mirror 4: 0x47c5c957 (stale) Mirror 5: 0x97c2bbd5 (stale) Mirror 6: 0x8fcbbe9d (stale) Mirror 7: 0x8fcbbe9d (stale) CRC-32 checksum value for chunk [0, 0x3fb2e): Mirror 1: 0x7646c9e9 Mirror 8: 0x7646c9e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb2e: skipped 000213 1789266354.147231 punch from 0x00b0f9 to 0x0103a6 (0x52ae bytes) 000214 1789266354.239587 write 0x01e43d thru 0x02182e (0x33f2 bytes) 000215 1789266354.248590 punch from 0x02e538 to 0x033a55 (0x551e bytes) 000216 1789266354.262539 mapread 0x00def9 thru 0x00f02d (0x1135 bytes) 000218 1789266354.303775 read 0x01d3f6 thru 0x02256a (0x5175 bytes) 000219 1789266354.316431 trunc from 0x03fb2e to 0x034292 (0xb89b bytes) 000220 1789266354.343613 mapread 0x019c16 thru 0x02520a (0xb5f5 bytes) 000221 1789266354.367483 punch from 0x00f7db to 0x01cc06 (0xd42c bytes) 000222 1789266354.376513 trunc from 0x034293 to 0x01d8d5 (0x169bd bytes) 000225 1789266354.416570 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d8d6): Mirror 4: 0x8da1036c (stale) Mirror 5: 0x9ba49a42 (stale) Mirror 6: 0x7439f9e6 (stale) Mirror 8: 0xf2ba674b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8d6): Mirror 4: 0x8da1036c (stale) Mirror 5: 0x9ba49a42 (stale) Mirror 6: 0x7439f9e6 (stale) Mirror 8: 0xf2ba674b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8d6: skipped 000226 1789266354.959880 trunc from 0x01d8d6 to 0x000e2e (0x1caa7 bytes) 000227 1789266354.985098 mapwrite 0x039b64 thru 0x03ffff (0x649c bytes) 000229 1789266355.094207 mapwrite 0x03fd71 thru 0x03ffff (0x28f bytes) 000230 1789266355.149860 read 0x03473e thru 0x036d47 (0x260a bytes) 000234 1789266355.151814 write 0x03974d thru 0x03ffff (0x68b3 bytes) 000235 1789266355.154346 punch from 0x01d10d to 0x029898 (0xc78c bytes) 000236 1789266355.173922 write 0x01a4dc thru 0x020fa2 (0x6ac7 bytes) 000237 1789266355.185793 mapread 0x004a14 thru 0x01237e (0xd96b bytes) 000238 1789266355.242262 trunc from 0x040000 to 0x026c1b (0x193e4 bytes) 000239 1789266355.284598 trunc from 0x026c1c to 0x01454a (0x126d1 bytes) 000241 1789266355.317421 punch from 0x00fb51 to 0x01454a (0x49fa bytes) 000243 1789266355.329947 trunc from 0x01454b to 0x009359 (0xb1f1 bytes) 000245 1789266355.385612 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x935a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x935a): Mirror 4: 0x6841ac2a (stale) Mirror 5: 0x6841ac2a (stale) Mirror 6: 0xd531e492 (stale) Mirror 8: 0x9bbe1e63 (stale) CRC-32 checksum value for chunk [0, 0x935a): Mirror 1: 0x30a55bdb Mirror 9: 0x30a55bdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x935a: skipped 000246 1789266356.667515 read 0x009057 thru 0x009359 (0x303 bytes) 000247 1789266356.686078 write 0x01c96a thru 0x02b2e8 (0xe97f bytes) 000248 1789266356.783638 write 0x02b853 thru 0x02f369 (0x3b17 bytes) 000249 1789266356.790485 write 0x02f888 thru 0x036309 (0x6a82 bytes) 000250 1789266356.791900 mapwrite 0x033352 thru 0x033899 (0x548 bytes) 000251 1789266356.924617 write 0x02c38c thru 0x02c879 (0x4ee bytes) 000252 1789266356.925915 trunc from 0x03630a to 0x030997 (0x5972 bytes) 000253 1789266356.982767 write 0x0010ea thru 0x0078b1 (0x67c8 bytes) 000257 1789266357.021112 punch from 0x00f8e0 to 0x0148fb (0x501c bytes) 000259 1789266357.054043 mapwrite 0x015a1f thru 0x018e12 (0x33f4 bytes) 000261 1789266357.194652 read 0x0280e5 thru 0x030997 (0x88b3 bytes) 000262 1789266357.195549 punch from 0x00d464 to 0x00e07c (0xc19 bytes) 000263 1789266357.207538 trunc from 0x030998 to 0x01ba32 (0x14f65 bytes) 000264 1789266357.233764 trunc from 0x01ba33 to 0x01fc91 (0x425f bytes) 000268 1789266357.296145 MIRROR_RESYNC 6 mirrors 000269 1789266358.941844 read 0x00044e thru 0x0029da (0x258d bytes) 000270 1789266358.954592 write 0x02d422 thru 0x030d8d (0x396c bytes) 000271 1789266359.048783 mapwrite 0x026d1d thru 0x02806d (0x1351 bytes) 000272 1789266359.199325 trunc from 0x030d8e to 0x03ccd6 (0xbf49 bytes) 000273 1789266359.232603 write 0x0385f8 thru 0x039fd4 (0x19dd bytes) 000275 1789266359.249777 mapread 0x01a693 thru 0x01a943 (0x2b1 bytes) 000276 1789266359.261473 mapwrite 0x019f61 thru 0x028400 (0xe4a0 bytes) 000278 1789266359.367796 trunc from 0x03ccd7 to 0x030bb3 (0xc123 bytes) 000279 1789266359.392929 write 0x02e4ae thru 0x03443f (0x5f92 bytes) 000280 1789266359.398507 mapwrite 0x025479 thru 0x033efd (0xea85 bytes) 000281 1789266359.460077 trunc from 0x034440 to 0x013f89 (0x204b6 bytes) 000282 1789266359.480050 read 0x005634 thru 0x0064e6 (0xeb3 bytes) 000283 1789266359.498798 trunc from 0x013f8a to 0x000a08 (0x13581 bytes) 000285 1789266359.525610 mapread 0x00059a thru 0x000a08 (0x46f bytes) 000286 1789266359.526535 mapwrite 0x036570 thru 0x03d826 (0x72b7 bytes) 000287 1789266359.632708 mapwrite 0x0004d4 thru 0x00cf7b (0xcaa8 bytes) 000289 1789266359.839602 read 0x012e66 thru 0x014fe5 (0x2180 bytes) 000292 1789266359.876147 write 0x00ada3 thru 0x00b560 (0x7be bytes) 000293 1789266359.885459 MIRROR_RESYNC 6 mirrors 000295 1789266362.276782 write 0x008c7b thru 0x010758 (0x7ade bytes) 000298 1789266362.381240 MIRROR_RESYNC 6 mirrors 000299 1789266364.287358 punch from 0x036b6d to 0x03c09a (0x552e bytes) 000300 1789266364.374258 MIRROR_RESYNC 6 mirrors 000301 1789266366.385472 write 0x030583 thru 0x037853 (0x72d1 bytes) 000303 1789266366.506191 write 0x0345f2 thru 0x037d55 (0x3764 bytes) 000304 1789266366.515786 mapread 0x03b23a thru 0x03d826 (0x25ed bytes) 000306 1789266366.551729 mapwrite 0x01fc91 thru 0x02224e (0x25be bytes) 000307 1789266366.653117 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d827): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d827): Mirror 4: 0xac22f331 (stale) Mirror 5: 0xac22f331 (stale) Mirror 6: 0xac22f331 (stale) Mirror 8: 0xac22f331 (stale) Mirror 9: 0xac22f331 (stale) CRC-32 checksum value for chunk [0, 0x3d827): Mirror 1: 0x1936aa3f Mirror 10: 0x1936aa3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d827: skipped 000308 1789266367.950509 punch from 0x00c1ab to 0x017c75 (0xbacb bytes) 000310 1789266368.065148 trunc from 0x03d827 to 0x0161ba (0x2766c bytes) 000311 1789266368.102803 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x161bb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x161bb): Mirror 4: 0x8950a37e (stale) Mirror 5: 0x8950a37e (stale) Mirror 6: 0x8950a37e (stale) Mirror 8: 0x8950a37e (stale) Mirror 9: 0x8950a37e (stale) Mirror 10: 0x8950a37e (stale) CRC-32 checksum value for chunk [0, 0x161bb): Mirror 1: 0xfbcc7103 Mirror 11: 0xfbcc7103 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x161bb: skipped 000312 1789266369.326748 read 0x015c8c thru 0x0161ba (0x52f bytes) 000313 1789266369.349670 trunc from 0x0161bb to 0x022c0e (0xca54 bytes) 000314 1789266369.489862 punch from 0x01f25d to 0x022c0e (0x39b2 bytes) 000315 1789266369.494483 mapwrite 0x02d1e2 thru 0x032115 (0x4f34 bytes) 000317 1789266369.608590 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32116): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32116): Mirror 4: 0x3c571ee8 (stale) Mirror 5: 0x3c571ee8 (stale) Mirror 6: 0x3c571ee8 (stale) Mirror 8: 0x3c571ee8 (stale) Mirror 9: 0x3c571ee8 (stale) Mirror 10: 0xa31d14cc (stale) Mirror 11: 0x271d5014 (stale) CRC-32 checksum value for chunk [0, 0x32116): Mirror 1: 0xe3912e75 Mirror 12: 0xe3912e75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32116: skipped 000320 1789266370.659836 mapread 0x006ad5 thru 0x012a5b (0xbf87 bytes) 000321 1789266370.715670 write 0x03bdcd thru 0x03ffff (0x4233 bytes) 000322 1789266370.825060 punch from 0x034e4f to 0x03e1fd (0x93af bytes) 000323 1789266370.842886 trunc from 0x040000 to 0x001548 (0x3eab7 bytes) 000324 1789266370.866211 write 0x02f6b5 thru 0x030aea (0x1436 bytes) 000326 1789266370.869045 read 0x004394 thru 0x00ed77 (0xa9e4 bytes) 000330 1789266370.882032 punch from 0x0252e9 to 0x026ba1 (0x18b9 bytes) 000332 1789266370.894512 mapwrite 0x00972f thru 0x00a6eb (0xfbd bytes) 000333 1789266370.942072 write 0x03b5b4 thru 0x03ffff (0x4a4c bytes) 000335 1789266370.943795 mapread 0x01ff9d thru 0x02573b (0x579f bytes) 000336 1789266370.955109 trunc from 0x040000 to 0x002ea6 (0x3d159 bytes) 000337 1789266370.985342 mapwrite 0x006a42 thru 0x00ff4d (0x950c bytes) 000338 1789266371.102618 punch from 0x003efd to 0x00ff4d (0xc051 bytes) 000339 1789266371.111219 mapread 0x008d8c thru 0x00ff4d (0x71c2 bytes) 000340 1789266371.133243 mapwrite 0x03a165 thru 0x03e8c8 (0x4764 bytes) 000341 1789266371.244280 trunc from 0x03e8c9 to 0x0026d9 (0x3c1ef bytes) 000342 1789266371.290301 mapread 0x0000b5 thru 0x0026d9 (0x2625 bytes) 000344 1789266371.303823 mapread 0x001395 thru 0x0017f4 (0x460 bytes) 000345 1789266371.305436 mapread 0x001559 thru 0x0026d9 (0x1181 bytes) 000347 1789266371.308013 write 0x01f5f1 thru 0x029254 (0x9c64 bytes) 000348 1789266371.320988 write 0x02cfa0 thru 0x036a52 (0x9ab3 bytes) 000349 1789266371.325155 punch from 0x016be4 to 0x01ba4c (0x4e69 bytes) 000350 1789266371.352866 punch from 0x00f21b to 0x015529 (0x630f bytes) 000353 1789266371.361507 mapread 0x00d505 thru 0x00e9c8 (0x14c4 bytes) 000357 1789266371.377605 read 0x0311f7 thru 0x036a52 (0x585c bytes) 000358 1789266371.384186 write 0x01abef thru 0x0227c0 (0x7bd2 bytes) 000359 1789266371.399082 write 0x03199b thru 0x03b90a (0x9f70 bytes) 000360 1789266371.405204 MIRROR_RESYNC 9 mirrors 000361 1789266373.346484 trunc from 0x03b90b to 0x03726f (0x469b bytes) 000362 1789266373.549828 read 0x02650e thru 0x031e81 (0xb974 bytes) 000364 1789266373.568534 mapwrite 0x0114f3 thru 0x01e682 (0xd190 bytes) 000365 1789266373.662822 write 0x0312f4 thru 0x038173 (0x6e80 bytes) 000366 1789266373.667789 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38174): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38174): Mirror 4: 0x5f2de487 (stale) Mirror 5: 0x5f2de487 (stale) Mirror 6: 0x5f2de487 (stale) Mirror 8: 0x5f2de487 (stale) Mirror 9: 0x5f2de487 (stale) Mirror 10: 0x5f2de487 (stale) Mirror 11: 0x5f2de487 (stale) Mirror 12: 0x5f2de487 (stale) CRC-32 checksum value for chunk [0, 0x38174): Mirror 1: 0x5b750064 Mirror 13: 0x5b750064 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38174: skipped 000367 1789266375.161973 mapwrite 0x0298cd thru 0x02d2b6 (0x39ea bytes) 000370 1789266375.388106 MIRROR_RESYNC 10 mirrors 000373 1789266377.387983 read 0x024624 thru 0x02e568 (0x9f45 bytes) 000374 1789266377.405065 trunc from 0x038174 to 0x01b390 (0x1cde3 bytes) 000375 1789266377.519715 mapwrite 0x00f71c thru 0x01baab (0xc390 bytes) 000376 1789266377.701300 mapread 0x0028f2 thru 0x004261 (0x1970 bytes) 000377 1789266377.724254 punch from 0x007069 to 0x00e779 (0x7711 bytes) 000378 1789266377.731297 trunc from 0x01baac to 0x00766e (0x1443d bytes) 000379 1789266377.761950 mapwrite 0x00aa69 thru 0x00dfe7 (0x357f bytes) 000380 1789266377.889265 MIRROR_RESYNC 10 mirrors 000382 1789266379.435744 mapwrite 0x0260b5 thru 0x02f96d (0x98b9 bytes) 000383 1789266379.665188 mapwrite 0x016564 thru 0x01a564 (0x4001 bytes) 000384 1789266379.729956 punch from 0x01082f to 0x01de33 (0xd605 bytes) 000388 1789266379.759081 MIRROR_RESYNC 10 mirrors 000389 1789266381.633043 mapwrite 0x01b14c thru 0x027220 (0xc0d5 bytes) 000391 1789266381.869104 mapwrite 0x03b52f thru 0x03ffff (0x4ad1 bytes) 000392 1789266381.987269 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x29aa7985 (stale) Mirror 6: 0x29aa7985 (stale) Mirror 8: 0x29aa7985 (stale) Mirror 9: 0x29aa7985 (stale) Mirror 10: 0x29aa7985 (stale) Mirror 11: 0x29aa7985 (stale) Mirror 12: 0x29aa7985 (stale) Mirror 13: 0x29aa7985 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x29aa7985 (stale) Mirror 6: 0x29aa7985 (stale) Mirror 8: 0x29aa7985 (stale) Mirror 9: 0x29aa7985 (stale) Mirror 10: 0x29aa7985 (stale) Mirror 11: 0x29aa7985 (stale) Mirror 12: 0x29aa7985 (stale) Mirror 13: 0x29aa7985 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000393 1789266382.633780 punch from 0x0208de to 0x022892 (0x1fb5 bytes) 000394 1789266382.645834 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x29aa7985 (stale) Mirror 6: 0x29aa7985 (stale) Mirror 8: 0x29aa7985 (stale) Mirror 9: 0x29aa7985 (stale) Mirror 10: 0x29aa7985 (stale) Mirror 11: 0x29aa7985 (stale) Mirror 12: 0x29aa7985 (stale) Mirror 13: 0x29aa7985 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa924d6d8 Mirror 14: 0xa924d6d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000396 1789266383.890955 punch from 0x00cc6b to 0x01683a (0x9bd0 bytes) 000397 1789266384.000757 read 0x03653d thru 0x03b30e (0x4dd2 bytes) 000398 1789266384.016516 mapread 0x024925 thru 0x02d51d (0x8bf9 bytes) 000399 1789266384.053620 punch from 0x01a2c3 to 0x02841f (0xe15d bytes) 000401 1789266384.074565 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x29aa7985 (stale) Mirror 6: 0x29aa7985 (stale) Mirror 8: 0x29aa7985 (stale) Mirror 9: 0x29aa7985 (stale) Mirror 11: 0x29aa7985 (stale) Mirror 12: 0x29aa7985 (stale) Mirror 13: 0x29aa7985 (stale) Mirror 14: 0xa924d6d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x29aa7985 (stale) Mirror 6: 0x29aa7985 (stale) Mirror 8: 0x29aa7985 (stale) Mirror 9: 0x29aa7985 (stale) Mirror 11: 0x29aa7985 (stale) Mirror 12: 0x29aa7985 (stale) Mirror 13: 0x29aa7985 (stale) Mirror 14: 0xa924d6d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000405 1789266384.736090 MIRROR_RESYNC 9 mirrors 000406 1789266386.458215 trunc from 0x040000 to 0x0185b0 (0x27a4f bytes) 000408 1789266386.561345 punch from 0x006dcb to 0x009da2 (0x2fd8 bytes) 000409 1789266386.570535 trunc from 0x0185b1 to 0x03a524 (0x21f74 bytes) 000410 1789266386.603624 write 0x03fa84 thru 0x03ffff (0x57c bytes) 000413 1789266386.608837 read 0x036462 thru 0x037402 (0xfa1 bytes) 000414 1789266386.615973 write 0x00127a thru 0x009767 (0x84ee bytes) 000415 1789266386.627667 write 0x039648 thru 0x03ffff (0x69b8 bytes) 000416 1789266386.635669 mapwrite 0x012bdd thru 0x01940a (0x682e bytes) 000417 1789266386.751976 mapwrite 0x008a48 thru 0x00e246 (0x57ff bytes) 000418 1789266386.833091 trunc from 0x040000 to 0x03f282 (0xd7d bytes) 000420 1789266386.868138 trunc from 0x03f283 to 0x02ca96 (0x127ec bytes) 000424 1789266386.914311 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ca97): Mirror 6: 0xa83e8dfc (stale) Mirror 8: 0xa83e8dfc (stale) Mirror 9: 0xa83e8dfc (stale) Mirror 11: 0xa83e8dfc (stale) Mirror 12: 0xa83e8dfc (stale) Mirror 13: 0xa83e8dfc (stale) Mirror 14: 0xa83e8dfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca97): Mirror 6: 0xa83e8dfc (stale) Mirror 8: 0xa83e8dfc (stale) Mirror 9: 0xa83e8dfc (stale) Mirror 11: 0xa83e8dfc (stale) Mirror 12: 0xa83e8dfc (stale) Mirror 13: 0xa83e8dfc (stale) Mirror 14: 0xa83e8dfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca97: skipped 000426 1789266387.467296 trunc from 0x02ca97 to 0x01601b (0x16a7b bytes) 000427 1789266387.493744 mapread 0x0077ce thru 0x015286 (0xdab9 bytes) 000428 1789266387.544908 read 0x001c5e thru 0x00b7b6 (0x9b59 bytes) 000430 1789266387.583042 MIRROR_RESYNC 8 mirrors 000432 1789266389.101968 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000435 1789266389.692486 trunc from 0x01601c to 0x00734e (0xeccd bytes) 000437 1789266389.769096 mapread 0x004cd6 thru 0x005b3c (0xe67 bytes) 000439 1789266389.778782 trunc from 0x00734f to 0x0040ab (0x32a3 bytes) 000441 1789266389.814776 read 0x000fc2 thru 0x0040ab (0x30ea bytes) 000442 1789266389.826677 mapread 0x000e6c thru 0x0040ab (0x3240 bytes) 000447 1789266389.832256 read 0x000b0e thru 0x00214e (0x1641 bytes) 000450 1789266389.833961 mapwrite 0x001f9d thru 0x00bf96 (0x9ffa bytes) 000451 1789266389.970852 punch from 0x0079e1 to 0x009da2 (0x23c2 bytes) 000452 1789266389.977141 mapwrite 0x007b92 thru 0x00e6e5 (0x6b54 bytes) 000453 1789266390.070623 mapwrite 0x01cc81 thru 0x021ff1 (0x5371 bytes) 000454 1789266390.167573 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21ff2): Mirror 6: 0xb94d1eb2 (stale) Mirror 9: 0xb94d1eb2 (stale) Mirror 12: 0xb94d1eb2 (stale) Mirror 13: 0xb94d1eb2 (stale) Mirror 14: 0xb94d1eb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ff2): Mirror 6: 0xb94d1eb2 (stale) Mirror 9: 0xb94d1eb2 (stale) Mirror 12: 0xb94d1eb2 (stale) Mirror 13: 0xb94d1eb2 (stale) Mirror 14: 0xb94d1eb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ff2: skipped 000456 1789266390.717227 trunc from 0x021ff2 to 0x032f8f (0x10f9e bytes) 000457 1789266390.736822 mapwrite 0x012f25 thru 0x022bd6 (0xfcb2 bytes) 000458 1789266390.824245 trunc from 0x032f90 to 0x027450 (0xbb3f bytes) 000459 1789266390.843896 punch from 0x023986 to 0x027450 (0x3acb bytes) 000461 1789266390.849093 mapread 0x008f1e thru 0x00d8cd (0x49b0 bytes) 000462 1789266390.858625 punch from 0x008bf7 to 0x017cb8 (0xf0c2 bytes) 000463 1789266390.869448 MIRROR_RESYNC 6 mirrors 000469 1789266392.322107 write 0x01f76a thru 0x02e9cb (0xf262 bytes) 000472 1789266392.454180 punch from 0x02a29f to 0x02e9cb (0x472d bytes) 000473 1789266392.499929 read 0x0018fd thru 0x0069db (0x50df bytes) 000475 1789266392.534817 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 6: 0x21319db5 (stale) Mirror 9: 0x21319db5 (stale) Mirror 12: 0x21319db5 (stale) Mirror 13: 0x21319db5 (stale) Mirror 14: 0x21319db5 (stale) CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 1: 0x81237556 Mirror 15: 0x81237556 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9cc: skipped 000477 1789266393.722775 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 6: 0x21319db5 (stale) Mirror 9: 0x21319db5 (stale) Mirror 12: 0x21319db5 (stale) Mirror 13: 0x21319db5 (stale) Mirror 14: 0x21319db5 (stale) CRC-32 checksum value for chunk [0, 0x2e9cc): Mirror 1: 0x81237556 Mirror 15: 0x81237556 Mirror 16: 0x81237556 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9cc: skipped 000478 1789266394.662330 mapread 0x005d4e thru 0x01465f (0xe912 bytes) 000480 1789266394.707163 mapwrite 0x0319df thru 0x0377c6 (0x5de8 bytes) 000481 1789266394.892909 write 0x00b2f3 thru 0x01892d (0xd63b bytes) 000482 1789266394.904777 mapwrite 0x02d517 thru 0x035dd1 (0x88bb bytes) 000483 1789266394.962346 read 0x0197de thru 0x019e90 (0x6b3 bytes) 000484 1789266394.982177 mapread 0x027a15 thru 0x02b7e6 (0x3dd2 bytes) 000485 1789266395.010029 mapwrite 0x0250cd thru 0x030ab6 (0xb9ea bytes) 000486 1789266395.066181 trunc from 0x0377c7 to 0x028e5f (0xe967 bytes) 000490 1789266395.100780 mapread 0x0174d8 thru 0x01829e (0xdc7 bytes) 000491 1789266395.126968 MIRROR_RESYNC 8 mirrors 000492 1789266396.684278 read 0x01fa0f thru 0x026969 (0x6f5b bytes) 000494 1789266396.693917 punch from 0x01bc20 to 0x022f6e (0x734f bytes) 000495 1789266396.791811 read 0x011106 thru 0x01acb0 (0x9bab bytes) 000496 1789266396.804256 read 0x025c84 thru 0x028e5f (0x31dc bytes) 000497 1789266396.807194 write 0x01f142 thru 0x023042 (0x3f01 bytes) 000498 1789266396.813566 MIRROR_RESYNC 8 mirrors 000500 1789266398.213925 read 0x02066d thru 0x028e5f (0x87f3 bytes) 000501 1789266398.224712 write 0x000ab2 thru 0x0020b5 (0x1604 bytes) 000502 1789266398.311725 punch from 0x013348 to 0x013cf5 (0x9ae bytes) 000505 1789266398.324098 read 0x005b25 thru 0x0135bd (0xda99 bytes) 000506 1789266398.336731 write 0x02c4e6 thru 0x02c8c8 (0x3e3 bytes) 000507 1789266398.337717 punch from 0x0046dd to 0x005530 (0xe54 bytes) 000511 1789266398.352537 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2c8c9): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c8c9): Mirror 6: 0xe51efb1b (stale) Mirror 9: 0xe51efb1b (stale) Mirror 12: 0xe51efb1b (stale) Mirror 13: 0xe51efb1b (stale) Mirror 14: 0xe51efb1b (stale) Mirror 15: 0xe51efb1b (stale) Mirror 16: 0xe51efb1b (stale) CRC-32 checksum value for chunk [0, 0x2c8c9): Mirror 1: 0x53ea2713 Mirror 17: 0x53ea2713 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c8c9: skipped 000512 1789266399.367931 mapread 0x01a96f thru 0x022995 (0x8027 bytes) 000513 1789266399.404044 MIRROR_RESYNC 9 mirrors 000515 1789266401.080384 write 0x01b20c thru 0x01e0c4 (0x2eb9 bytes) 000518 1789266401.175837 mapwrite 0x00e2a7 thru 0x01a729 (0xc483 bytes) 000519 1789266401.307570 trunc from 0x02c8c9 to 0x01bf66 (0x10962 bytes) 000521 1789266401.369227 trunc from 0x01bf67 to 0x01d517 (0x15b1 bytes) 000522 1789266401.386989 trunc from 0x01d518 to 0x011300 (0xc217 bytes) 000523 1789266401.412947 mapwrite 0x012178 thru 0x016f1c (0x4da5 bytes) 000524 1789266401.541028 read 0x00ca98 thru 0x016f1c (0xa485 bytes) 000525 1789266401.541894 mapread 0x009203 thru 0x011252 (0x8050 bytes) 000526 1789266401.543749 trunc from 0x016f1d to 0x002236 (0x14ce6 bytes) 000528 1789266401.573421 write 0x03b230 thru 0x03ffff (0x4dd0 bytes) 000529 1789266401.575369 mapread 0x01f99b thru 0x022504 (0x2b6a bytes) 000530 1789266401.590710 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x46822efa (stale) Mirror 9: 0x46822efa (stale) Mirror 12: 0x46822efa (stale) Mirror 13: 0x46822efa (stale) Mirror 14: 0x46822efa (stale) Mirror 15: 0x46822efa (stale) Mirror 16: 0x46822efa (stale) Mirror 17: 0x46822efa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7aa6ebe5 Mirror 18: 0x7aa6ebe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000531 1789266403.036829 read 0x01ed85 thru 0x0203f2 (0x166e bytes) 000532 1789266403.060643 read 0x01bb79 thru 0x02612a (0xa5b2 bytes) 000533 1789266403.092350 punch from 0x009b3f to 0x00e37a (0x483c bytes) 000534 1789266403.242711 punch from 0x02bdf1 to 0x02f2ad (0x34bd bytes) 000535 1789266403.253705 mapread 0x0143ee thru 0x0239ec (0xf5ff bytes) 000537 1789266403.424503 mapwrite 0x01041e thru 0x01d831 (0xd414 bytes) 000540 1789266403.540046 mapwrite 0x01f77b thru 0x02e8a1 (0xf127 bytes) 000541 1789266403.715289 trunc from 0x040000 to 0x006776 (0x39889 bytes) 000542 1789266403.744149 trunc from 0x006777 to 0x00e71f (0x7fa9 bytes) 000543 1789266403.766740 read 0x003aae thru 0x0056ed (0x1c40 bytes) 000544 1789266403.792063 read 0x004b8b thru 0x00c16f (0x75e5 bytes) 000545 1789266403.798338 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xe720): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe720): Mirror 6: 0xcac0c7f9 (stale) Mirror 9: 0xcac0c7f9 (stale) Mirror 12: 0xcac0c7f9 (stale) Mirror 13: 0xcac0c7f9 (stale) Mirror 14: 0xcac0c7f9 (stale) Mirror 15: 0xcac0c7f9 (stale) Mirror 16: 0xcac0c7f9 (stale) Mirror 17: 0xcac0c7f9 (stale) Mirror 18: 0x414a2001 (stale) CRC-32 checksum value for chunk [0, 0xe720): Mirror 1: 0x414a2001 Mirror 19: 0x414a2001 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe720: skipped 000547 1789266404.907933 mapwrite 0x00a8b5 thru 0x00f87c (0x4fc8 bytes) 000548 1789266405.078871 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf87d): Mirror 6: 0xbcb91e7a (stale) Mirror 9: 0xbcb91e7a (stale) Mirror 12: 0xbcb91e7a (stale) Mirror 13: 0xbcb91e7a (stale) Mirror 14: 0xbcb91e7a (stale) Mirror 15: 0xbcb91e7a (stale) Mirror 16: 0xbcb91e7a (stale) Mirror 18: 0x1fae5c55 (stale) Mirror 19: 0x1fae5c55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf87d): Mirror 6: 0xbcb91e7a (stale) Mirror 9: 0xbcb91e7a (stale) Mirror 12: 0xbcb91e7a (stale) Mirror 13: 0xbcb91e7a (stale) Mirror 14: 0xbcb91e7a (stale) Mirror 15: 0xbcb91e7a (stale) Mirror 16: 0xbcb91e7a (stale) Mirror 18: 0x1fae5c55 (stale) Mirror 19: 0x1fae5c55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf87d: skipped 000550 1789266405.722075 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf87d): Mirror 6: 0xbcb91e7a (stale) Mirror 9: 0xbcb91e7a (stale) Mirror 12: 0xbcb91e7a (stale) Mirror 13: 0xbcb91e7a (stale) Mirror 14: 0xbcb91e7a (stale) Mirror 15: 0xbcb91e7a (stale) Mirror 18: 0x1fae5c55 (stale) Mirror 19: 0x1fae5c55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf87d): Mirror 6: 0xbcb91e7a (stale) Mirror 9: 0xbcb91e7a (stale) Mirror 12: 0xbcb91e7a (stale) Mirror 13: 0xbcb91e7a (stale) Mirror 14: 0xbcb91e7a (stale) Mirror 15: 0xbcb91e7a (stale) Mirror 18: 0x1fae5c55 (stale) Mirror 19: 0x1fae5c55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf87d: skipped 000551 1789266406.256599 punch from 0x00f7cc to 0x00f87c (0xb1 bytes) 000553 1789266406.264050 mapread 0x00db2d thru 0x00f87c (0x1d50 bytes) 000554 1789266406.289697 mapread 0x007e4f thru 0x009427 (0x15d9 bytes) 000555 1789266406.302380 write 0x014094 thru 0x01c7b7 (0x8724 bytes) 000558 1789266406.309102 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c7b8): Mirror 6: 0xadcfe077 (stale) Mirror 9: 0xadcfe077 (stale) Mirror 12: 0xadcfe077 (stale) Mirror 13: 0xadcfe077 (stale) Mirror 15: 0xadcfe077 (stale) Mirror 18: 0x64d3e691 (stale) Mirror 19: 0x64d3e691 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c7b8): Mirror 6: 0xadcfe077 (stale) Mirror 9: 0xadcfe077 (stale) Mirror 12: 0xadcfe077 (stale) Mirror 13: 0xadcfe077 (stale) Mirror 15: 0xadcfe077 (stale) Mirror 18: 0x64d3e691 (stale) Mirror 19: 0x64d3e691 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c7b8: skipped 000560 1789266406.879716 mapwrite 0x034898 thru 0x03c8df (0x8048 bytes) 000561 1789266406.984049 punch from 0x02dea5 to 0x02f251 (0x13ad bytes) 000565 1789266406.995841 read 0x00148a thru 0x00fd66 (0xe8dd bytes) 000566 1789266407.024612 mapread 0x00d846 thru 0x013bf1 (0x63ac bytes) 000568 1789266407.038272 trunc from 0x03c8e0 to 0x01183e (0x2b0a1 bytes) 000569 1789266407.067529 trunc from 0x01183f to 0x002af9 (0xed45 bytes) 000570 1789266407.099778 write 0x00020b thru 0x0015b2 (0x13a8 bytes) 000571 1789266407.112035 write 0x0153f4 thru 0x0170dd (0x1cea bytes) 000572 1789266407.127128 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x170de): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x170de): Mirror 6: 0x94bf448a (stale) Mirror 9: 0x94bf448a (stale) Mirror 12: 0x94bf448a (stale) Mirror 13: 0x94bf448a (stale) Mirror 15: 0x94bf448a (stale) Mirror 18: 0x879114 (stale) Mirror 19: 0x879114 (stale) CRC-32 checksum value for chunk [0, 0x170de): Mirror 1: 0x6ab06b0 Mirror 20: 0x6ab06b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x170de: skipped 000574 1789266408.143642 mapread 0x01042d thru 0x0170dd (0x6cb1 bytes) 000576 1789266408.206113 trunc from 0x0170de to 0x029085 (0x11fa8 bytes) 000577 1789266408.349665 write 0x037d65 thru 0x03ffff (0x829b bytes) 000578 1789266408.351147 mapwrite 0x00bba7 thru 0x01a4d3 (0xe92d bytes) 000580 1789266408.503623 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x46822efa (stale) Mirror 9: 0x46822efa (stale) Mirror 12: 0x46822efa (stale) Mirror 13: 0x46822efa (stale) Mirror 15: 0x46822efa (stale) Mirror 18: 0x7aa6ebe5 (stale) Mirror 19: 0xc14de831 (stale) Mirror 20: 0x772d38bb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaee1692c Mirror 21: 0xaee1692c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1789266409.929776 trunc from 0x040000 to 0x0189fb (0x27604 bytes) 000584 1789266410.035722 write 0x0098d3 thru 0x011f49 (0x8677 bytes) 000585 1789266410.055020 read 0x016c88 thru 0x0189fb (0x1d74 bytes) 000586 1789266410.062896 read 0x00ba80 thru 0x0189fb (0xcf7c bytes) 000589 1789266410.077604 read 0x00c89d thru 0x00cca1 (0x405 bytes) 000591 1789266410.081102 trunc from 0x0189fc to 0x00a6b0 (0xe34b bytes) 000592 1789266410.111312 trunc from 0x00a6b1 to 0x011434 (0x6d84 bytes) 000593 1789266410.148560 mapread 0x008247 thru 0x00b6f6 (0x34b0 bytes) 000594 1789266410.158728 read 0x000bc7 thru 0x00d5d3 (0xca0d bytes) 000595 1789266410.159770 trunc from 0x011435 to 0x032789 (0x21355 bytes) 000596 1789266410.180107 trunc from 0x03278a to 0x027707 (0xb082 bytes) 000598 1789266410.198900 read 0x018563 thru 0x025696 (0xd134 bytes) 000600 1789266410.208699 write 0x038a34 thru 0x03ffff (0x75cc bytes) 000601 1789266410.210021 read 0x02318e thru 0x02b062 (0x7ed5 bytes) 000604 1789266410.224741 write 0x003771 thru 0x0091a2 (0x5a32 bytes) 000605 1789266410.226837 read 0x02a951 thru 0x0323ef (0x7a9f bytes) 000606 1789266410.242741 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x46822efa (stale) Mirror 9: 0x46822efa (stale) Mirror 12: 0x46822efa (stale) Mirror 13: 0x46822efa (stale) Mirror 15: 0x46822efa (stale) Mirror 18: 0x7aa6ebe5 (stale) Mirror 19: 0xc14de831 (stale) Mirror 20: 0x772d38bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x46822efa (stale) Mirror 9: 0x46822efa (stale) Mirror 12: 0x46822efa (stale) Mirror 13: 0x46822efa (stale) Mirror 15: 0x46822efa (stale) Mirror 18: 0x7aa6ebe5 (stale) Mirror 19: 0xc14de831 (stale) Mirror 20: 0x772d38bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000607 1789266410.895891 MIRROR_RESYNC 9 mirrors 000609 1789266413.861119 read 0x0118d9 thru 0x020f6a (0xf692 bytes) 000611 1789266413.882434 write 0x02d8ff thru 0x02f2fe (0x1a00 bytes) 000612 1789266413.957660 read 0x025bd0 thru 0x035274 (0xf6a5 bytes) 000613 1789266413.975204 trunc from 0x040000 to 0x028c8e (0x17371 bytes) 000614 1789266414.019138 punch from 0x002f68 to 0x00a2ac (0x7345 bytes) 000615 1789266414.025317 punch from 0x01d880 to 0x01f660 (0x1de1 bytes) 000618 1789266414.031999 write 0x0312a6 thru 0x039790 (0x84eb bytes) 000619 1789266414.034128 mapwrite 0x03b7c2 thru 0x03ffff (0x483e bytes) 000622 1789266414.150303 write 0x02cbe1 thru 0x02fb33 (0x2f53 bytes) 000623 1789266414.163797 trunc from 0x040000 to 0x008e93 (0x3716c bytes) 000625 1789266414.190593 punch from 0x0082cf to 0x008e93 (0xbc5 bytes) 000626 1789266414.198338 write 0x016ac4 thru 0x01b20a (0x4747 bytes) 000628 1789266414.202078 write 0x026e5b thru 0x0364b4 (0xf65a bytes) 000629 1789266414.204212 mapread 0x0270c0 thru 0x0290d6 (0x2017 bytes) 000630 1789266414.206321 mapwrite 0x00d1cd thru 0x01993e (0xc772 bytes) 000631 1789266414.363052 mapread 0x014eab thru 0x02206c (0xd1c2 bytes) 000632 1789266414.398527 mapwrite 0x00e969 thru 0x01ccf8 (0xe390 bytes) 000633 1789266414.460594 write 0x01a215 thru 0x027737 (0xd523 bytes) 000635 1789266414.463099 trunc from 0x0364b5 to 0x035c53 (0x861 bytes) 000637 1789266414.505837 mapread 0x00be6d thru 0x01a96e (0xeb02 bytes) 000639 1789266414.537827 read 0x00bc07 thru 0x00df4b (0x2345 bytes) 000640 1789266414.538504 write 0x02ff55 thru 0x03d81a (0xd8c6 bytes) 000641 1789266414.541973 mapread 0x02a609 thru 0x0356b9 (0xb0b1 bytes) 000642 1789266414.543274 read 0x0247c4 thru 0x0249c1 (0x1fe bytes) 000645 1789266414.548791 write 0x01240d thru 0x018231 (0x5e25 bytes) 000646 1789266414.559797 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d81b): Mirror 6: 0x730479af (stale) Mirror 9: 0x730479af (stale) Mirror 12: 0x730479af (stale) Mirror 13: 0x730479af (stale) Mirror 15: 0x730479af (stale) Mirror 18: 0x730479af (stale) Mirror 19: 0x730479af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d81b): Mirror 6: 0x730479af (stale) Mirror 9: 0x730479af (stale) Mirror 12: 0x730479af (stale) Mirror 13: 0x730479af (stale) Mirror 15: 0x730479af (stale) Mirror 18: 0x730479af (stale) Mirror 19: 0x730479af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d81b: skipped 000647 1789266415.346098 MIRROR_RESYNC 8 mirrors 000648 1789266417.137302 trunc from 0x03d81b to 0x028c0d (0x14c0d bytes) 000650 1789266417.225148 mapread 0x002dbc thru 0x00b406 (0x864b bytes) 000651 1789266417.248179 punch from 0x004f61 to 0x01445f (0xf4ff bytes) 000652 1789266417.261296 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c0e): Mirror 6: 0x6b4319e (stale) Mirror 9: 0x6b4319e (stale) Mirror 12: 0x6b4319e (stale) Mirror 13: 0x6b4319e (stale) Mirror 15: 0x6b4319e (stale) Mirror 19: 0x6b4319e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c0e): Mirror 6: 0x6b4319e (stale) Mirror 9: 0x6b4319e (stale) Mirror 12: 0x6b4319e (stale) Mirror 13: 0x6b4319e (stale) Mirror 15: 0x6b4319e (stale) Mirror 19: 0x6b4319e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c0e: skipped 000653 1789266417.784383 trunc from 0x028c0e to 0x031a98 (0x8e8b bytes) 000654 1789266417.822584 mapwrite 0x0052a1 thru 0x00699f (0x16ff bytes) 000656 1789266417.947778 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31a99): Mirror 6: 0x29274f4d (stale) Mirror 9: 0x29274f4d (stale) Mirror 13: 0x29274f4d (stale) Mirror 15: 0x29274f4d (stale) Mirror 19: 0x29274f4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a99): Mirror 6: 0x29274f4d (stale) Mirror 9: 0x29274f4d (stale) Mirror 13: 0x29274f4d (stale) Mirror 15: 0x29274f4d (stale) Mirror 19: 0x29274f4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a99: skipped 000657 1789266418.525657 punch from 0x024c6f to 0x02f8d7 (0xac69 bytes) 000658 1789266418.536293 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31a99): Mirror 6: 0x29274f4d (stale) Mirror 13: 0x29274f4d (stale) Mirror 15: 0x29274f4d (stale) Mirror 19: 0x29274f4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a99): Mirror 6: 0x29274f4d (stale) Mirror 13: 0x29274f4d (stale) Mirror 15: 0x29274f4d (stale) Mirror 19: 0x29274f4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a99: skipped 000659 1789266419.014250 mapwrite 0x039bbf thru 0x03a731 (0xb73 bytes) 000660 1789266419.111068 read 0x02e5dd thru 0x03a731 (0xc155 bytes) 000661 1789266419.134926 MIRROR_RESYNC 5 mirrors 000662 1789266420.293371 write 0x023293 thru 0x028288 (0x4ff6 bytes) 000665 1789266420.408622 write 0x00fd25 thru 0x01ab5f (0xae3b bytes) 000666 1789266420.423299 trunc from 0x03a732 to 0x00dde2 (0x2c94f bytes) 000667 1789266420.494377 write 0x00734f thru 0x013b15 (0xc7c7 bytes) 000669 1789266420.511977 write 0x017fc1 thru 0x02043a (0x847a bytes) 000670 1789266420.526800 MIRROR_RESYNC 5 mirrors 000671 1789266421.776565 read 0x01663a thru 0x01ba65 (0x542c bytes) 000673 1789266421.824218 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000674 1789266422.181652 trunc from 0x02043b to 0x03b867 (0x1b42d bytes) 000675 1789266422.290822 trunc from 0x03b868 to 0x021536 (0x1a331 bytes) 000676 1789266422.324147 trunc from 0x021537 to 0x03052b (0xeff5 bytes) 000677 1789266422.342966 trunc from 0x03052c to 0x000679 (0x2feb2 bytes) 000678 1789266422.360093 trunc from 0x00067a to 0x02083f (0x201c6 bytes) 000679 1789266422.380524 mapwrite 0x039670 thru 0x03ffff (0x6990 bytes) 000680 1789266422.472043 read 0x019a45 thru 0x01d262 (0x381e bytes) 000681 1789266422.482468 mapwrite 0x03bb74 thru 0x03ffff (0x448c bytes) 000682 1789266422.528971 write 0x00c609 thru 0x00ce5e (0x856 bytes) 000685 1789266422.535259 write 0x007c4a thru 0x014beb (0xcfa2 bytes) 000687 1789266422.546851 trunc from 0x040000 to 0x008db1 (0x3724e bytes) 000688 1789266422.584355 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8db2): Mirror 6: 0xe065b668 (stale) Mirror 15: 0xe065b668 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8db2): Mirror 6: 0xe065b668 (stale) Mirror 15: 0xe065b668 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8db2: skipped 000689 1789266423.061385 read 0x007dd9 thru 0x008d75 (0xf9d bytes) 000690 1789266423.067978 punch from 0x004fda to 0x005c03 (0xc2a bytes) 000691 1789266423.072942 trunc from 0x008db2 to 0x03b86d (0x32abc bytes) 000692 1789266423.093498 punch from 0x03502a to 0x03b86d (0x6844 bytes) 000693 1789266423.101225 trunc from 0x03b86e to 0x015848 (0x26025 bytes) 000696 1789266423.129252 mapwrite 0x03ae04 thru 0x03ffff (0x51fc bytes) 000698 1789266423.224293 read 0x03d278 thru 0x03ffff (0x2d88 bytes) 000700 1789266423.225513 mapwrite 0x014ec7 thru 0x01da09 (0x8b43 bytes) 000701 1789266423.319475 trunc from 0x040000 to 0x00e3c9 (0x31c36 bytes) 000702 1789266423.365025 trunc from 0x00e3ca to 0x02af74 (0x1cbab bytes) 000703 1789266423.400278 write 0x01cd95 thru 0x02c056 (0xf2c2 bytes) 000705 1789266423.411135 trunc from 0x02c057 to 0x010fbd (0x1b099 bytes) 000707 1789266423.457813 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10fbe): Mirror 6: 0x57add0e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fbe): Mirror 6: 0x57add0e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fbe: skipped 000708 1789266423.775362 read 0x010cad thru 0x010fbd (0x311 bytes) 000709 1789266423.780354 mapread 0x009caa thru 0x010fbd (0x7314 bytes) 000711 1789266423.801133 mapwrite 0x038392 thru 0x03d869 (0x54d8 bytes) 000712 1789266423.890299 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 6: 0x7efe0184 (stale) CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 1: 0xe57582e6 Mirror 7: 0xe57582e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d86a: skipped 000714 1789266425.212204 mapwrite 0x02c842 thru 0x0383ce (0xbb8d bytes) 000715 1789266425.415471 trunc from 0x03d86a to 0x0176fa (0x2616f bytes) 000717 1789266425.437137 mapread 0x00b404 thru 0x0100b9 (0x4cb6 bytes) 000718 1789266425.449119 punch from 0x014289 to 0x0176fa (0x3472 bytes) 000720 1789266425.455190 write 0x01c2da thru 0x024e77 (0x8b9e bytes) 000721 1789266425.456588 mapread 0x00223b thru 0x00c5ad (0xa373 bytes) 000723 1789266425.464298 write 0x008dc0 thru 0x009dee (0x102f bytes) 000724 1789266425.465371 mapread 0x01bf4c thru 0x024e77 (0x8f2c bytes) 000726 1789266425.476268 read 0x00594b thru 0x0102ac (0xa962 bytes) 000727 1789266425.477116 trunc from 0x024e78 to 0x022810 (0x2667 bytes) 000728 1789266425.495380 write 0x000c04 thru 0x001eb5 (0x12b2 bytes) 000730 1789266425.501906 trunc from 0x022811 to 0x03976b (0x16f5b bytes) 000731 1789266425.519005 mapread 0x010b2f thru 0x01f6a4 (0xeb76 bytes) 000733 1789266425.522917 trunc from 0x03976c to 0x036041 (0x372a bytes) 000734 1789266425.540982 mapread 0x02ab5f thru 0x02c7b6 (0x1c58 bytes) 000735 1789266425.552222 write 0x005665 thru 0x00a810 (0x51ac bytes) 000736 1789266425.555122 write 0x00aa5f thru 0x00e455 (0x39f7 bytes) 000738 1789266425.557547 trunc from 0x036042 to 0x02fa7b (0x65c6 bytes) 000740 1789266425.604915 MIRROR_RESYNC 3 mirrors 000742 1789266426.679166 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000743 1789266427.155807 MIRROR_RESYNC 2 mirrors 000745 1789266427.708791 MIRROR_EXTEND to 3 mirrors 000746 1789266428.862931 MIRROR_EXTEND to 4 mirrors 000748 1789266430.273752 punch from 0x02025f to 0x0207e8 (0x58a bytes) 000750 1789266430.363825 punch from 0x016af0 to 0x025ed4 (0xf3e5 bytes) 000751 1789266430.375449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2fa7c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa7c): Mirror 7: 0x583d7fc8 (stale) Mirror 8: 0x583d7fc8 (stale) Mirror 9: 0x583d7fc8 (stale) CRC-32 checksum value for chunk [0, 0x2fa7c): Mirror 1: 0xec173d38 Mirror 10: 0xec173d38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa7c: skipped 000752 1789266431.768143 trunc from 0x02fa7c to 0x0354f3 (0x5a78 bytes) 000753 1789266431.890144 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x354f4): Mirror 7: 0x6a5a4741 (stale) Mirror 8: 0x6a5a4741 (stale) Mirror 10: 0x32419e3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354f4): Mirror 7: 0x6a5a4741 (stale) Mirror 8: 0x6a5a4741 (stale) Mirror 10: 0x32419e3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354f4: skipped 000754 1789266432.296070 write 0x004d93 thru 0x011ba0 (0xce0e bytes) 000755 1789266432.307378 punch from 0x00e302 to 0x014147 (0x5e46 bytes) 000756 1789266432.327441 mapwrite 0x019bab thru 0x023979 (0x9dcf bytes) 000758 1789266432.406370 mapread 0x0255c3 thru 0x030c7d (0xb6bb bytes) 000759 1789266432.436959 read 0x009feb thru 0x00a525 (0x53b bytes) 000761 1789266432.438756 write 0x020db0 thru 0x02bb8e (0xaddf bytes) 000762 1789266432.445015 MIRROR_RESYNC 4 mirrors 000763 1789266433.776673 MIRROR_EXTEND to 5 mirrors 000764 1789266434.819907 trunc from 0x0354f4 to 0x03ed26 (0x9833 bytes) 000765 1789266434.922559 trunc from 0x03ed27 to 0x0066df (0x38647 bytes) 000766 1789266434.957264 trunc from 0x0066e0 to 0x00d55b (0x6e7c bytes) 000770 1789266434.990975 trunc from 0x00d55c to 0x02e53b (0x20fe0 bytes) 000771 1789266435.025616 MIRROR_RESYNC 5 mirrors 000772 1789266436.501929 read 0x023a82 thru 0x02e53b (0xaaba bytes) 000773 1789266436.515085 mapwrite 0x01bca2 thru 0x02414f (0x84ae bytes) 000774 1789266436.700123 trunc from 0x02e53c to 0x0008da (0x2dc61 bytes) 000777 1789266436.748091 mapwrite 0x004bff thru 0x012085 (0xd487 bytes) 000779 1789266436.878858 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12086): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12086): Mirror 7: 0x4d3597f2 (stale) Mirror 8: 0x4d3597f2 (stale) Mirror 10: 0x4d3597f2 (stale) Mirror 11: 0x4d3597f2 (stale) CRC-32 checksum value for chunk [0, 0x12086): Mirror 1: 0xd8e0259f Mirror 12: 0xd8e0259f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12086: skipped 000780 1789266437.988533 write 0x0222e9 thru 0x03043b (0xe153 bytes) 000782 1789266438.098878 read 0x020699 thru 0x02ada3 (0xa70b bytes) 000785 1789266438.109720 write 0x02cf89 thru 0x0343ad (0x7425 bytes) 000786 1789266438.112005 mapwrite 0x03cf5d thru 0x03ffff (0x30a3 bytes) 000787 1789266438.291116 trunc from 0x040000 to 0x0308d8 (0xf727 bytes) 000788 1789266438.347655 MIRROR_RESYNC 6 mirrors 000789 1789266439.772951 trunc from 0x0308d9 to 0x03e08f (0xd7b7 bytes) 000790 1789266439.856202 write 0x03a653 thru 0x03ffff (0x59ad bytes) 000792 1789266439.864290 mapwrite 0x03faa3 thru 0x03ffff (0x55d bytes) 000794 1789266439.917198 read 0x01badd thru 0x020516 (0x4a3a bytes) 000795 1789266439.932529 punch from 0x03f35b to 0x03ffff (0xca5 bytes) 000796 1789266439.943109 mapread 0x036467 thru 0x03eb3a (0x86d4 bytes) 000797 1789266439.973740 read 0x024653 thru 0x024795 (0x143 bytes) 000798 1789266439.988099 punch from 0x02ec91 to 0x036ada (0x7e4a bytes) 000799 1789266439.993258 punch from 0x03fdf3 to 0x03ffff (0x20d bytes) 000800 1789266439.997676 write 0x00c786 thru 0x017f75 (0xb7f0 bytes) 000803 1789266440.006183 MIRROR_RESYNC 6 mirrors 000804 1789266441.065980 MIRROR_RESYNC 6 mirrors 000805 1789266441.651280 mapwrite 0x03effe thru 0x03ffff (0x1002 bytes) 000807 1789266441.832991 trunc from 0x040000 to 0x020ebf (0x1f140 bytes) 000808 1789266441.865028 write 0x02431f thru 0x02f32d (0xb00f bytes) 000809 1789266441.867988 mapwrite 0x021ded thru 0x02c37c (0xa590 bytes) 000810 1789266441.936687 mapread 0x00c701 thru 0x013fce (0x78ce bytes) 000811 1789266441.947976 mapwrite 0x018a4e thru 0x01e382 (0x5935 bytes) 000812 1789266442.035992 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f32e): Mirror 7: 0x15497bbb (stale) Mirror 8: 0x15497bbb (stale) Mirror 11: 0x15497bbb (stale) Mirror 12: 0x15497bbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f32e): Mirror 7: 0x15497bbb (stale) Mirror 8: 0x15497bbb (stale) Mirror 11: 0x15497bbb (stale) Mirror 12: 0x15497bbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f32e: skipped 000813 1789266442.561437 mapread 0x00bf36 thru 0x0105b5 (0x4680 bytes) 000814 1789266442.601603 read 0x00c2b8 thru 0x00e4bf (0x2208 bytes) 000815 1789266442.618103 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f32e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f32e): Mirror 7: 0x15497bbb (stale) Mirror 8: 0x15497bbb (stale) Mirror 11: 0x15497bbb (stale) Mirror 12: 0x15497bbb (stale) CRC-32 checksum value for chunk [0, 0x2f32e): Mirror 1: 0x9b3ce151 Mirror 13: 0x9b3ce151 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f32e: skipped 000818 1789266444.084668 read 0x01ec07 thru 0x02e799 (0xfb93 bytes) 000820 1789266444.114170 read 0x01f1ee thru 0x02dc28 (0xea3b bytes) 000821 1789266444.121509 punch from 0x01e539 to 0x02bc95 (0xd75d bytes) 000822 1789266444.205592 punch from 0x00812c to 0x00d2c4 (0x5199 bytes) 000823 1789266444.211821 trunc from 0x02f32e to 0x013d18 (0x1b615 bytes) 000824 1789266444.259737 mapwrite 0x028edf thru 0x02c801 (0x3923 bytes) 000826 1789266444.356133 trunc from 0x02c802 to 0x003aa8 (0x28d59 bytes) 000828 1789266444.394119 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3aa9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa9): Mirror 7: 0xb568897e (stale) Mirror 8: 0xb568897e (stale) Mirror 11: 0xb568897e (stale) Mirror 12: 0xb568897e (stale) Mirror 13: 0xb568897e (stale) CRC-32 checksum value for chunk [0, 0x3aa9): Mirror 1: 0xb568897e Mirror 14: 0xb568897e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa9: skipped 000829 1789266445.477592 mapread 0x001825 thru 0x0037b6 (0x1f92 bytes) 000832 1789266445.535727 write 0x03e29e thru 0x03ffff (0x1d62 bytes) 000834 1789266445.631854 mapread 0x02234f thru 0x0317e2 (0xf494 bytes) 000837 1789266445.653351 read 0x01a508 thru 0x028cb7 (0xe7b0 bytes) 000838 1789266445.665865 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd7ff6d6 (stale) Mirror 8: 0xd7ff6d6 (stale) Mirror 11: 0xd7ff6d6 (stale) Mirror 12: 0xd7ff6d6 (stale) Mirror 14: 0x8d58247f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd7ff6d6 (stale) Mirror 8: 0xd7ff6d6 (stale) Mirror 11: 0xd7ff6d6 (stale) Mirror 12: 0xd7ff6d6 (stale) Mirror 14: 0x8d58247f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1789266446.306056 write 0x032ddd thru 0x0377cd (0x49f1 bytes) 000840 1789266446.322103 mapread 0x00b7bc thru 0x00c9a6 (0x11eb bytes) 000841 1789266446.331795 read 0x02b1fb thru 0x02f87d (0x4683 bytes) 000842 1789266446.337823 read 0x028e42 thru 0x02e7fb (0x59ba bytes) 000843 1789266446.345884 write 0x01bf33 thru 0x01f3ab (0x3479 bytes) 000845 1789266446.348125 read 0x012fbb thru 0x01c7ea (0x9830 bytes) 000846 1789266446.348994 write 0x018bd3 thru 0x01de91 (0x52bf bytes) 000847 1789266446.350723 trunc from 0x040000 to 0x00603b (0x39fc4 bytes) 000850 1789266446.381795 MIRROR_RESYNC 6 mirrors 000852 1789266447.691474 MIRROR_RESYNC 6 mirrors 000853 1789266448.436184 write 0x0352f3 thru 0x0372d0 (0x1fde bytes) 000854 1789266448.554004 mapread 0x00780f thru 0x01133c (0x9b2e bytes) 000855 1789266448.578334 punch from 0x00e825 to 0x014cce (0x64aa bytes) 000856 1789266448.590311 punch from 0x02c408 to 0x02d377 (0xf70 bytes) 000859 1789266448.604035 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x372d1): Mirror 7: 0xefd011bf (stale) Mirror 8: 0xefd011bf (stale) Mirror 11: 0xefd011bf (stale) Mirror 14: 0xefd011bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x372d1): Mirror 7: 0xefd011bf (stale) Mirror 8: 0xefd011bf (stale) Mirror 11: 0xefd011bf (stale) Mirror 14: 0xefd011bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x372d1: skipped 000860 1789266449.266199 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x372d1): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x372d1): Mirror 7: 0xefd011bf (stale) Mirror 8: 0xefd011bf (stale) Mirror 11: 0xefd011bf (stale) Mirror 14: 0xefd011bf (stale) CRC-32 checksum value for chunk [0, 0x372d1): Mirror 1: 0x760f14af Mirror 15: 0x760f14af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x372d1: skipped 000861 1789266450.617788 trunc from 0x0372d1 to 0x02ad62 (0xc56e bytes) 000862 1789266450.707319 trunc from 0x02ad63 to 0x02116e (0x9bf4 bytes) 000864 1789266450.772862 MIRROR_RESYNC 6 mirrors 000866 1789266452.518219 write 0x01f24f thru 0x02611b (0x6ecd bytes) 000867 1789266452.583184 trunc from 0x02611c to 0x009668 (0x1cab3 bytes) 000868 1789266452.607711 punch from 0x0024bf to 0x009668 (0x71aa bytes) 000870 1789266452.619290 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9669): Mirror 7: 0xc3ff8ca9 (stale) Mirror 8: 0xc3ff8ca9 (stale) Mirror 11: 0xc3ff8ca9 (stale) Mirror 14: 0xc3ff8ca9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9669): Mirror 7: 0xc3ff8ca9 (stale) Mirror 8: 0xc3ff8ca9 (stale) Mirror 11: 0xc3ff8ca9 (stale) Mirror 14: 0xc3ff8ca9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9669: skipped 000871 1789266453.111667 trunc from 0x009669 to 0x034b49 (0x2b4e1 bytes) 000872 1789266453.139166 MIRROR_RESYNC 5 mirrors 000873 1789266454.388207 punch from 0x011edf to 0x0203bc (0xe4de bytes) 000874 1789266454.453928 write 0x004ea7 thru 0x00e89a (0x99f4 bytes) 000875 1789266454.473749 trunc from 0x034b4a to 0x01072e (0x2441b bytes) 000876 1789266454.496287 punch from 0x000690 to 0x009fcb (0x993c bytes) 000878 1789266454.508898 mapwrite 0x002abc thru 0x00f336 (0xc87b bytes) 000882 1789266454.638447 MIRROR_RESYNC 5 mirrors 000883 1789266456.109795 punch from 0x006807 to 0x00db1b (0x7315 bytes) 000884 1789266456.189172 read 0x00219d thru 0x003b0e (0x1972 bytes) 000886 1789266456.197962 punch from 0x00ba88 to 0x00fa12 (0x3f8b bytes) 000887 1789266456.204262 punch from 0x00fb6d to 0x01072e (0xbc2 bytes) 000888 1789266456.210124 trunc from 0x01072f to 0x037e72 (0x27744 bytes) 000889 1789266456.234759 read 0x01eb58 thru 0x02061d (0x1ac6 bytes) 000890 1789266456.260905 trunc from 0x037e73 to 0x00a8c5 (0x2d5ad bytes) 000891 1789266456.288663 punch from 0x0087d8 to 0x00a8c5 (0x20ee bytes) 000893 1789266456.297436 read 0x008f98 thru 0x00a8c5 (0x192e bytes) 000894 1789266456.306959 trunc from 0x00a8c6 to 0x023a6f (0x191aa bytes) 000896 1789266456.332785 mapwrite 0x0126c4 thru 0x018e0e (0x674b bytes) 000897 1789266456.404928 write 0x01c519 thru 0x01fb65 (0x364d bytes) 000899 1789266456.407648 punch from 0x0045a8 to 0x00a68e (0x60e7 bytes) 000900 1789266456.421128 punch from 0x00a54b to 0x016137 (0xbbed bytes) 000904 1789266456.436262 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23a70): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a70): Mirror 7: 0x121e7990 (stale) Mirror 8: 0x121e7990 (stale) Mirror 11: 0x121e7990 (stale) Mirror 14: 0x121e7990 (stale) CRC-32 checksum value for chunk [0, 0x23a70): Mirror 1: 0xa0a93621 Mirror 15: 0xa0a93621 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a70: skipped 000906 1789266457.773381 mapwrite 0x012f55 thru 0x016c2f (0x3cdb bytes) 000907 1789266457.963948 MIRROR_RESYNC 6 mirrors 000908 1789266459.893737 read 0x0079b7 thru 0x011aaf (0xa0f9 bytes) 000909 1789266459.936566 mapread 0x018dcd thru 0x01ce07 (0x403b bytes) 000911 1789266459.984951 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000912 1789266460.696739 read 0x021914 thru 0x023a6f (0x215c bytes) 000913 1789266460.705842 write 0x03daa7 thru 0x03ffff (0x2559 bytes) 000914 1789266460.821644 punch from 0x008020 to 0x00ceaf (0x4e90 bytes) 000915 1789266460.842579 read 0x02c7fb thru 0x036d83 (0xa589 bytes) 000916 1789266460.867530 read 0x01a553 thru 0x020210 (0x5cbe bytes) 000917 1789266460.868496 read 0x00c002 thru 0x00dc14 (0x1c13 bytes) 000918 1789266460.871217 mapwrite 0x034201 thru 0x039733 (0x5533 bytes) 000919 1789266460.937159 mapwrite 0x02099c thru 0x02ec77 (0xe2dc bytes) 000922 1789266461.014510 mapwrite 0x00bf1d thru 0x015cb6 (0x9d9a bytes) 000924 1789266461.136916 mapread 0x036a28 thru 0x03cbed (0x61c6 bytes) 000925 1789266461.145119 mapread 0x00cb29 thru 0x00f539 (0x2a11 bytes) 000926 1789266461.152304 MIRROR_RESYNC 5 mirrors 000927 1789266462.358105 punch from 0x007c51 to 0x00b748 (0x3af8 bytes) 000928 1789266462.469849 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6cd298a6 (stale) Mirror 8: 0x6cd298a6 (stale) Mirror 14: 0x6cd298a6 (stale) Mirror 15: 0x6cd298a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6cd298a6 Mirror 16: 0x6cd298a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1789266463.720663 mapwrite 0x01205e thru 0x015142 (0x30e5 bytes) 000930 1789266463.884473 MIRROR_RESYNC 6 mirrors 000931 1789266465.245077 trunc from 0x040000 to 0x032ce6 (0xd319 bytes) 000932 1789266465.330497 mapwrite 0x02a5a5 thru 0x039b56 (0xf5b2 bytes) 000934 1789266465.488125 read 0x00045e thru 0x00bedd (0xba80 bytes) 000935 1789266465.504262 MIRROR_RESYNC 6 mirrors 000936 1789266466.771277 punch from 0x00c8a1 to 0x00effc (0x275c bytes) 000937 1789266466.859314 mapread 0x00a5f2 thru 0x0111fd (0x6c0c bytes) 000939 1789266466.886278 trunc from 0x039b57 to 0x036de3 (0x2d73 bytes) 000941 1789266466.909679 trunc from 0x036de4 to 0x02ea12 (0x83d1 bytes) 000942 1789266466.931073 write 0x0107b8 thru 0x012852 (0x209b bytes) 000943 1789266466.932203 mapwrite 0x0365cc thru 0x03c38d (0x5dc2 bytes) 000944 1789266467.040597 read 0x031976 thru 0x03a742 (0x8dcd bytes) 000945 1789266467.041464 mapread 0x015703 thru 0x0187b3 (0x30b1 bytes) 000947 1789266467.055559 mapwrite 0x005e9b thru 0x0108b5 (0xaa1b bytes) 000948 1789266467.113577 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c38e): Mirror 7: 0x4ae1c0ce (stale) Mirror 8: 0x4ae1c0ce (stale) Mirror 14: 0x4ae1c0ce (stale) Mirror 15: 0x4ae1c0ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c38e): Mirror 7: 0x4ae1c0ce (stale) Mirror 8: 0x4ae1c0ce (stale) Mirror 14: 0x4ae1c0ce (stale) Mirror 15: 0x4ae1c0ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c38e: skipped 000949 1789266467.476696 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c38e): Mirror 7: 0x4ae1c0ce (stale) Mirror 8: 0x4ae1c0ce (stale) Mirror 15: 0x4ae1c0ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c38e): Mirror 7: 0x4ae1c0ce (stale) Mirror 8: 0x4ae1c0ce (stale) Mirror 15: 0x4ae1c0ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c38e: skipped 000951 1789266467.865875 trunc from 0x03c38e to 0x02e313 (0xe07a bytes) 000952 1789266467.888996 read 0x01a8b0 thru 0x02395f (0x90b0 bytes) 000953 1789266467.900323 read 0x001960 thru 0x001e05 (0x4a6 bytes) 000955 1789266467.929161 mapread 0x006588 thru 0x011e09 (0xb882 bytes) 000956 1789266467.930196 read 0x02c4a0 thru 0x02e313 (0x1e74 bytes) 000960 1789266467.938046 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e314): Mirror 7: 0xcff7a199 (stale) Mirror 8: 0xcff7a199 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e314): Mirror 7: 0xcff7a199 (stale) Mirror 8: 0xcff7a199 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e314: skipped 000961 1789266468.413837 mapwrite 0x007eb6 thru 0x017bc3 (0xfd0e bytes) 000962 1789266468.515187 mapread 0x027353 thru 0x02e313 (0x6fc1 bytes) 000965 1789266468.528171 punch from 0x0091f2 to 0x011098 (0x7ea7 bytes) 000966 1789266468.533934 read 0x00abcc thru 0x010f4a (0x637f bytes) 000967 1789266468.548522 trunc from 0x02e314 to 0x031a58 (0x3745 bytes) 000968 1789266468.563966 mapread 0x017e4e thru 0x01bb93 (0x3d46 bytes) 000969 1789266468.576193 punch from 0x00b22f to 0x01a35b (0xf12d bytes) 000970 1789266468.581449 trunc from 0x031a59 to 0x018961 (0x190f7 bytes) 000972 1789266468.604649 mapread 0x002fa0 thru 0x00dec9 (0xaf2a bytes) 000973 1789266468.618036 punch from 0x00fe86 to 0x011074 (0x11ef bytes) 000974 1789266468.623589 write 0x00556f thru 0x00a513 (0x4fa5 bytes) 000975 1789266468.630681 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18962): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18962): Mirror 7: 0x2c67cbdc (stale) Mirror 8: 0x2c67cbdc (stale) CRC-32 checksum value for chunk [0, 0x18962): Mirror 1: 0x64626514 Mirror 9: 0x64626514 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18962: skipped 000976 1789266469.629422 read 0x00dbdd thru 0x00e9bd (0xde1 bytes) 000978 1789266469.646306 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x18962): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18962): Mirror 7: 0x2c67cbdc (stale) Mirror 8: 0x2c67cbdc (stale) CRC-32 checksum value for chunk [0, 0x18962): Mirror 1: 0x64626514 Mirror 9: 0x64626514 Mirror 10: 0x64626514 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18962: skipped 000979 1789266470.631678 write 0x006ef2 thru 0x0120f4 (0xb203 bytes) 000981 1789266470.729750 trunc from 0x018962 to 0x03d085 (0x24724 bytes) 000983 1789266470.757016 trunc from 0x03d086 to 0x02a8ec (0x12799 bytes) 000984 1789266470.773949 mapwrite 0x03b1a7 thru 0x03ffff (0x4e59 bytes) 000985 1789266470.874795 write 0x0126ed thru 0x01f113 (0xca27 bytes) 000987 1789266470.881890 trunc from 0x040000 to 0x001499 (0x3eb66 bytes) 000988 1789266470.904154 trunc from 0x00149a to 0x0392e8 (0x37e4f bytes) 000990 1789266470.929530 write 0x00001f thru 0x006f19 (0x6efb bytes) 000991 1789266470.950151 mapwrite 0x01dfeb thru 0x02b556 (0xd56c bytes) 000992 1789266471.076373 trunc from 0x0392e9 to 0x039471 (0x189 bytes) 000994 1789266471.105902 read 0x01bee2 thru 0x027dce (0xbeed bytes) 000998 1789266471.113601 mapread 0x00c689 thru 0x01c202 (0xfb7a bytes) 000999 1789266471.142852 mapread 0x02e6a7 thru 0x0350ba (0x6a14 bytes) All operations completed A-OK! PASS 70a (155s) == sanity-flr test 71: check mirror extend parent fid ==== 22:27:57 (1789266477) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 22:28:01 (1789266481) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.398328 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284995 s, 3.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 22:28:11 (1789266491) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3764320 .. Extending file size to 5898080 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6026752 .. Extending file size to 5035456 .. Extending file size to 6250752 .. Extending file size to 8477760 .. Extending file size to 8259200 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8857088 .. Extending file size to 8539840 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8555456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8949408 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9114816 .. Extending file size to 8851936 .. Waiting 57423 57424 57426 57427 57430 done 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:29:30 (1789266570) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5255200 .. Extending file size to 7339392 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6364960 .. Extending file size to 9068000 .. Extending file size to 8145152 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8673600 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8695168 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58947 58949 58950 58951 58952 done 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:30:48 (1789266648) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132614 s, 15.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.346635 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xfe17e492 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:31:08 (1789266668) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124849 s, 16.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 22:31:14 (1789266674) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:31:21 (1789266681) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218676 s, 9.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:31:29 (1789266689) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220386 s, 9.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:31:36 (1789266696) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31024 s, 6.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:32:09 (1789266729) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166453 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:33:08 (1789266788) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.302546 s, 6.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:34:06 (1789266846) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Creating new pool test_204f oleg457-server: Pool lustre.test_204f created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33406 s, 6.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg457-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:35:27 (1789266927) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:35:34 (1789266934) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:35:42 (1789266942) oleg457-server: Pool lustre.test_206 created Adding targets to pool oleg457-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg457-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg457-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg457-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:36:09 (1789266969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734953 s, 14.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592194 s, 17.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x90465c41 Mirror 3: 0x90465c41 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:36:22 (1789266982) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:36:28 (1789266988) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:36:32 (1789266992) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.13676 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ce0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1390000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2260000 Read this (4 bytes): BBBB 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:37:10 (1789267030) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0202469 s, 405 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:37:17 (1789267037) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580056 s, 18.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834447 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 22:37:19.000000000 -0400 Modify: 2026-09-12 22:37:19.000000000 -0400 Change: 2026-09-12 22:37:21.000000000 -0400 Birth: 2026-09-12 22:37:20.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:37:26 (1789267046) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:37:29 (1789267049) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.55745 s, 73.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0685357 s, 59.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 12 22:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 12 22:37 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:37:38 (1789267058) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:37:46 (1789267066) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1848 sec =========== 22:37:56 (1789267076) === sanity-flr: start cleanup 22:37:57 (1789267077) === === sanity-flr: finish cleanup 22:38:05 (1789267085) ===