-----============= acceptance-small: sanity-flr ============----- Thu May 28 09:29:15 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg131-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:29:39 (1779974979) === oleg131-client.virtnet: executing check_config_client /mnt/lustre oleg131-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg131-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92ef85339800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92ef85339800.idle_timeout=debug disable quota as required oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:30:14 (1779975014) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7576] == sanity-flr test 0a: lfs mirror create with -N option == 09:30:17 (1779975017) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:30:36 (1779975036) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:30:39 (1779975039) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:30:43 (1779975043) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:31:08 (1779975068) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:31:12 (1779975072) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:31:15 (1779975075) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92ef85339800.stats=clear osc.lustre-OST0001-osc-ffff92ef85339800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:31:23 (1779975083) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:31:34 (1779975094) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:31:43 (1779975103) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:31:47 (1779975107) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:31:56 (1779975116) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:32:02 (1779975122) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:32:06 (1779975126) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:32:15 (1779975135) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:32:25 (1779975145) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:32:33 (1779975153) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.2499 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144522 s, 7.3 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:32:45 (1779975165) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.209006 s, 20.1 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:32:54 (1779975174) Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:34:07 (1779975247) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.29134 s, 14.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:34:48 (1779975288) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:36:10 (1779975370) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.23252 s, 7.2 MB/s -rw-r--r-- 1 root root 52428800 May 28 09:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.333s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.186s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg131-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg131-server oleg131-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.207s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg131-server: oleg131-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg131-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg131-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92ef85339800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:37:44 (1779975464) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:37:48 (1779975468) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:37:51 (1779975471) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:37:54 (1779975474) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:38:02 (1779975482) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.490258 s, 25.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:38:15 (1779975495) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311592 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568698 s, 18.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:38:23 (1779975503) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:38:32 (1779975512) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00439094 s, 233 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130148 s, 787 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00150056 s, 682 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000987452 s, 1.0 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:38:42 (1779975522) create test files with size 15731238 .. 493bd74ddc6d81317c6d355858f0cf12 - 211ad629b6ab51bdbcf0d0cfc748fd31 - 157397033bba2b9572afb12ec29a9120 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 09:39:06 (1779975546) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:39:22 (1779975562) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:39:30 (1779975570) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133755 s, 7.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328284 s, 31.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 09:39:39 (1779975579) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121452 s, 17.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.414727 s, 10.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244087 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 09:39:59 (1779975599) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:40:02 (1779975602) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:40:06 (1779975606) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669004 s, 15.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 09:40:14 (1779975614) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:40:18 (1779975618) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:40:29 (1779975629) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.315305 s, 33.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:40:41 (1779975641) llite.lustre-ffff92ef83046000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.150021 s, 69.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg131-client.virtnet /mnt/lustre (opts:) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff92ef92369000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:40:56 (1779975656) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:41:42 (1779975702) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:42:24 (1779975744) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:42:28 (1779975748) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223592 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.141065 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x986b940d (stale) Mirror 2: 0x986b940d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: bf02cb37a2f75767363b4c7ad561f5e7 - mirror 1 checksum: d35bfd989ff98b9a8200eb7fe93cbd0e - mirror 2 checksum: bf02cb37a2f75767363b4c7ad561f5e7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 28 09:42:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 28 09:42:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bf02cb37a2f75767363b4c7ad561f5e7 - mirror 1 checksum: d35bfd989ff98b9a8200eb7fe93cbd0e - mirror 2 checksum: d35bfd989ff98b9a8200eb7fe93cbd0e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 28 09:42:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 28 09:42:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:42:40 (1779975760) Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 09:42:47 (1779975767) shut down facet: mds1 facet_host: oleg131-server facet_failover_host: oleg131-server Failover mds1 to oleg131-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 09:43:20 (1779975800) targets are mounted 09:43:20 (1779975800) facet_failover done oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283824 s, 36.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:43:39 (1779975819) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:43:44 (1779975824) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:43:48 (1779975828) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0120401 s, 1.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:43:56 (1779975836) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:44:00 (1779975840) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:44:05 (1779975845) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:44:29 (1779975869) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:44:59 (1779975899) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:45:23 (1779975923) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:45:48 (1779975948) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 09:45:56 (1779975956) mirror create pid 52342 mirror split pid 52343 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7703: 52343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 52342 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:47:05 (1779976025) Seed set to 6027 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1779976027.810158 write 0x00242e thru 0x00cd7c (0xa94f bytes) 000003 1779976027.819082 mapwrite 0x02f1bb thru 0x033202 (0x4048 bytes) 000004 1779976028.013685 mapread 0x011745 thru 0x01ea77 (0xd333 bytes) 000006 1779976028.048692 mapwrite 0x03c360 thru 0x03caf6 (0x797 bytes) 000008 1779976028.157261 mapread 0x023391 thru 0x02ad38 (0x79a8 bytes) 000009 1779976028.171119 write 0x017d34 thru 0x01d29f (0x556c bytes) 000010 1779976028.174880 mapread 0x010eee thru 0x01b0d9 (0xa1ec bytes) 000011 1779976028.177060 read 0x021111 thru 0x02fe3e (0xed2e bytes) 000012 1779976028.178186 mapread 0x0360b9 thru 0x037c80 (0x1bc8 bytes) 000013 1779976028.188871 read 0x001306 thru 0x00f05d (0xdd58 bytes) truncating to largest ever: 0x13525 000015 1779976028.202836 trunc from 0x03caf7 to 0x013524 (0x295d2 bytes) 000017 1779976028.272304 mapwrite 0x03930b thru 0x03ffff (0x6cf5 bytes) 000022 1779976028.510440 mapread 0x03abe9 thru 0x03ffff (0x5417 bytes) 000023 1779976028.513190 read 0x006444 thru 0x013877 (0xd434 bytes) 000024 1779976028.544306 MIRROR_EXTEND to 2 mirrors 000029 1779976029.850977 write 0x03f032 thru 0x03ffff (0xfce bytes) 000031 1779976030.006578 mapread 0x0179d6 thru 0x01e0ea (0x6715 bytes) 000032 1779976030.056749 read 0x019df7 thru 0x01fc48 (0x5e52 bytes) 000033 1779976030.069936 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdd2444e3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2674429e Mirror 3: 0x2674429e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000035 1779976031.197227 mapwrite 0x0093a9 thru 0x00f63e (0x6296 bytes) 000036 1779976031.319667 mapread 0x0125b8 thru 0x01f0c5 (0xcb0e bytes) 000037 1779976031.340180 mapwrite 0x023d6e thru 0x029975 (0x5c08 bytes) truncating to largest ever: 0x18dc5 000038 1779976031.386327 trunc from 0x040000 to 0x018dc4 (0x2723b bytes) 000039 1779976031.410022 mapread 0x0109a6 thru 0x018dc4 (0x841f bytes) 000041 1779976031.412031 mapread 0x004fff thru 0x00c7a3 (0x77a5 bytes) 000042 1779976031.417131 mapwrite 0x03d55e thru 0x03ffff (0x2aa2 bytes) 000043 1779976031.471808 mapread 0x011279 thru 0x01e814 (0xd59c bytes) 000045 1779976031.484014 mapread 0x0149c2 thru 0x01cc29 (0x8268 bytes) 000046 1779976031.485262 mapread 0x000af2 thru 0x004457 (0x3966 bytes) truncating to largest ever: 0x3f192 000047 1779976031.493309 trunc from 0x040000 to 0x03f191 (0xe6e bytes) 000048 1779976031.511116 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f192): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 2: 0x8b0f9773 (stale) Mirror 3: 0x57e1770 (stale) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 1: 0xd5fbca8b Mirror 4: 0xd5fbca8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000049 1779976032.425901 mapread 0x006a26 thru 0x00a3e6 (0x39c1 bytes) 000051 1779976032.476907 mapread 0x037097 thru 0x03a745 (0x36af bytes) 000054 1779976032.583513 write 0x01767a thru 0x01b6a2 (0x4029 bytes) 000055 1779976032.918160 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f192): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 2: 0x8b0f9773 (stale) Mirror 3: 0x57e1770 (stale) Mirror 4: 0xd5fbca8b (stale) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 1: 0xdc66aac2 Mirror 5: 0xdc66aac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000056 1779976034.620737 mapwrite 0x01af49 thru 0x01f26b (0x4323 bytes) 000058 1779976035.192240 read 0x02eafc thru 0x035bd7 (0x70dc bytes) 000060 1779976035.204921 write 0x01050d thru 0x017786 (0x727a bytes) 000062 1779976035.242513 MIRROR_RESYNC 5 mirrors 000065 1779976037.007055 mapwrite 0x02d308 thru 0x03c984 (0xf67d bytes) 000066 1779976037.219172 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: 0x469cc6d6 (stale) Mirror 4: 0x469cc6d6 (stale) Mirror 5: 0x469cc6d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: 0x469cc6d6 (stale) Mirror 4: 0x469cc6d6 (stale) Mirror 5: 0x469cc6d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000067 1779976037.929133 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: 0x469cc6d6 (stale) Mirror 4: 0x469cc6d6 (stale) Mirror 5: 0x469cc6d6 (stale) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 1: 0xf870bc51 Mirror 6: 0xf870bc51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000068 1779976039.257139 mapwrite 0x00c4ac thru 0x01a6fc (0xe251 bytes) 000070 1779976039.692501 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: 0x469cc6d6 (stale) Mirror 4: 0x469cc6d6 (stale) Mirror 5: 0x469cc6d6 (stale) Mirror 6: 0xf870bc51 (stale) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 1: 0xf5d93f6b Mirror 7: 0xf5d93f6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000072 1779976041.316508 mapwrite 0x00bc31 thru 0x00cf25 (0x12f5 bytes) 000073 1779976041.629896 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f192): Mirror 3: 0x469cc6d6 (stale) Mirror 4: 0x469cc6d6 (stale) Mirror 5: 0x469cc6d6 (stale) Mirror 6: 0xf870bc51 (stale) Mirror 7: 0xf5d93f6b (stale) CRC-32 checksum value for chunk [0, 0x3f192): Mirror 1: 0x7fb2ccd7 Mirror 8: 0x7fb2ccd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f192: skipped 000076 1779976043.754037 read 0x0002a9 thru 0x000300 (0x58 bytes) 000077 1779976043.774189 read 0x035802 thru 0x03a609 (0x4e08 bytes) 000078 1779976043.803812 write 0x033065 thru 0x036c8e (0x3c2a bytes) 000079 1779976043.986406 mapwrite 0x01e16a thru 0x028fb7 (0xae4e bytes) 000080 1779976044.305528 mapread 0x00a3da thru 0x0161a1 (0xbdc8 bytes) 000081 1779976044.478625 trunc from 0x03f192 to 0x03c4f8 (0x2c99 bytes) 000084 1779976044.527593 mapwrite 0x003802 thru 0x008833 (0x5032 bytes) 000086 1779976044.778681 read 0x010d8a thru 0x01ee8c (0xe103 bytes) 000087 1779976044.810387 mapwrite 0x00583a thru 0x012028 (0xc7ef bytes) 000090 1779976044.943919 write 0x003431 thru 0x00439c (0xf6c bytes) 000091 1779976044.950117 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c4f9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4f9): Mirror 3: 0xddd35c94 (stale) Mirror 4: 0xddd35c94 (stale) Mirror 5: 0xddd35c94 (stale) Mirror 6: 0x83e4481 (stale) Mirror 7: 0x8ee79a9e (stale) Mirror 8: 0x1cd4469a (stale) CRC-32 checksum value for chunk [0, 0x3c4f9): Mirror 1: 0xe5a3a76e Mirror 9: 0xe5a3a76e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4f9: skipped 000092 1779976046.864051 read 0x00ccec thru 0x01c6a0 (0xf9b5 bytes) 000094 1779976046.939819 trunc from 0x03c4f9 to 0x0020eb (0x3a40d bytes) 000095 1779976047.175730 write 0x00879f thru 0x00c6c8 (0x3f2a bytes) 000096 1779976047.204526 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc6c9): Mirror 4: 0xdaa06ad5 (stale) Mirror 5: 0xdaa06ad5 (stale) Mirror 6: 0xdaa06ad5 (stale) Mirror 7: 0xc42337fc (stale) Mirror 8: 0x3d7205ca (stale) Mirror 9: 0x5c87f0a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6c9): Mirror 4: 0xdaa06ad5 (stale) Mirror 5: 0xdaa06ad5 (stale) Mirror 6: 0xdaa06ad5 (stale) Mirror 7: 0xc42337fc (stale) Mirror 8: 0x3d7205ca (stale) Mirror 9: 0x5c87f0a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6c9: skipped 000098 1779976048.312596 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xc6c9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6c9): Mirror 4: 0xdaa06ad5 (stale) Mirror 5: 0xdaa06ad5 (stale) Mirror 6: 0xdaa06ad5 (stale) Mirror 7: 0xc42337fc (stale) Mirror 8: 0x3d7205ca (stale) Mirror 9: 0x5c87f0a8 (stale) CRC-32 checksum value for chunk [0, 0xc6c9): Mirror 1: 0xfcb30cf2 Mirror 10: 0xfcb30cf2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6c9: skipped 000099 1779976050.153737 trunc from 0x00c6c9 to 0x02aebb (0x1e7f3 bytes) 000100 1779976050.355583 trunc from 0x02aebc to 0x037ca6 (0xcdeb bytes) 000102 1779976050.396413 mapwrite 0x03a13f thru 0x03ffff (0x5ec1 bytes) 000104 1779976050.579466 read 0x02b04d thru 0x02d437 (0x23eb bytes) 000105 1779976050.593251 write 0x031ee5 thru 0x03ffff (0xe11b bytes) 000110 1779976050.636906 mapwrite 0x00f4de thru 0x015bb5 (0x66d8 bytes) 000111 1779976050.797260 mapread 0x00ddb7 thru 0x011149 (0x3393 bytes) 000116 1779976050.804527 read 0x0241ec thru 0x02a5c7 (0x63dc bytes) 000120 1779976050.842034 write 0x0390c7 thru 0x03ffff (0x6f39 bytes) 000122 1779976050.852956 write 0x02f7f0 thru 0x035495 (0x5ca6 bytes) 000123 1779976050.857262 read 0x003d1c thru 0x00687b (0x2b60 bytes) 000124 1779976050.863581 mapread 0x004f11 thru 0x014eba (0xffaa bytes) 000125 1779976050.865444 trunc from 0x040000 to 0x01615d (0x29ea2 bytes) 000126 1779976050.927507 mapwrite 0x01cdcc thru 0x022913 (0x5b48 bytes) 000127 1779976051.197125 write 0x016479 thru 0x025d5d (0xf8e5 bytes) 000131 1779976051.208018 mapread 0x00e2ab thru 0x0123b2 (0x4108 bytes) 000132 1779976051.217331 write 0x025e90 thru 0x02cde6 (0x6f57 bytes) 000133 1779976051.218802 mapread 0x0263cf thru 0x02c6f2 (0x6324 bytes) 000136 1779976051.233922 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 4: 0x29df4e7 (stale) Mirror 5: 0x29df4e7 (stale) Mirror 6: 0x29df4e7 (stale) Mirror 7: 0xa3b8dd87 (stale) Mirror 8: 0x2b14ad5d (stale) Mirror 9: 0xdc90e508 (stale) Mirror 10: 0xf4328c9c (stale) CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 1: 0xf5c527b4 Mirror 11: 0xf5c527b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cde7: skipped 000138 1779976052.823701 write 0x009692 thru 0x016632 (0xcfa1 bytes) 000139 1779976052.992315 read 0x029498 thru 0x02cde6 (0x394f bytes) 000140 1779976053.002617 read 0x015858 thru 0x01a7ab (0x4f54 bytes) 000142 1779976053.021299 mapread 0x00f35f thru 0x014b99 (0x583b bytes) 000143 1779976053.023314 read 0x0091d9 thru 0x016b7d (0xd9a5 bytes) 000145 1779976053.025865 read 0x00d979 thru 0x00ef6d (0x15f5 bytes) 000146 1779976053.030788 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 4: 0x29df4e7 (stale) Mirror 5: 0x29df4e7 (stale) Mirror 6: 0x29df4e7 (stale) Mirror 7: 0xa3b8dd87 (stale) Mirror 8: 0x2b14ad5d (stale) Mirror 9: 0xdc90e508 (stale) Mirror 10: 0xf4328c9c (stale) Mirror 11: 0xf5c527b4 (stale) CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 1: 0x8a78e44b Mirror 12: 0x8a78e44b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cde7: skipped 000147 1779976054.854484 write 0x014173 thru 0x01aa10 (0x689e bytes) 000149 1779976055.045651 mapread 0x007cc7 thru 0x008d24 (0x105e bytes) 000150 1779976055.061747 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 5: 0x29df4e7 (stale) Mirror 6: 0x29df4e7 (stale) Mirror 7: 0xa3b8dd87 (stale) Mirror 8: 0x2b14ad5d (stale) Mirror 9: 0xdc90e508 (stale) Mirror 10: 0xf4328c9c (stale) Mirror 11: 0xf5c527b4 (stale) Mirror 12: 0x8a78e44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cde7): Mirror 5: 0x29df4e7 (stale) Mirror 6: 0x29df4e7 (stale) Mirror 7: 0xa3b8dd87 (stale) Mirror 8: 0x2b14ad5d (stale) Mirror 9: 0xdc90e508 (stale) Mirror 10: 0xf4328c9c (stale) Mirror 11: 0xf5c527b4 (stale) Mirror 12: 0x8a78e44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cde7: skipped 000151 1779976055.785452 trunc from 0x02cde7 to 0x0101a7 (0x1cc3f bytes) 000153 1779976055.798469 mapwrite 0x02990a thru 0x035459 (0xbb50 bytes) 000154 1779976056.001178 mapread 0x02d873 thru 0x035459 (0x7be7 bytes) 000155 1779976056.002438 read 0x00fb7b thru 0x0164b3 (0x6939 bytes) 000156 1779976056.017662 read 0x008a41 thru 0x01393f (0xaeff bytes) 000159 1779976056.025540 mapwrite 0x01c70f thru 0x029e46 (0xd738 bytes) 000162 1779976056.129109 write 0x03e506 thru 0x03ffff (0x1afa bytes) 000163 1779976056.133094 write 0x01888c thru 0x02326a (0xa9df bytes) 000164 1779976056.135887 read 0x015e58 thru 0x0189e2 (0x2b8b bytes) 000166 1779976056.166316 read 0x00d2c7 thru 0x010f5f (0x3c99 bytes) 000169 1779976056.174496 write 0x03d665 thru 0x03ffff (0x299b bytes) 000170 1779976056.199322 mapwrite 0x005543 thru 0x00ac65 (0x5723 bytes) 000171 1779976056.331553 read 0x01cb46 thru 0x027be9 (0xb0a4 bytes) 000173 1779976056.353375 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5cf7da1a (stale) Mirror 7: 0xa66f0dca (stale) Mirror 8: 0xfb70debe (stale) Mirror 9: 0x6602f355 (stale) Mirror 10: 0x296537d8 (stale) Mirror 11: 0xceb833dd (stale) Mirror 12: 0x895345b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5cf7da1a (stale) Mirror 7: 0xa66f0dca (stale) Mirror 8: 0xfb70debe (stale) Mirror 9: 0x6602f355 (stale) Mirror 10: 0x296537d8 (stale) Mirror 11: 0xceb833dd (stale) Mirror 12: 0x895345b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000176 1779976057.055208 mapwrite 0x01d9f9 thru 0x02c1bb (0xe7c3 bytes) 000177 1779976057.276833 read 0x0022e8 thru 0x0085bf (0x62d8 bytes) 000179 1779976057.314633 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5cf7da1a (stale) Mirror 7: 0xa66f0dca (stale) Mirror 8: 0xfb70debe (stale) Mirror 9: 0x6602f355 (stale) Mirror 10: 0x296537d8 (stale) Mirror 11: 0xceb833dd (stale) Mirror 12: 0x895345b4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xda4443 Mirror 13: 0xda4443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000181 1779976058.994495 write 0x01f80f thru 0x02418e (0x4980 bytes) 000182 1779976059.169228 read 0x00ab8e thru 0x0119bd (0x6e30 bytes) 000184 1779976059.187116 write 0x003d4f thru 0x00ccf4 (0x8fa6 bytes) 000185 1779976059.206094 read 0x00bdae thru 0x01556f (0x97c2 bytes) 000187 1779976059.227790 mapread 0x02a58e thru 0x032d98 (0x880b bytes) 000188 1779976059.229626 mapwrite 0x0209b8 thru 0x029d90 (0x93d9 bytes) 000190 1779976059.379399 MIRROR_RESYNC 9 mirrors 000191 1779976062.164430 trunc from 0x040000 to 0x00a34c (0x35cb3 bytes) 000192 1779976062.416359 trunc from 0x00a34d to 0x0077f0 (0x2b5c bytes) 000193 1779976062.467402 mapwrite 0x00a0c3 thru 0x0150dd (0xb01b bytes) 000194 1779976062.617421 mapread 0x002ba5 thru 0x00c74b (0x9ba7 bytes) 000195 1779976062.623640 mapwrite 0x021172 thru 0x02a00a (0x8e99 bytes) 000196 1779976062.698910 trunc from 0x02a00b to 0x0084f6 (0x21b14 bytes) 000199 1779976062.730937 read 0x005122 thru 0x0084f6 (0x33d5 bytes) 000200 1779976062.742169 mapread 0x006b9f thru 0x0084f6 (0x1958 bytes) 000202 1779976062.744798 mapwrite 0x00c3f0 thru 0x019df6 (0xda07 bytes) 000203 1779976062.834062 read 0x0002dd thru 0x0024c1 (0x21e5 bytes) 000206 1779976062.842260 trunc from 0x019df7 to 0x02b1e8 (0x113f2 bytes) 000207 1779976062.857672 write 0x012e7d thru 0x020d67 (0xdeeb bytes) 000209 1779976062.866110 read 0x011019 thru 0x01d95d (0xc945 bytes) 000210 1779976062.867388 trunc from 0x02b1e9 to 0x02b823 (0x63b bytes) 000211 1779976062.891239 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b824): Mirror 6: 0x3b33d1f5 (stale) Mirror 7: 0x3b33d1f5 (stale) Mirror 8: 0x3b33d1f5 (stale) Mirror 9: 0x3b33d1f5 (stale) Mirror 10: 0x3b33d1f5 (stale) Mirror 11: 0x3b33d1f5 (stale) Mirror 13: 0x3b33d1f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b824): Mirror 6: 0x3b33d1f5 (stale) Mirror 7: 0x3b33d1f5 (stale) Mirror 8: 0x3b33d1f5 (stale) Mirror 9: 0x3b33d1f5 (stale) Mirror 10: 0x3b33d1f5 (stale) Mirror 11: 0x3b33d1f5 (stale) Mirror 13: 0x3b33d1f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b824: skipped 000213 1779976063.641307 trunc from 0x02b824 to 0x01af15 (0x1090e bytes) 000214 1779976063.712057 write 0x038f1a thru 0x03ffff (0x70e6 bytes) 000215 1779976063.716988 trunc from 0x040000 to 0x007122 (0x38edd bytes) 000216 1779976063.768588 write 0x026c43 thru 0x032441 (0xb7ff bytes) 000217 1779976063.780367 trunc from 0x032442 to 0x03d732 (0xb2f1 bytes) 000218 1779976063.808070 mapwrite 0x02c2e7 thru 0x032842 (0x655c bytes) 000219 1779976064.027983 trunc from 0x03d733 to 0x00d3fc (0x30336 bytes) 000220 1779976064.084511 write 0x007006 thru 0x014f2f (0xdf2a bytes) 000221 1779976064.113456 trunc from 0x014f30 to 0x0177a8 (0x2879 bytes) 000223 1779976064.167233 trunc from 0x0177a9 to 0x027b8f (0x103e7 bytes) 000224 1779976064.225177 read 0x00d59f thru 0x01903d (0xba9f bytes) 000227 1779976064.250397 mapwrite 0x024555 thru 0x02713b (0x2be7 bytes) 000228 1779976064.453465 trunc from 0x027b90 to 0x017090 (0x10aff bytes) 000229 1779976064.497318 trunc from 0x017091 to 0x005071 (0x1201f bytes) 000232 1779976064.564605 mapread 0x002996 thru 0x005071 (0x26dc bytes) 000233 1779976064.592112 mapread 0x0008d7 thru 0x005071 (0x479b bytes) 000235 1779976064.598450 trunc from 0x005072 to 0x0351d5 (0x30164 bytes) 000237 1779976064.624128 mapwrite 0x00ce6e thru 0x010716 (0x38a9 bytes) 000239 1779976064.718184 mapread 0x01d169 thru 0x02a13c (0xcfd4 bytes) 000241 1779976064.746243 trunc from 0x0351d6 to 0x03e9cf (0x97fa bytes) 000244 1779976064.784305 write 0x01a728 thru 0x0285be (0xde97 bytes) 000246 1779976064.792068 read 0x0383ad thru 0x03ac81 (0x28d5 bytes) 000247 1779976064.805344 mapwrite 0x001742 thru 0x00800e (0x68cd bytes) 000248 1779976064.869076 mapwrite 0x0362ae thru 0x03ffff (0x9d52 bytes) 000249 1779976064.963262 read 0x021aef thru 0x02478f (0x2ca1 bytes) 000250 1779976064.964011 mapread 0x00b756 thru 0x01af76 (0xf821 bytes) 000251 1779976064.965322 mapread 0x00a87a thru 0x00c107 (0x188e bytes) 000255 1779976064.972700 read 0x012c9a thru 0x021c1a (0xef81 bytes) 000256 1779976064.974179 mapwrite 0x03aee0 thru 0x03ffff (0x5120 bytes) 000257 1779976065.040319 mapwrite 0x02e86d thru 0x03cfc4 (0xe758 bytes) 000259 1779976065.101877 trunc from 0x040000 to 0x01b45c (0x24ba3 bytes) 000263 1779976065.131183 mapread 0x012338 thru 0x0188a3 (0x656c bytes) 000264 1779976065.145637 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 6: 0x5e3f6a0b (stale) Mirror 7: 0x5e3f6a0b (stale) Mirror 8: 0x5e3f6a0b (stale) Mirror 9: 0x5e3f6a0b (stale) Mirror 10: 0x5e3f6a0b (stale) Mirror 11: 0x5e3f6a0b (stale) Mirror 13: 0x5e3f6a0b (stale) CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 1: 0xa4839eda Mirror 14: 0xa4839eda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b45d: skipped 000266 1779976066.478955 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 7: 0x5e3f6a0b (stale) Mirror 8: 0x5e3f6a0b (stale) Mirror 9: 0x5e3f6a0b (stale) Mirror 10: 0x5e3f6a0b (stale) Mirror 11: 0x5e3f6a0b (stale) Mirror 13: 0x5e3f6a0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 7: 0x5e3f6a0b (stale) Mirror 8: 0x5e3f6a0b (stale) Mirror 9: 0x5e3f6a0b (stale) Mirror 10: 0x5e3f6a0b (stale) Mirror 11: 0x5e3f6a0b (stale) Mirror 13: 0x5e3f6a0b (stale) CRC-32 checksum value for chunk [0, 0x1b45d): Mirror 1: 0xa4839eda Mirror 14: 0xa4839eda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b45d: skipped 000267 1779976067.867010 read 0x00feb5 thru 0x01b45c (0xb5a8 bytes) 000268 1779976067.907150 mapread 0x0009c4 thru 0x00c357 (0xb994 bytes) 000270 1779976068.006026 trunc from 0x01b45d to 0x022b26 (0x76ca bytes) 000271 1779976068.217621 write 0x00ef0a thru 0x01d404 (0xe4fb bytes) 000273 1779976068.255480 mapread 0x01788c thru 0x01f175 (0x78ea bytes) 000274 1779976068.277046 write 0x0333cf thru 0x035d1b (0x294d bytes) 000278 1779976068.281436 mapread 0x0080f8 thru 0x008799 (0x6a2 bytes) 000279 1779976068.282931 trunc from 0x035d1c to 0x02a6b4 (0xb667 bytes) 000280 1779976068.332654 trunc from 0x02a6b5 to 0x027bf1 (0x2ac3 bytes) 000283 1779976068.355701 read 0x00df7b thru 0x01a434 (0xc4ba bytes) 000284 1779976068.356468 read 0x005edc thru 0x0084e2 (0x2607 bytes) 000285 1779976068.364709 read 0x016baa thru 0x023e1d (0xd274 bytes) 000286 1779976068.373046 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27bf2): Mirror 8: 0xa85766d3 (stale) Mirror 9: 0xa85766d3 (stale) Mirror 10: 0xa85766d3 (stale) Mirror 11: 0xa85766d3 (stale) Mirror 13: 0xa85766d3 (stale) Mirror 14: 0x59646bb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27bf2): Mirror 8: 0xa85766d3 (stale) Mirror 9: 0xa85766d3 (stale) Mirror 10: 0xa85766d3 (stale) Mirror 11: 0xa85766d3 (stale) Mirror 13: 0xa85766d3 (stale) Mirror 14: 0x59646bb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27bf2: skipped 000287 1779976069.119001 mapwrite 0x024a84 thru 0x02a89f (0x5e1c bytes) 000288 1779976069.439152 write 0x02a076 thru 0x032b4f (0x8ada bytes) 000289 1779976069.452309 mapread 0x018f0c thru 0x01cf5a (0x404f bytes) 000290 1779976069.489162 mapread 0x012778 thru 0x019f0a (0x7793 bytes) 000291 1779976069.498977 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32b50): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b50): Mirror 8: 0x9ffee33d (stale) Mirror 9: 0x9ffee33d (stale) Mirror 10: 0x9ffee33d (stale) Mirror 11: 0x9ffee33d (stale) Mirror 13: 0x9ffee33d (stale) Mirror 14: 0x784d75e7 (stale) CRC-32 checksum value for chunk [0, 0x32b50): Mirror 1: 0x69e2083b Mirror 15: 0x69e2083b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b50: skipped 000292 1779976071.035772 MIRROR_RESYNC 8 mirrors 000293 1779976072.774668 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000296 1779976073.234507 mapwrite 0x009bf7 thru 0x017bcf (0xdfd9 bytes) 000297 1779976073.469981 mapread 0x00ea36 thru 0x01e3b0 (0xf97b bytes) 000298 1779976073.486953 read 0x00ad69 thru 0x00fc3d (0x4ed5 bytes) 000299 1779976073.494834 read 0x014449 thru 0x0233ab (0xef63 bytes) 000300 1779976073.495825 mapread 0x014ce7 thru 0x023a3b (0xed55 bytes) 000303 1779976073.499058 read 0x007aa9 thru 0x0135f5 (0xbb4d bytes) 000307 1779976073.501789 mapwrite 0x019fa3 thru 0x027058 (0xd0b6 bytes) 000311 1779976073.572663 mapwrite 0x018b2d thru 0x01ed1d (0x61f1 bytes) 000312 1779976073.722247 mapread 0x0307ea thru 0x032262 (0x1a79 bytes) 000313 1779976073.743264 MIRROR_RESYNC 7 mirrors 000314 1779976075.759948 mapwrite 0x030399 thru 0x03732a (0x6f92 bytes) 000317 1779976075.958136 mapwrite 0x02d5cf thru 0x03b5cd (0xdfff bytes) 000318 1779976076.103967 mapwrite 0x00291e thru 0x0052cd (0x29b0 bytes) 000319 1779976076.194893 mapread 0x032002 thru 0x03b5cd (0x95cc bytes) 000321 1779976076.202687 read 0x0223fe thru 0x02a6f7 (0x82fa bytes) 000322 1779976076.232249 MIRROR_RESYNC 7 mirrors 000323 1779976078.437231 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000324 1779976079.201120 write 0x035384 thru 0x037f8d (0x2c0a bytes) 000325 1779976079.300760 mapwrite 0x030f1a thru 0x032d53 (0x1e3a bytes) 000326 1779976079.375160 read 0x018cdb thru 0x0229dd (0x9d03 bytes) 000327 1779976079.388046 write 0x03cefc thru 0x03ffff (0x3104 bytes) 000329 1779976079.389897 trunc from 0x040000 to 0x001d26 (0x3e2d9 bytes) 000332 1779976079.418538 mapread 0x00000a thru 0x001d26 (0x1d1d bytes) 000333 1779976079.427828 read 0x00016d thru 0x001d26 (0x1bba bytes) 000334 1779976079.435021 MIRROR_RESYNC 6 mirrors 000337 1779976081.030079 read 0x0002b2 thru 0x001d26 (0x1a75 bytes) 000340 1779976081.038483 read 0x000548 thru 0x001d26 (0x17df bytes) 000341 1779976081.040076 trunc from 0x001d27 to 0x0326e4 (0x309be bytes) 000342 1779976081.175798 trunc from 0x0326e5 to 0x025910 (0xcdd4 bytes) 000343 1779976081.235903 read 0x012933 thru 0x0193a3 (0x6a71 bytes) 000344 1779976081.252264 trunc from 0x025911 to 0x021008 (0x4908 bytes) 000345 1779976081.268224 read 0x0001f1 thru 0x00dace (0xd8de bytes) 000347 1779976081.270783 read 0x007e4d thru 0x011f74 (0xa128 bytes) 000348 1779976081.271725 write 0x02468d thru 0x02c1fb (0x7b6f bytes) 000349 1779976081.275477 mapread 0x005d57 thru 0x010497 (0xa741 bytes) 000350 1779976081.282866 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 9: 0x53c51589 (stale) Mirror 10: 0x53c51589 (stale) Mirror 11: 0x53c51589 (stale) Mirror 13: 0x53c51589 (stale) Mirror 15: 0x53c51589 (stale) CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 1: 0xd084d65a Mirror 16: 0xd084d65a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c1fc: skipped 000351 1779976082.870257 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 9: 0x53c51589 (stale) Mirror 11: 0x53c51589 (stale) Mirror 13: 0x53c51589 (stale) Mirror 15: 0x53c51589 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 9: 0x53c51589 (stale) Mirror 11: 0x53c51589 (stale) Mirror 13: 0x53c51589 (stale) Mirror 15: 0x53c51589 (stale) CRC-32 checksum value for chunk [0, 0x2c1fc): Mirror 1: 0xd084d65a Mirror 16: 0xd084d65a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c1fc: skipped 000353 1779976083.667709 mapread 0x00c2d4 thru 0x01a6c1 (0xe3ee bytes) 000354 1779976083.723237 write 0x03faf2 thru 0x03ffff (0x50e bytes) 000357 1779976083.892315 mapwrite 0x029aac thru 0x02b7a7 (0x1cfc bytes) 000358 1779976083.945395 trunc from 0x040000 to 0x032b1a (0xd4e5 bytes) 000359 1779976083.974638 trunc from 0x032b1b to 0x023401 (0xf719 bytes) 000363 1779976083.992575 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23402): Mirror 9: 0xfd215125 (stale) Mirror 13: 0xfd215125 (stale) Mirror 15: 0xfd215125 (stale) Mirror 16: 0xfd215125 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23402): Mirror 9: 0xfd215125 (stale) Mirror 13: 0xfd215125 (stale) Mirror 15: 0xfd215125 (stale) Mirror 16: 0xfd215125 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23402: skipped 000364 1779976084.520197 mapread 0x007df5 thru 0x00bc56 (0x3e62 bytes) 000365 1779976084.595973 mapread 0x018f27 thru 0x023401 (0xa4db bytes) 000366 1779976084.643733 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23402): Mirror 9: 0xfd215125 (stale) Mirror 13: 0xfd215125 (stale) Mirror 15: 0xfd215125 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23402): Mirror 9: 0xfd215125 (stale) Mirror 13: 0xfd215125 (stale) Mirror 15: 0xfd215125 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23402: skipped 000369 1779976085.009197 write 0x00f916 thru 0x018a7f (0x916a bytes) 000371 1779976085.029820 mapread 0x01d483 thru 0x023401 (0x5f7f bytes) 000372 1779976085.045749 mapwrite 0x0275b3 thru 0x035f9e (0xe9ec bytes) 000375 1779976085.246616 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f9f): Mirror 13: 0x7345fa0f (stale) Mirror 15: 0x7345fa0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f9f): Mirror 13: 0x7345fa0f (stale) Mirror 15: 0x7345fa0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f9f: skipped 000376 1779976085.863997 write 0x03a0cb thru 0x03ffff (0x5f35 bytes) 000377 1779976085.865534 read 0x0318ed thru 0x036e7d (0x5591 bytes) 000378 1779976085.887668 trunc from 0x040000 to 0x01738f (0x28c70 bytes) 000379 1779976085.929010 mapwrite 0x032d75 thru 0x03aadc (0x7d68 bytes) truncating to largest ever: 0x3fe8d 000383 1779976086.101350 trunc from 0x03aadd to 0x03fe8c (0x53b0 bytes) 000385 1779976086.142666 read 0x00dc5c thru 0x00f667 (0x1a0c bytes) 000386 1779976086.165609 read 0x001d40 thru 0x010cc8 (0xef89 bytes) 000388 1779976086.175245 write 0x023798 thru 0x02ef4f (0xb7b8 bytes) 000389 1779976086.179728 write 0x00bb9e thru 0x019232 (0xd695 bytes) 000391 1779976086.188018 mapwrite 0x02af49 thru 0x0320f7 (0x71af bytes) 000392 1779976086.313202 read 0x0176cc thru 0x01c69f (0x4fd4 bytes) 000393 1779976086.314082 trunc from 0x03fe8d to 0x037155 (0x8d37 bytes) 000395 1779976086.336055 read 0x00eb5e thru 0x01cea0 (0xe343 bytes) 000396 1779976086.336974 write 0x03e377 thru 0x03ffff (0x1c89 bytes) 000399 1779976086.347493 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x4cb623e3 (stale) Mirror 15: 0x4cb623e3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6728325d Mirror 16: 0x6728325d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000400 1779976087.570988 trunc from 0x040000 to 0x023add (0x1c522 bytes) 000401 1779976087.740946 mapread 0x015b16 thru 0x01c005 (0x64f0 bytes) 000402 1779976087.801145 trunc from 0x023ade to 0x01f5bd (0x4520 bytes) 000405 1779976087.868096 MIRROR_RESYNC 4 mirrors 000406 1779976089.013051 mapread 0x017ead thru 0x01e4f0 (0x6644 bytes) 000407 1779976089.056019 mapread 0x0119e3 thru 0x014dd8 (0x33f6 bytes) 000410 1779976089.071914 mapread 0x0033b3 thru 0x0129c2 (0xf610 bytes) 000411 1779976089.094669 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000412 1779976089.478741 read 0x00497d thru 0x0096fb (0x4d7f bytes) 000413 1779976089.497635 mapread 0x016dd3 thru 0x01de00 (0x702e bytes) 000414 1779976089.551697 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000417 1779976089.995021 trunc from 0x01f5be to 0x008352 (0x1726b bytes) 000419 1779976090.088489 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000422 1779976090.280988 read 0x003867 thru 0x008352 (0x4aec bytes) 000423 1779976090.292959 mapread 0x0040e7 thru 0x008352 (0x426c bytes) 000426 1779976090.301247 mapwrite 0x01a7f5 thru 0x023295 (0x8aa1 bytes) 000427 1779976090.470669 read 0x01edde thru 0x02240c (0x362f bytes) 000428 1779976090.487686 MIRROR_EXTEND to 2 mirrors 000429 1779976091.343779 MIRROR_EXTEND to 3 mirrors 000431 1779976092.385637 mapread 0x00ab3d thru 0x016030 (0xb4f4 bytes) 000432 1779976092.430381 mapwrite 0x036577 thru 0x03ffff (0x9a89 bytes) 000433 1779976092.661137 mapwrite 0x0270ab thru 0x02d7c3 (0x6719 bytes) 000434 1779976092.735815 write 0x004ddc thru 0x00dc7c (0x8ea1 bytes) 000435 1779976092.751913 mapwrite 0x0334fe thru 0x0362fa (0x2dfd bytes) 000437 1779976092.827680 mapread 0x03b6f3 thru 0x03ffff (0x490d bytes) 000438 1779976092.834041 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x23bd054e (stale) Mirror 3: 0x23bd054e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfeafac30 Mirror 4: 0xfeafac30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000439 1779976093.855626 trunc from 0x040000 to 0x0102a7 (0x2fd58 bytes) 000441 1779976093.932070 mapread 0x0093ed thru 0x0102a7 (0x6ebb bytes) 000443 1779976093.986108 mapwrite 0x00a488 thru 0x014b8b (0xa704 bytes) 000444 1779976094.083373 mapread 0x000784 thru 0x00fdfd (0xf67a bytes) 000445 1779976094.095846 mapwrite 0x01de36 thru 0x020d98 (0x2f63 bytes) 000447 1779976094.166607 read 0x019a8f thru 0x020d98 (0x730a bytes) 000451 1779976094.174492 trunc from 0x020d99 to 0x0125df (0xe7b9 bytes) 000452 1779976094.204146 trunc from 0x0125e0 to 0x021db4 (0xf7d5 bytes) 000453 1779976094.229831 mapwrite 0x003321 thru 0x00c522 (0x9202 bytes) 000454 1779976094.319845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21db5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21db5): Mirror 2: 0x56060cf6 (stale) Mirror 3: 0x56060cf6 (stale) Mirror 4: 0xbb86d65f (stale) CRC-32 checksum value for chunk [0, 0x21db5): Mirror 1: 0xb1e257ad Mirror 5: 0xb1e257ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21db5: skipped 000455 1779976095.416878 read 0x0023d9 thru 0x007264 (0x4e8c bytes) 000456 1779976095.441163 trunc from 0x021db5 to 0x0003ce (0x219e6 bytes) 000460 1779976095.640917 read 0x0001fe thru 0x0003ce (0x1d1 bytes) 000461 1779976095.651727 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf): Mirror 2: 0xd39b754d (stale) Mirror 3: 0xd39b754d (stale) Mirror 4: 0xd39b754d (stale) Mirror 5: 0xd39b754d (stale) CRC-32 checksum value for chunk [0, 0x3cf): Mirror 1: 0xd39b754d Mirror 6: 0xd39b754d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf: skipped 000463 1779976096.541716 MIRROR_RESYNC 6 mirrors 000465 1779976097.895454 MIRROR_RESYNC 6 mirrors 000468 1779976098.451528 MIRROR_RESYNC 6 mirrors 000469 1779976099.081799 mapread 0x000054 thru 0x0003ce (0x37b bytes) 000470 1779976099.092613 trunc from 0x0003cf to 0x0114ce (0x11100 bytes) 000480 1779976099.273616 mapwrite 0x029cb6 thru 0x036f9b (0xd2e6 bytes) 000481 1779976099.412980 mapread 0x0132aa thru 0x01b48b (0x81e2 bytes) 000482 1779976099.432324 read 0x005e72 thru 0x014c5f (0xedee bytes) 000483 1779976099.442003 read 0x025e0b thru 0x02e50e (0x8704 bytes) 000487 1779976099.446516 trunc from 0x036f9c to 0x02ce4a (0xa151 bytes) 000488 1779976099.463887 trunc from 0x02ce4b to 0x03bba8 (0xed5e bytes) 000489 1779976099.480548 read 0x02cbe5 thru 0x0357d3 (0x8bef bytes) 000490 1779976099.499597 mapwrite 0x02e685 thru 0x03e232 (0xfbae bytes) 000491 1779976099.566102 mapwrite 0x02370f thru 0x031ed9 (0xe7cb bytes) 000492 1779976099.610472 trunc from 0x03e233 to 0x03bfb9 (0x2279 bytes) 000495 1779976099.631023 trunc from 0x03bfba to 0x00003a (0x3bf7f bytes) 000496 1779976099.663577 mapread 0x000001 thru 0x00003a (0x3a bytes) 000497 1779976099.671082 MIRROR_RESYNC 6 mirrors 000499 1779976100.967050 write 0x00a25e thru 0x00df9e (0x3d41 bytes) 000504 1779976101.063759 write 0x004484 thru 0x00a43f (0x5fbc bytes) 000505 1779976101.072757 write 0x01eb3c thru 0x02d8e7 (0xedac bytes) 000507 1779976101.076510 read 0x00b08d thru 0x00bd95 (0xd09 bytes) 000508 1779976101.077824 trunc from 0x02d8e8 to 0x0324ff (0x4c18 bytes) 000510 1779976101.128022 mapread 0x02073e thru 0x02a2d0 (0x9b93 bytes) 000511 1779976101.130145 write 0x007321 thru 0x00bdec (0x4acc bytes) 000512 1779976101.132750 mapread 0x026de4 thru 0x027993 (0xbb0 bytes) 000513 1779976101.134723 write 0x02d82d thru 0x02efea (0x17be bytes) 000514 1779976101.142747 trunc from 0x032500 to 0x00df1d (0x245e2 bytes) 000516 1779976101.181125 mapread 0x00050d thru 0x003458 (0x2f4c bytes) 000518 1779976101.195603 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xdf1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdf1e): Mirror 2: 0xbdf3849e (stale) Mirror 3: 0xbdf3849e (stale) Mirror 4: 0xbdf3849e (stale) Mirror 5: 0xbdf3849e (stale) Mirror 6: 0xbdf3849e (stale) CRC-32 checksum value for chunk [0, 0xdf1e): Mirror 1: 0x725796a7 Mirror 7: 0x725796a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdf1e: skipped 000520 1779976102.526052 trunc from 0x00df1e to 0x0133dc (0x54bf bytes) 000521 1779976102.641067 trunc from 0x0133dd to 0x003f9c (0xf440 bytes) 000523 1779976102.674430 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f9d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9d): Mirror 2: 0x750a2018 (stale) Mirror 3: 0x750a2018 (stale) Mirror 4: 0x750a2018 (stale) Mirror 5: 0x750a2018 (stale) Mirror 6: 0x750a2018 (stale) Mirror 7: 0x750a2018 (stale) CRC-32 checksum value for chunk [0, 0x3f9d): Mirror 1: 0x750a2018 Mirror 8: 0x750a2018 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9d: skipped 000524 1779976103.631909 mapread 0x003471 thru 0x003f9c (0xb2c bytes) 000525 1779976103.717010 write 0x01ae58 thru 0x020942 (0x5aeb bytes) 000526 1779976103.865651 mapwrite 0x03b9a6 thru 0x03ffff (0x465a bytes) 000527 1779976104.122653 read 0x014a59 thru 0x022074 (0xd61c bytes) 000531 1779976104.185938 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0x91b19703 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x323def5d Mirror 9: 0x323def5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000534 1779976105.791170 MIRROR_RESYNC 9 mirrors 000537 1779976108.014934 trunc from 0x040000 to 0x028ea2 (0x1715d bytes) 000538 1779976108.109821 mapread 0x0271fa thru 0x028ea2 (0x1ca9 bytes) 000540 1779976108.136591 trunc from 0x028ea3 to 0x002758 (0x2674a bytes) 000541 1779976108.181348 trunc from 0x002759 to 0x019a23 (0x172cb bytes) 000542 1779976108.202559 mapwrite 0x02b225 thru 0x035366 (0xa142 bytes) 000546 1779976108.305485 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35367): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35367): Mirror 2: 0xd4a842ed (stale) Mirror 3: 0xd4a842ed (stale) Mirror 4: 0xd4a842ed (stale) Mirror 5: 0xd4a842ed (stale) Mirror 6: 0xd4a842ed (stale) Mirror 7: 0xd4a842ed (stale) Mirror 8: 0xd4a842ed (stale) Mirror 9: 0xd4a842ed (stale) CRC-32 checksum value for chunk [0, 0x35367): Mirror 1: 0x124f3912 Mirror 10: 0x124f3912 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35367: skipped 000548 1779976108.990132 read 0x02a68a thru 0x02f484 (0x4dfb bytes) 000551 1779976109.007073 write 0x00ce65 thru 0x00fb3f (0x2cdb bytes) 000553 1779976109.079823 read 0x018b7e thru 0x01cbf9 (0x407c bytes) 000557 1779976109.109923 write 0x01df06 thru 0x020ecf (0x2fca bytes) 000563 1779976109.114679 mapwrite 0x025b7d thru 0x029620 (0x3aa4 bytes) 000564 1779976109.179178 trunc from 0x035367 to 0x02dbd7 (0x778f bytes) 000566 1779976109.252023 read 0x025590 thru 0x029a3a (0x44ab bytes) 000568 1779976109.256602 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 2: 0x4515f001 (stale) Mirror 3: 0x4515f001 (stale) Mirror 4: 0x4515f001 (stale) Mirror 5: 0x4515f001 (stale) Mirror 6: 0x4515f001 (stale) Mirror 7: 0x4515f001 (stale) Mirror 8: 0x4515f001 (stale) Mirror 9: 0x4515f001 (stale) Mirror 10: 0x1d3f0b55 (stale) CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 1: 0x16d3b62b Mirror 11: 0x16d3b62b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dbd8: skipped 000569 1779976110.313353 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 2: 0x4515f001 (stale) Mirror 3: 0x4515f001 (stale) Mirror 4: 0x4515f001 (stale) Mirror 5: 0x4515f001 (stale) Mirror 6: 0x4515f001 (stale) Mirror 7: 0x4515f001 (stale) Mirror 8: 0x4515f001 (stale) Mirror 9: 0x4515f001 (stale) Mirror 10: 0x1d3f0b55 (stale) CRC-32 checksum value for chunk [0, 0x2dbd8): Mirror 1: 0x16d3b62b Mirror 11: 0x16d3b62b Mirror 12: 0x16d3b62b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dbd8: skipped 000571 1779976111.354611 read 0x016c35 thru 0x018cb8 (0x2084 bytes) 000572 1779976111.359905 trunc from 0x02dbd8 to 0x02b01e (0x2bb9 bytes) 000573 1779976111.436328 write 0x039666 thru 0x03ffff (0x699a bytes) 000574 1779976111.441008 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x323def5d (stale) Mirror 4: 0x323def5d (stale) Mirror 5: 0x323def5d (stale) Mirror 6: 0x323def5d (stale) Mirror 7: 0x323def5d (stale) Mirror 8: 0x323def5d (stale) Mirror 9: 0x323def5d (stale) Mirror 10: 0x2e128c0 (stale) Mirror 11: 0x63933b90 (stale) Mirror 12: 0x63933b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x323def5d (stale) Mirror 4: 0x323def5d (stale) Mirror 5: 0x323def5d (stale) Mirror 6: 0x323def5d (stale) Mirror 7: 0x323def5d (stale) Mirror 8: 0x323def5d (stale) Mirror 9: 0x323def5d (stale) Mirror 10: 0x2e128c0 (stale) Mirror 11: 0x63933b90 (stale) Mirror 12: 0x63933b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000575 1779976111.901893 mapwrite 0x02b963 thru 0x03881d (0xcebb bytes) 000576 1779976112.031712 mapwrite 0x03f04e thru 0x03ffff (0xfb2 bytes) 000578 1779976112.115251 write 0x01e0fb thru 0x0218c0 (0x37c6 bytes) 000579 1779976112.133849 trunc from 0x040000 to 0x0202fc (0x1fd03 bytes) 000580 1779976112.166397 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x202fd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202fd): Mirror 3: 0xcc03ac27 (stale) Mirror 4: 0xcc03ac27 (stale) Mirror 5: 0xcc03ac27 (stale) Mirror 6: 0xcc03ac27 (stale) Mirror 7: 0xcc03ac27 (stale) Mirror 8: 0xcc03ac27 (stale) Mirror 9: 0xcc03ac27 (stale) Mirror 10: 0x29256978 (stale) Mirror 11: 0xc197492a (stale) Mirror 12: 0xc197492a (stale) CRC-32 checksum value for chunk [0, 0x202fd): Mirror 1: 0xc5a894f3 Mirror 13: 0xc5a894f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202fd: skipped 000581 1779976113.571204 write 0x02c7c0 thru 0x037af5 (0xb336 bytes) 000582 1779976113.660392 write 0x01cb21 thru 0x024982 (0x7e62 bytes) 000583 1779976113.674091 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x37af6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37af6): Mirror 3: 0xebbd7c7f (stale) Mirror 4: 0xebbd7c7f (stale) Mirror 5: 0xebbd7c7f (stale) Mirror 6: 0xebbd7c7f (stale) Mirror 7: 0xebbd7c7f (stale) Mirror 8: 0xebbd7c7f (stale) Mirror 9: 0xebbd7c7f (stale) Mirror 10: 0x13178c92 (stale) Mirror 11: 0xb61c08fd (stale) Mirror 12: 0xb61c08fd (stale) Mirror 13: 0x825db00f (stale) CRC-32 checksum value for chunk [0, 0x37af6): Mirror 1: 0xf6f45151 Mirror 14: 0xf6f45151 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37af6: skipped 000584 1779976114.820228 write 0x01ba4c thru 0x02a5cf (0xeb84 bytes) 000585 1779976114.901525 mapread 0x030deb thru 0x037af5 (0x6d0b bytes) 000588 1779976114.917351 trunc from 0x037af6 to 0x012584 (0x25571 bytes) 000589 1779976114.943605 write 0x03eb54 thru 0x03ffff (0x14ac bytes) 000590 1779976114.945802 trunc from 0x040000 to 0x0154b6 (0x2ab49 bytes) 000593 1779976114.976522 trunc from 0x0154b7 to 0x01d3ff (0x7f49 bytes) 000595 1779976114.999159 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1d400): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d400): Mirror 3: 0x2a109e79 (stale) Mirror 4: 0x2a109e79 (stale) Mirror 5: 0x2a109e79 (stale) Mirror 6: 0x2a109e79 (stale) Mirror 7: 0x2a109e79 (stale) Mirror 8: 0x2a109e79 (stale) Mirror 9: 0x2a109e79 (stale) Mirror 10: 0xd0588697 (stale) Mirror 11: 0xb5bfdc8a (stale) Mirror 12: 0xb5bfdc8a (stale) Mirror 13: 0xb5bfdc8a (stale) Mirror 14: 0x2313e744 (stale) CRC-32 checksum value for chunk [0, 0x1d400): Mirror 1: 0xb5bfdc8a Mirror 15: 0xb5bfdc8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d400: skipped 000597 1779976116.524502 read 0x002fc7 thru 0x00b695 (0x86cf bytes) 000599 1779976116.584861 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d400): Mirror 4: 0x2a109e79 (stale) Mirror 5: 0x2a109e79 (stale) Mirror 6: 0x2a109e79 (stale) Mirror 7: 0x2a109e79 (stale) Mirror 8: 0x2a109e79 (stale) Mirror 9: 0x2a109e79 (stale) Mirror 10: 0xd0588697 (stale) Mirror 11: 0xb5bfdc8a (stale) Mirror 12: 0xb5bfdc8a (stale) Mirror 13: 0xb5bfdc8a (stale) Mirror 14: 0x2313e744 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d400): Mirror 4: 0x2a109e79 (stale) Mirror 5: 0x2a109e79 (stale) Mirror 6: 0x2a109e79 (stale) Mirror 7: 0x2a109e79 (stale) Mirror 8: 0x2a109e79 (stale) Mirror 9: 0x2a109e79 (stale) Mirror 10: 0xd0588697 (stale) Mirror 11: 0xb5bfdc8a (stale) Mirror 12: 0xb5bfdc8a (stale) Mirror 13: 0xb5bfdc8a (stale) Mirror 14: 0x2313e744 (stale) CRC-32 checksum value for chunk [0, 0x1d400): Mirror 1: 0xb5bfdc8a Mirror 15: 0xb5bfdc8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d400: skipped 000600 1779976117.618472 mapread 0x016da6 thru 0x01d3ff (0x665a bytes) 000601 1779976117.647845 write 0x03b495 thru 0x03ffff (0x4b6b bytes) 000602 1779976117.761723 MIRROR_RESYNC 13 mirrors 000603 1779976119.690669 mapwrite 0x03b591 thru 0x03d17f (0x1bef bytes) 000604 1779976119.846352 trunc from 0x040000 to 0x02bf00 (0x140ff bytes) 000605 1779976119.872746 read 0x013962 thru 0x01e90b (0xafaa bytes) 000606 1779976119.883359 mapwrite 0x02ed5d thru 0x033eb7 (0x515b bytes) 000609 1779976119.968258 read 0x00b892 thru 0x01989e (0xe00d bytes) 000611 1779976120.015183 MIRROR_RESYNC 13 mirrors 000612 1779976122.579113 mapwrite 0x00e0de thru 0x015f41 (0x7e64 bytes) 000615 1779976122.795893 write 0x02dc15 thru 0x03b549 (0xd935 bytes) 000618 1779976122.805582 mapread 0x01ac91 thru 0x01e147 (0x34b7 bytes) 000620 1779976122.814589 write 0x009291 thru 0x0112ae (0x801e bytes) 000621 1779976122.816829 mapread 0x011c31 thru 0x01f1e4 (0xd5b4 bytes) 000622 1779976122.822903 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b54a): Mirror 4: 0x1c9dd61c (stale) Mirror 5: 0x1c9dd61c (stale) Mirror 6: 0x1c9dd61c (stale) Mirror 7: 0x1c9dd61c (stale) Mirror 8: 0x1c9dd61c (stale) Mirror 10: 0x1c9dd61c (stale) Mirror 11: 0x1c9dd61c (stale) Mirror 12: 0x1c9dd61c (stale) Mirror 13: 0x1c9dd61c (stale) Mirror 14: 0x1c9dd61c (stale) Mirror 15: 0x1c9dd61c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b54a): Mirror 4: 0x1c9dd61c (stale) Mirror 5: 0x1c9dd61c (stale) Mirror 6: 0x1c9dd61c (stale) Mirror 7: 0x1c9dd61c (stale) Mirror 8: 0x1c9dd61c (stale) Mirror 10: 0x1c9dd61c (stale) Mirror 11: 0x1c9dd61c (stale) Mirror 12: 0x1c9dd61c (stale) Mirror 13: 0x1c9dd61c (stale) Mirror 14: 0x1c9dd61c (stale) Mirror 15: 0x1c9dd61c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b54a: skipped 000623 1779976123.798565 mapwrite 0x02da0d thru 0x03460f (0x6c03 bytes) 000625 1779976123.962049 read 0x00e20e thru 0x015967 (0x775a bytes) 000626 1779976123.995518 mapwrite 0x003dbf thru 0x00d734 (0x9976 bytes) 000627 1779976124.131434 read 0x008803 thru 0x010dc4 (0x85c2 bytes) 000629 1779976124.132981 write 0x03a6cd thru 0x03ffff (0x5933 bytes) 000630 1779976124.161162 write 0x039916 thru 0x03ffff (0x66ea bytes) 000633 1779976124.181079 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaa70418c (stale) Mirror 5: 0xaa70418c (stale) Mirror 6: 0xaa70418c (stale) Mirror 8: 0xaa70418c (stale) Mirror 10: 0xaa70418c (stale) Mirror 11: 0xaa70418c (stale) Mirror 12: 0xaa70418c (stale) Mirror 13: 0xaa70418c (stale) Mirror 14: 0xaa70418c (stale) Mirror 15: 0xaa70418c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaa70418c (stale) Mirror 5: 0xaa70418c (stale) Mirror 6: 0xaa70418c (stale) Mirror 8: 0xaa70418c (stale) Mirror 10: 0xaa70418c (stale) Mirror 11: 0xaa70418c (stale) Mirror 12: 0xaa70418c (stale) Mirror 13: 0xaa70418c (stale) Mirror 14: 0xaa70418c (stale) Mirror 15: 0xaa70418c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000636 1779976125.169710 read 0x01e7c7 thru 0x022e5a (0x4694 bytes) 000639 1779976125.229969 mapread 0x02eef3 thru 0x038cbd (0x9dcb bytes) 000641 1779976125.297248 write 0x0132b5 thru 0x013efd (0xc49 bytes) 000642 1779976125.344807 read 0x00f6f1 thru 0x01f6a3 (0xffb3 bytes) 000644 1779976125.372328 mapwrite 0x016973 thru 0x017477 (0xb05 bytes) 000646 1779976125.520413 read 0x02d9ee thru 0x03a864 (0xce77 bytes) 000647 1779976125.521305 trunc from 0x040000 to 0x016b4c (0x294b3 bytes) 000648 1779976125.556339 mapread 0x01427b thru 0x016b4c (0x28d2 bytes) 000649 1779976125.562061 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x16b4d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b4d): Mirror 4: 0x83099258 (stale) Mirror 5: 0x83099258 (stale) Mirror 6: 0x83099258 (stale) Mirror 8: 0x83099258 (stale) Mirror 10: 0x83099258 (stale) Mirror 11: 0x83099258 (stale) Mirror 12: 0x83099258 (stale) Mirror 13: 0x83099258 (stale) Mirror 14: 0x83099258 (stale) Mirror 15: 0x83099258 (stale) CRC-32 checksum value for chunk [0, 0x16b4d): Mirror 1: 0x67b32b76 Mirror 16: 0x67b32b76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b4d: skipped 000650 1779976127.360665 mapread 0x015ee6 thru 0x016b4c (0xc67 bytes) 000653 1779976127.468386 trunc from 0x016b4d to 0x00fa03 (0x7149 bytes) 000655 1779976127.635060 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfa04): Mirror 4: 0x8b39663c (stale) Mirror 5: 0x8b39663c (stale) Mirror 6: 0x8b39663c (stale) Mirror 8: 0x8b39663c (stale) Mirror 10: 0x8b39663c (stale) Mirror 11: 0x8b39663c (stale) Mirror 12: 0x8b39663c (stale) Mirror 14: 0x8b39663c (stale) Mirror 15: 0x8b39663c (stale) Mirror 16: 0xd9dedb18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa04): Mirror 4: 0x8b39663c (stale) Mirror 5: 0x8b39663c (stale) Mirror 6: 0x8b39663c (stale) Mirror 8: 0x8b39663c (stale) Mirror 10: 0x8b39663c (stale) Mirror 11: 0x8b39663c (stale) Mirror 12: 0x8b39663c (stale) Mirror 14: 0x8b39663c (stale) Mirror 15: 0x8b39663c (stale) Mirror 16: 0xd9dedb18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa04: skipped 000656 1779976128.322511 write 0x025676 thru 0x0299a7 (0x4332 bytes) 000657 1779976128.324225 write 0x02f68f thru 0x037efe (0x8870 bytes) 000658 1779976128.330852 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37eff): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eff): Mirror 4: 0xbe91cc8e (stale) Mirror 5: 0xbe91cc8e (stale) Mirror 6: 0xbe91cc8e (stale) Mirror 8: 0xbe91cc8e (stale) Mirror 10: 0xbe91cc8e (stale) Mirror 11: 0xbe91cc8e (stale) Mirror 12: 0xbe91cc8e (stale) Mirror 14: 0xbe91cc8e (stale) Mirror 15: 0xbe91cc8e (stale) Mirror 16: 0xc4f00ee8 (stale) CRC-32 checksum value for chunk [0, 0x37eff): Mirror 1: 0x8b2c0b32 Mirror 17: 0x8b2c0b32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eff: skipped 000660 1779976129.958959 mapread 0x02eeea thru 0x03588b (0x69a2 bytes) 000662 1779976130.053872 trunc from 0x037eff to 0x014afa (0x23404 bytes) 000663 1779976130.161618 read 0x0027d1 thru 0x011d93 (0xf5c3 bytes) 000664 1779976130.172157 mapwrite 0x020ebf thru 0x02424b (0x338d bytes) 000665 1779976130.271297 read 0x007a3f thru 0x016a88 (0xf04a bytes) 000666 1779976130.286071 trunc from 0x02424c to 0x010141 (0x1410a bytes) 000670 1779976130.319487 mapwrite 0x00fb34 thru 0x01c5e6 (0xcab3 bytes) 000675 1779976130.511374 mapwrite 0x023cee thru 0x029d99 (0x60ac bytes) 000677 1779976130.597732 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x29d9a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d9a): Mirror 4: 0x388f6d99 (stale) Mirror 5: 0x388f6d99 (stale) Mirror 6: 0x388f6d99 (stale) Mirror 8: 0x388f6d99 (stale) Mirror 10: 0x388f6d99 (stale) Mirror 11: 0x388f6d99 (stale) Mirror 12: 0x388f6d99 (stale) Mirror 14: 0x388f6d99 (stale) Mirror 15: 0x388f6d99 (stale) Mirror 16: 0x98c08beb (stale) Mirror 17: 0xa6afc988 (stale) CRC-32 checksum value for chunk [0, 0x29d9a): Mirror 1: 0xc7b5bdb0 Mirror 18: 0xc7b5bdb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d9a: skipped 000679 1779976131.542531 write 0x02412c thru 0x032812 (0xe6e7 bytes) 000680 1779976131.658060 mapwrite 0x022eab thru 0x02fd21 (0xce77 bytes) 000681 1779976131.794258 mapread 0x028e9d thru 0x032812 (0x9976 bytes) 000682 1779976131.797075 mapwrite 0x02c1b4 thru 0x03b6f4 (0xf541 bytes) 000683 1779976131.974176 write 0x0243a7 thru 0x029e44 (0x5a9e bytes) 000684 1779976131.977198 trunc from 0x03b6f5 to 0x03d9bc (0x22c8 bytes) 000685 1779976132.009609 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d9bd): Mirror 4: 0x63a9e494 (stale) Mirror 6: 0x63a9e494 (stale) Mirror 8: 0x63a9e494 (stale) Mirror 10: 0x63a9e494 (stale) Mirror 11: 0x63a9e494 (stale) Mirror 12: 0x63a9e494 (stale) Mirror 14: 0x63a9e494 (stale) Mirror 15: 0x63a9e494 (stale) Mirror 16: 0x4ea9ae07 (stale) Mirror 17: 0x154fd539 (stale) Mirror 18: 0xfc7f7ed3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9bd): Mirror 4: 0x63a9e494 (stale) Mirror 6: 0x63a9e494 (stale) Mirror 8: 0x63a9e494 (stale) Mirror 10: 0x63a9e494 (stale) Mirror 11: 0x63a9e494 (stale) Mirror 12: 0x63a9e494 (stale) Mirror 14: 0x63a9e494 (stale) Mirror 15: 0x63a9e494 (stale) Mirror 16: 0x4ea9ae07 (stale) Mirror 17: 0x154fd539 (stale) Mirror 18: 0xfc7f7ed3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9bd: skipped 000686 1779976133.142373 mapread 0x01e7f2 thru 0x02da6a (0xf279 bytes) 000687 1779976133.197327 MIRROR_RESYNC 12 mirrors 000688 1779976135.494562 MIRROR_RESYNC 12 mirrors 000689 1779976136.023070 trunc from 0x03d9bd to 0x03d8e9 (0xd3 bytes) 000690 1779976136.090460 trunc from 0x03d8ea to 0x03ecdb (0x13f2 bytes) 000692 1779976136.138816 MIRROR_RESYNC 12 mirrors 000694 1779976138.380834 write 0x0009c1 thru 0x006176 (0x57b6 bytes) 000695 1779976138.456251 read 0x01e85e thru 0x02d7af (0xef52 bytes) 000697 1779976138.474052 mapread 0x0010b1 thru 0x00f97c (0xe8cc bytes) 000699 1779976138.501244 MIRROR_RESYNC 12 mirrors 000700 1779976140.731906 read 0x01a665 thru 0x021b60 (0x74fc bytes) 000701 1779976140.749603 trunc from 0x03ecdc to 0x029f59 (0x14d82 bytes) 000702 1779976140.917149 MIRROR_RESYNC 12 mirrors 000703 1779976143.032259 mapread 0x00b788 thru 0x01a85c (0xf0d5 bytes) 000705 1779976143.065753 mapread 0x0297af thru 0x029f59 (0x7ab bytes) 000706 1779976143.080088 read 0x0292e6 thru 0x029f59 (0xc74 bytes) 000710 1779976143.082956 mapwrite 0x00dba9 thru 0x01d789 (0xfbe1 bytes) 000711 1779976143.292463 read 0x00b785 thru 0x00fb65 (0x43e1 bytes) 000712 1779976143.293239 mapread 0x00c589 thru 0x00de1a (0x1892 bytes) 000713 1779976143.294210 mapread 0x01a268 thru 0x023a7d (0x9816 bytes) 000714 1779976143.311984 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x29f5a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f5a): Mirror 4: 0x4865aa98 (stale) Mirror 6: 0x4865aa98 (stale) Mirror 8: 0x4865aa98 (stale) Mirror 10: 0x4865aa98 (stale) Mirror 11: 0x4865aa98 (stale) Mirror 12: 0x4865aa98 (stale) Mirror 14: 0x4865aa98 (stale) Mirror 15: 0x4865aa98 (stale) Mirror 16: 0x4865aa98 (stale) Mirror 17: 0x4865aa98 (stale) Mirror 18: 0x4865aa98 (stale) CRC-32 checksum value for chunk [0, 0x29f5a): Mirror 1: 0x4ad078ff Mirror 19: 0x4ad078ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f5a: skipped 000715 1779976145.170912 write 0x028b6c thru 0x036191 (0xd626 bytes) 000716 1779976145.395409 mapwrite 0x00abad thru 0x0129e2 (0x7e36 bytes) 000718 1779976145.553001 mapwrite 0x01ca23 thru 0x0259e5 (0x8fc3 bytes) 000719 1779976145.691874 read 0x023751 thru 0x030d9b (0xd64b bytes) 000720 1779976145.702868 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36192): Mirror 4: 0xaa7eaff1 (stale) Mirror 6: 0xaa7eaff1 (stale) Mirror 8: 0xaa7eaff1 (stale) Mirror 11: 0xaa7eaff1 (stale) Mirror 12: 0xaa7eaff1 (stale) Mirror 14: 0xaa7eaff1 (stale) Mirror 15: 0xaa7eaff1 (stale) Mirror 16: 0xaa7eaff1 (stale) Mirror 17: 0xaa7eaff1 (stale) Mirror 18: 0xaa7eaff1 (stale) Mirror 19: 0xe0da67ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36192): Mirror 4: 0xaa7eaff1 (stale) Mirror 6: 0xaa7eaff1 (stale) Mirror 8: 0xaa7eaff1 (stale) Mirror 11: 0xaa7eaff1 (stale) Mirror 12: 0xaa7eaff1 (stale) Mirror 14: 0xaa7eaff1 (stale) Mirror 15: 0xaa7eaff1 (stale) Mirror 16: 0xaa7eaff1 (stale) Mirror 17: 0xaa7eaff1 (stale) Mirror 18: 0xaa7eaff1 (stale) Mirror 19: 0xe0da67ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36192: skipped 000721 1779976146.562117 trunc from 0x036192 to 0x010def (0x253a2 bytes) 000722 1779976146.610065 trunc from 0x010df0 to 0x019674 (0x8885 bytes) 000724 1779976146.634236 trunc from 0x019675 to 0x01ec79 (0x5605 bytes) 000726 1779976146.659201 read 0x001e31 thru 0x008e3f (0x700f bytes) 000728 1779976146.670953 mapread 0x011993 thru 0x011f81 (0x5ef bytes) 000729 1779976146.677966 trunc from 0x01ec7a to 0x010632 (0xe647 bytes) 000731 1779976146.706842 read 0x003ffb thru 0x005c3f (0x1c45 bytes) 000732 1779976146.707995 read 0x00b7c5 thru 0x00f1ab (0x39e7 bytes) 000733 1779976146.720113 read 0x00e3bc thru 0x010632 (0x2277 bytes) 000735 1779976146.722580 mapread 0x0071a7 thru 0x010632 (0x948c bytes) 000738 1779976146.731648 mapwrite 0x03dbdb thru 0x03ffff (0x2425 bytes) 000740 1779976146.818429 write 0x0188e4 thru 0x0277b0 (0xeecd bytes) 000741 1779976146.827969 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 8: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf82ad17f Mirror 20: 0xf82ad17f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000742 1779976148.260996 mapread 0x01b5ad thru 0x01c453 (0xea7 bytes) 000743 1779976148.308719 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf82ad17f Mirror 20: 0xf82ad17f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000745 1779976149.480427 mapwrite 0x0164c4 thru 0x019a79 (0x35b6 bytes) 000746 1779976149.642670 mapread 0x02d1a4 thru 0x02fe95 (0x2cf2 bytes) 000747 1779976149.654563 mapwrite 0x036b5e thru 0x03b7a8 (0x4c4b bytes) 000749 1779976149.694156 write 0x01fd00 thru 0x029310 (0x9611 bytes) 000750 1779976149.695895 read 0x0322e3 thru 0x0325b1 (0x2cf bytes) 000752 1779976149.697545 read 0x0157b4 thru 0x025785 (0xffd2 bytes) 000753 1779976149.702064 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000755 1779976150.311921 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xded736c4 Mirror 20: 0xded736c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000756 1779976151.549852 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x937c9065 (stale) Mirror 6: 0x937c9065 (stale) Mirror 11: 0x937c9065 (stale) Mirror 12: 0x937c9065 (stale) Mirror 14: 0x937c9065 (stale) Mirror 15: 0x937c9065 (stale) Mirror 16: 0x937c9065 (stale) Mirror 17: 0x937c9065 (stale) Mirror 18: 0x937c9065 (stale) Mirror 19: 0x1092756f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xded736c4 Mirror 20: 0xded736c4 Mirror 21: 0xded736c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1779976153.007939 read 0x000f65 thru 0x009796 (0x8832 bytes) 000759 1779976153.015785 read 0x007311 thru 0x00be3b (0x4b2b bytes) 000760 1779976153.060241 mapwrite 0x0124fa thru 0x0163ec (0x3ef3 bytes) 000762 1779976153.236730 trunc from 0x040000 to 0x00a253 (0x35dac bytes) 000764 1779976153.318212 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0xa254): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa254): Mirror 4: 0xe4f1283c (stale) Mirror 6: 0xe4f1283c (stale) Mirror 11: 0xe4f1283c (stale) Mirror 12: 0xe4f1283c (stale) Mirror 14: 0xe4f1283c (stale) Mirror 15: 0xe4f1283c (stale) Mirror 16: 0xe4f1283c (stale) Mirror 17: 0xe4f1283c (stale) Mirror 18: 0xe4f1283c (stale) Mirror 19: 0xe4f1283c (stale) Mirror 20: 0xe4f1283c (stale) Mirror 21: 0xe4f1283c (stale) CRC-32 checksum value for chunk [0, 0xa254): Mirror 1: 0xe4f1283c Mirror 22: 0xe4f1283c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa254: skipped 000767 1779976154.680031 mapwrite 0x0342ea thru 0x037800 (0x3517 bytes) 000770 1779976154.947109 mapwrite 0x00de6f thru 0x01a02a (0xc1bc bytes) 000774 1779976155.070176 mapwrite 0x02407e thru 0x02e762 (0xa6e5 bytes) 000775 1779976155.147217 mapwrite 0x02fc8a thru 0x033385 (0x36fc bytes) 000776 1779976155.198095 trunc from 0x037801 to 0x02d6b9 (0xa147 bytes) 000777 1779976155.230661 mapwrite 0x0078e5 thru 0x012982 (0xb09e bytes) 000778 1779976155.287063 write 0x00c545 thru 0x00e2e6 (0x1da2 bytes) 000779 1779976155.289179 read 0x00759f thru 0x015e48 (0xe8aa bytes) 000781 1779976155.292102 read 0x01812b thru 0x02178c (0x9662 bytes) 000782 1779976155.294198 mapread 0x02a843 thru 0x02d6b9 (0x2e77 bytes) 000784 1779976155.301718 write 0x03e913 thru 0x03ffff (0x16ed bytes) 000785 1779976155.303983 write 0x01800d thru 0x01a494 (0x2488 bytes) 000787 1779976155.307638 read 0x01d57a thru 0x02c695 (0xf11c bytes) 000789 1779976155.310011 trunc from 0x040000 to 0x035ec5 (0xa13a bytes) 000790 1779976155.332029 read 0x02eda9 thru 0x030843 (0x1a9b bytes) 000792 1779976155.343674 trunc from 0x035ec6 to 0x03dcb1 (0x7dec bytes) 000794 1779976155.369480 trunc from 0x03dcb2 to 0x0152eb (0x289c6 bytes) 000795 1779976155.391116 mapread 0x007c17 thru 0x014369 (0xc753 bytes) 000796 1779976155.394014 trunc from 0x0152ec to 0x006c2c (0xe6bf bytes) 000797 1779976155.414722 trunc from 0x006c2d to 0x006b4d (0xdf bytes) 000800 1779976155.447003 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b4e): Mirror 4: 0x41ff4225 (stale) Mirror 11: 0x41ff4225 (stale) Mirror 12: 0x41ff4225 (stale) Mirror 14: 0x41ff4225 (stale) Mirror 15: 0x41ff4225 (stale) Mirror 16: 0x41ff4225 (stale) Mirror 17: 0x41ff4225 (stale) Mirror 18: 0x41ff4225 (stale) Mirror 19: 0x41ff4225 (stale) Mirror 20: 0x41ff4225 (stale) Mirror 21: 0x41ff4225 (stale) Mirror 22: 0x41ff4225 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b4e): Mirror 4: 0x41ff4225 (stale) Mirror 11: 0x41ff4225 (stale) Mirror 12: 0x41ff4225 (stale) Mirror 14: 0x41ff4225 (stale) Mirror 15: 0x41ff4225 (stale) Mirror 16: 0x41ff4225 (stale) Mirror 17: 0x41ff4225 (stale) Mirror 18: 0x41ff4225 (stale) Mirror 19: 0x41ff4225 (stale) Mirror 20: 0x41ff4225 (stale) Mirror 21: 0x41ff4225 (stale) Mirror 22: 0x41ff4225 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b4e: skipped 000804 1779976156.079658 trunc from 0x006b4e to 0x021a5c (0x1af0f bytes) 000806 1779976156.098611 write 0x019567 thru 0x01de41 (0x48db bytes) 000807 1779976156.118166 read 0x01623e thru 0x01735b (0x111e bytes) 000808 1779976156.148852 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 15: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 20: 0xb000783d (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 15: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 20: 0xb000783d (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21a5d: skipped 000811 1779976157.005765 read 0x00c599 thru 0x018fa8 (0xca10 bytes) 000812 1779976157.021920 mapwrite 0x002923 thru 0x00df58 (0xb636 bytes) 000814 1779976157.097161 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 20: 0xb000783d (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 20: 0xb000783d (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21a5d: skipped 000815 1779976157.693735 mapread 0x00bbec thru 0x00fed1 (0x42e6 bytes) 000818 1779976157.720618 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21a5d: skipped 000819 1779976158.280821 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 4: 0xbca90260 (stale) Mirror 11: 0xbca90260 (stale) Mirror 12: 0xbca90260 (stale) Mirror 14: 0xbca90260 (stale) Mirror 16: 0xbca90260 (stale) Mirror 17: 0xbca90260 (stale) Mirror 19: 0x9446fbe2 (stale) Mirror 21: 0xb000783d (stale) Mirror 22: 0x1a1e36f7 (stale) CRC-32 checksum value for chunk [0, 0x21a5d): Mirror 1: 0x24714e30 Mirror 23: 0x24714e30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21a5d: skipped 000820 1779976159.714586 mapread 0x020dd8 thru 0x021a5c (0xc85 bytes) 000821 1779976159.766402 mapread 0x01b1dd thru 0x01decd (0x2cf1 bytes) 000822 1779976159.804943 trunc from 0x021a5d to 0x02e57d (0xcb21 bytes) 000824 1779976159.979965 read 0x01c7f9 thru 0x01d007 (0x80f bytes) 000825 1779976159.990087 MIRROR_RESYNC 11 mirrors 000828 1779976162.595988 write 0x02d4ed thru 0x035aa5 (0x85b9 bytes) 000829 1779976162.674682 mapread 0x023a7b thru 0x02d782 (0x9d08 bytes) 000830 1779976162.694696 mapread 0x034bd6 thru 0x035aa5 (0xed0 bytes) 000832 1779976162.697781 read 0x00f4da thru 0x01df6e (0xea95 bytes) 000833 1779976162.714834 trunc from 0x035aa6 to 0x0174bb (0x1e5ea bytes) 000834 1779976162.741839 mapread 0x00d33e thru 0x0174bb (0xa17e bytes) 000835 1779976162.752635 write 0x00a073 thru 0x0186c8 (0xe656 bytes) 000836 1779976162.754992 mapread 0x00be21 thru 0x00e6a3 (0x2883 bytes) 000837 1779976162.756215 write 0x00ae25 thru 0x014f8c (0xa168 bytes) 000839 1779976162.759404 read 0x010b94 thru 0x012a2b (0x1e98 bytes) 000843 1779976162.762757 read 0x015406 thru 0x0186c8 (0x32c3 bytes) 000846 1779976162.766798 mapwrite 0x006cff thru 0x013b72 (0xce74 bytes) 000847 1779976162.824997 mapwrite 0x009f16 thru 0x014c7a (0xad65 bytes) 000848 1779976162.868245 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x186c9): Mirror 4: 0x2018df00 (stale) Mirror 11: 0x2018df00 (stale) Mirror 12: 0x2018df00 (stale) Mirror 14: 0x2018df00 (stale) Mirror 17: 0x2018df00 (stale) Mirror 19: 0x2018df00 (stale) Mirror 21: 0x2018df00 (stale) Mirror 22: 0x2018df00 (stale) Mirror 23: 0x2018df00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x186c9): Mirror 4: 0x2018df00 (stale) Mirror 11: 0x2018df00 (stale) Mirror 12: 0x2018df00 (stale) Mirror 14: 0x2018df00 (stale) Mirror 17: 0x2018df00 (stale) Mirror 19: 0x2018df00 (stale) Mirror 21: 0x2018df00 (stale) Mirror 22: 0x2018df00 (stale) Mirror 23: 0x2018df00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x186c9: skipped 000850 1779976163.441310 mapwrite 0x0113fb thru 0x017ac4 (0x66ca bytes) 000851 1779976163.514531 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x186c9): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x186c9): Mirror 4: 0x2018df00 (stale) Mirror 11: 0x2018df00 (stale) Mirror 12: 0x2018df00 (stale) Mirror 14: 0x2018df00 (stale) Mirror 17: 0x2018df00 (stale) Mirror 19: 0x2018df00 (stale) Mirror 21: 0x2018df00 (stale) Mirror 22: 0x2018df00 (stale) Mirror 23: 0x2018df00 (stale) CRC-32 checksum value for chunk [0, 0x186c9): Mirror 1: 0xacdb6345 Mirror 24: 0xacdb6345 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x186c9: skipped 000852 1779976164.606796 mapread 0x00968a thru 0x00ead3 (0x544a bytes) 000853 1779976164.633097 trunc from 0x0186c9 to 0x01875a (0x92 bytes) 000854 1779976164.790363 read 0x00acf4 thru 0x00bc42 (0xf4f bytes) 000855 1779976164.804714 mapwrite 0x0109eb thru 0x015d6a (0x5380 bytes) 000857 1779976164.852784 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1875b): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1875b): Mirror 4: 0x56c2f5d3 (stale) Mirror 11: 0x56c2f5d3 (stale) Mirror 12: 0x56c2f5d3 (stale) Mirror 14: 0x56c2f5d3 (stale) Mirror 17: 0x56c2f5d3 (stale) Mirror 19: 0x56c2f5d3 (stale) Mirror 21: 0x56c2f5d3 (stale) Mirror 22: 0x56c2f5d3 (stale) Mirror 23: 0x56c2f5d3 (stale) Mirror 24: 0x9b11accc (stale) CRC-32 checksum value for chunk [0, 0x1875b): Mirror 1: 0x2f81c141 Mirror 25: 0x2f81c141 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1875b: skipped 000858 1779976165.946821 write 0x00a3ab thru 0x0170a9 (0xccff bytes) 000859 1779976166.025049 trunc from 0x01875b to 0x023036 (0xa8dc bytes) 000864 1779976166.065812 trunc from 0x023037 to 0x037258 (0x14222 bytes) 000865 1779976166.086905 mapread 0x035955 thru 0x036f38 (0x15e4 bytes) 000869 1779976166.107995 mapread 0x0288d3 thru 0x030f08 (0x8636 bytes) 000872 1779976166.124895 read 0x015b1d thru 0x01c435 (0x6919 bytes) 000873 1779976166.136813 mapwrite 0x020740 thru 0x027f00 (0x77c1 bytes) 000874 1779976166.218945 trunc from 0x037259 to 0x00e33f (0x28f19 bytes) 000875 1779976166.255784 trunc from 0x00e340 to 0x005604 (0x8d3b bytes) 000876 1779976166.290774 trunc from 0x005605 to 0x020fee (0x1b9ea bytes) 000877 1779976166.329972 mapwrite 0x008a7a thru 0x008cb5 (0x23c bytes) 000878 1779976166.445016 mapread 0x011d1d thru 0x01a3bb (0x869f bytes) 000879 1779976166.495396 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x20fef): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fef): Mirror 4: 0xbff6df00 (stale) Mirror 11: 0xbff6df00 (stale) Mirror 12: 0xbff6df00 (stale) Mirror 14: 0xbff6df00 (stale) Mirror 17: 0xbff6df00 (stale) Mirror 19: 0xbff6df00 (stale) Mirror 21: 0xbff6df00 (stale) Mirror 22: 0xbff6df00 (stale) Mirror 23: 0xbff6df00 (stale) Mirror 24: 0x31d54a66 (stale) Mirror 25: 0x8e3c7d33 (stale) CRC-32 checksum value for chunk [0, 0x20fef): Mirror 1: 0x3d381859 Mirror 26: 0x3d381859 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fef: skipped 000880 1779976167.654664 MIRROR_RESYNC 13 mirrors 000881 1779976169.443735 trunc from 0x020fef to 0x02b3d0 (0xa3e2 bytes) 000883 1779976169.592394 write 0x03fc33 thru 0x03ffff (0x3cd bytes) 000884 1779976169.601379 MIRROR_RESYNC 13 mirrors 000885 1779976171.687374 mapread 0x01bd18 thru 0x0292d5 (0xd5be bytes) 000886 1779976171.721413 read 0x00fa64 thru 0x010cb1 (0x124e bytes) 000888 1779976171.726880 mapread 0x027baa thru 0x02fb79 (0x7fd0 bytes) 000890 1779976171.737958 mapwrite 0x03851c thru 0x03ffff (0x7ae4 bytes) 000893 1779976171.961462 write 0x02e182 thru 0x03954d (0xb3cc bytes) 000894 1779976171.976361 mapwrite 0x006917 thru 0x0116ff (0xade9 bytes) 000895 1779976172.070823 trunc from 0x040000 to 0x015f76 (0x2a089 bytes) 000896 1779976172.096199 trunc from 0x015f77 to 0x02db61 (0x17beb bytes) 000902 1779976172.125882 read 0x00cef8 thru 0x014e89 (0x7f92 bytes) 000904 1779976172.128857 mapwrite 0x03fc5a thru 0x03ffff (0x3a6 bytes) 000906 1779976172.227096 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4074ca00 (stale) Mirror 11: 0x4074ca00 (stale) Mirror 12: 0x4074ca00 (stale) Mirror 14: 0x4074ca00 (stale) Mirror 17: 0x4074ca00 (stale) Mirror 19: 0x4074ca00 (stale) Mirror 21: 0x4074ca00 (stale) Mirror 22: 0x4074ca00 (stale) Mirror 23: 0x4074ca00 (stale) Mirror 24: 0x4074ca00 (stale) Mirror 25: 0x4074ca00 (stale) Mirror 26: 0x4074ca00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c31205e Mirror 27: 0x3c31205e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000907 1779976173.676607 MIRROR_RESYNC 14 mirrors 000908 1779976175.398677 mapread 0x03e1e0 thru 0x03ffff (0x1e20 bytes) 000909 1779976175.430126 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000910 1779976176.245142 read 0x036451 thru 0x03ffff (0x9baf bytes) 000912 1779976176.253509 read 0x034c18 thru 0x03ffff (0xb3e8 bytes) 000915 1779976176.316690 trunc from 0x040000 to 0x013911 (0x2c6ee bytes) 000916 1779976176.487659 read 0x011128 thru 0x013911 (0x27ea bytes) 000919 1779976176.499302 read 0x00dda0 thru 0x013911 (0x5b72 bytes) 000921 1779976176.502557 mapwrite 0x017b9b thru 0x01b8a5 (0x3d0b bytes) 000924 1779976176.579768 mapread 0x006d92 thru 0x014ec5 (0xe134 bytes) 000925 1779976176.581801 write 0x00f9e0 thru 0x011f46 (0x2567 bytes) 000927 1779976176.586105 mapread 0x012656 thru 0x01b8a5 (0x9250 bytes) 000928 1779976176.588600 trunc from 0x01b8a6 to 0x025f71 (0xa6cc bytes) 000935 1779976176.621577 mapwrite 0x00737e thru 0x0143bb (0xd03e bytes) 000937 1779976176.667155 mapwrite 0x01a04a thru 0x023792 (0x9749 bytes) 000938 1779976176.728018 write 0x028683 thru 0x02b94b (0x32c9 bytes) 000940 1779976176.730545 mapread 0x011ffd thru 0x012213 (0x217 bytes) 000941 1779976176.736813 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 4: 0xe6eaf1f7 (stale) Mirror 11: 0xe6eaf1f7 (stale) Mirror 12: 0xe6eaf1f7 (stale) Mirror 19: 0xe6eaf1f7 (stale) Mirror 21: 0xe6eaf1f7 (stale) Mirror 22: 0xe6eaf1f7 (stale) Mirror 23: 0xe6eaf1f7 (stale) Mirror 24: 0xe6eaf1f7 (stale) Mirror 25: 0xe6eaf1f7 (stale) Mirror 26: 0xe6eaf1f7 (stale) Mirror 27: 0xe6eaf1f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 4: 0xe6eaf1f7 (stale) Mirror 11: 0xe6eaf1f7 (stale) Mirror 12: 0xe6eaf1f7 (stale) Mirror 19: 0xe6eaf1f7 (stale) Mirror 21: 0xe6eaf1f7 (stale) Mirror 22: 0xe6eaf1f7 (stale) Mirror 23: 0xe6eaf1f7 (stale) Mirror 24: 0xe6eaf1f7 (stale) Mirror 25: 0xe6eaf1f7 (stale) Mirror 26: 0xe6eaf1f7 (stale) Mirror 27: 0xe6eaf1f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b94c: skipped 000942 1779976177.787719 mapread 0x01a3f2 thru 0x01f106 (0x4d15 bytes) 000944 1779976177.811112 mapread 0x01049e thru 0x015602 (0x5165 bytes) 000946 1779976177.830994 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 11: 0xe6eaf1f7 (stale) Mirror 12: 0xe6eaf1f7 (stale) Mirror 19: 0xe6eaf1f7 (stale) Mirror 21: 0xe6eaf1f7 (stale) Mirror 22: 0xe6eaf1f7 (stale) Mirror 23: 0xe6eaf1f7 (stale) Mirror 24: 0xe6eaf1f7 (stale) Mirror 25: 0xe6eaf1f7 (stale) Mirror 26: 0xe6eaf1f7 (stale) Mirror 27: 0xe6eaf1f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 11: 0xe6eaf1f7 (stale) Mirror 12: 0xe6eaf1f7 (stale) Mirror 19: 0xe6eaf1f7 (stale) Mirror 21: 0xe6eaf1f7 (stale) Mirror 22: 0xe6eaf1f7 (stale) Mirror 23: 0xe6eaf1f7 (stale) Mirror 24: 0xe6eaf1f7 (stale) Mirror 25: 0xe6eaf1f7 (stale) Mirror 26: 0xe6eaf1f7 (stale) Mirror 27: 0xe6eaf1f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b94c: skipped 000949 1779976178.711366 trunc from 0x02b94c to 0x01ce71 (0xeada bytes) 000954 1779976178.737016 mapwrite 0x02f593 thru 0x02fd3c (0x7aa bytes) 000956 1779976178.821535 mapwrite 0x02dcde thru 0x031a89 (0x3dac bytes) 000958 1779976179.028019 trunc from 0x031a8a to 0x02a1c8 (0x78c1 bytes) 000959 1779976179.067191 mapread 0x021148 thru 0x02a1c8 (0x9081 bytes) 000963 1779976179.102768 mapwrite 0x02f823 thru 0x02fdb2 (0x590 bytes) 000964 1779976179.212691 read 0x01fb8e thru 0x0261ce (0x6641 bytes) 000965 1779976179.215780 read 0x012a32 thru 0x0208c8 (0xde97 bytes) 000966 1779976179.236473 mapread 0x00d6fe thru 0x01d356 (0xfc59 bytes) 000967 1779976179.238508 read 0x01461c thru 0x01c502 (0x7ee7 bytes) 000968 1779976179.239334 trunc from 0x02fdb3 to 0x007302 (0x28ab0 bytes) 000970 1779976179.270203 read 0x000a08 thru 0x007302 (0x68fb bytes) 000972 1779976179.278549 read 0x005e27 thru 0x006bab (0xd85 bytes) 000974 1779976179.280710 trunc from 0x007303 to 0x0222aa (0x1afa8 bytes) 000975 1779976179.342062 mapwrite 0x011654 thru 0x0157e2 (0x418f bytes) 000976 1779976179.592017 write 0x004820 thru 0x00dc1b (0x93fc bytes) 000977 1779976179.594993 read 0x009f64 thru 0x00d058 (0x30f5 bytes) 000979 1779976179.605450 MIRROR_RESYNC 11 mirrors 000980 1779976181.633396 MIRROR_RESYNC 11 mirrors 000981 1779976182.410956 trunc from 0x0222ab to 0x036620 (0x14376 bytes) 000982 1779976182.553378 read 0x032858 thru 0x036620 (0x3dc9 bytes) 000985 1779976182.584351 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36621): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36621): Mirror 11: 0xcb122d68 (stale) Mirror 12: 0xcb122d68 (stale) Mirror 19: 0xcb122d68 (stale) Mirror 21: 0xcb122d68 (stale) Mirror 22: 0xcb122d68 (stale) Mirror 23: 0xcb122d68 (stale) Mirror 24: 0xcb122d68 (stale) Mirror 25: 0xcb122d68 (stale) Mirror 26: 0xcb122d68 (stale) Mirror 27: 0xcb122d68 (stale) CRC-32 checksum value for chunk [0, 0x36621): Mirror 1: 0xcb122d68 Mirror 28: 0xcb122d68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36621: skipped 000986 1779976184.307651 mapread 0x00695b thru 0x00900b (0x26b1 bytes) 000987 1779976184.371883 trunc from 0x036621 to 0x028555 (0xe0cb bytes) 000988 1779976184.601396 read 0x011d54 thru 0x011d5c (0x9 bytes) 000989 1779976184.608578 write 0x01112a thru 0x017525 (0x63fc bytes) 000990 1779976184.618603 mapread 0x014060 thru 0x021d7b (0xdd1c bytes) 000991 1779976184.640511 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x28556): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28556): Mirror 11: 0x3314d006 (stale) Mirror 12: 0x3314d006 (stale) Mirror 19: 0x3314d006 (stale) Mirror 21: 0x3314d006 (stale) Mirror 22: 0x3314d006 (stale) Mirror 23: 0x3314d006 (stale) Mirror 24: 0x3314d006 (stale) Mirror 25: 0x3314d006 (stale) Mirror 26: 0x3314d006 (stale) Mirror 27: 0x3314d006 (stale) Mirror 28: 0x3314d006 (stale) CRC-32 checksum value for chunk [0, 0x28556): Mirror 1: 0xc507e68 Mirror 29: 0xc507e68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28556: skipped 000992 1779976186.075872 mapread 0x00f1db thru 0x01235f (0x3185 bytes) 000994 1779976186.135433 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x28556): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28556): Mirror 11: 0x3314d006 (stale) Mirror 12: 0x3314d006 (stale) Mirror 19: 0x3314d006 (stale) Mirror 21: 0x3314d006 (stale) Mirror 22: 0x3314d006 (stale) Mirror 23: 0x3314d006 (stale) Mirror 24: 0x3314d006 (stale) Mirror 25: 0x3314d006 (stale) Mirror 26: 0x3314d006 (stale) Mirror 27: 0x3314d006 (stale) Mirror 28: 0x3314d006 (stale) CRC-32 checksum value for chunk [0, 0x28556): Mirror 1: 0xc507e68 Mirror 29: 0xc507e68 Mirror 30: 0xc507e68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28556: skipped 000995 1779976187.467399 mapread 0x0030d0 thru 0x00a668 (0x7599 bytes) 000996 1779976187.490066 write 0x0164ed thru 0x02202e (0xbb42 bytes) 000997 1779976187.591008 trunc from 0x028556 to 0x03db24 (0x155cf bytes) 000999 1779976187.610970 trunc from 0x03db25 to 0x0290ba (0x14a6a bytes) 001000 1779976187.623528 write 0x02c0f9 thru 0x02c4f4 (0x3fc bytes) All operations completed A-OK! PASS 70a (168s) == sanity-flr test 71: check mirror extend parent fid ==== 09:49:53 (1779976193) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 09:49:59 (1779976199) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.455349 s, 4.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306762 s, 3.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x230:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x230:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x22d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 09:50:10 (1779976210) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1225664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4092896 .. Extending file size to 7782464 .. Extending file size to 5244352 .. Extending file size to 7008992 .. Extending file size to 8129472 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8616000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8644704 .. Extending file size to 8902720 .. Extending file size to 8187680 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8250240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8587424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8770528 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8909120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 55475 55477 55478 55480 55481 done 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:51:29 (1779976289) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5990688 .. Extending file size to 7199296 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8573504 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6539584 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7440544 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8414304 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8882400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9362752 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8965280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195072 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57281 57282 57283 57287 57288 done 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) 192.168.201.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:52:48 (1779976368) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131805 s, 15.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369807 s, 5.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x13e05423 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:53:14 (1779976394) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125108 s, 16.8 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 09:53:23 (1779976403) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1db:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x246:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x246:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x242:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:53:31 (1779976411) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143165 s, 14.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x243:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x244:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:53:39 (1779976419) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263722 s, 8.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x245:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x245:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x246:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:53:47 (1779976427) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298908 s, 7.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x247:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x247:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x248:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:22 (1779976462) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133679 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24d:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:23 (1779976523) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192386 s, 10.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:56:17 (1779976577) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg131-server mds-ost sync done. Creating new pool test_204f oleg131-server: Pool lustre.test_204f created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.449498 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg131-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:34 (1779976654) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:41 (1779976661) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:48 (1779976668) oleg131-server: Pool lustre.test_206 created Adding targets to pool oleg131-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg131-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg131-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg131-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:58:13 (1779976693) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117498 s, 8.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790734 s, 13.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x30a555b7 Mirror 3: 0x30a555b7 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:58:24 (1779976704) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:58:27 (1779976707) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:58:30 (1779976710) Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.172233 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x22c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c10000 Read this (4 bytes): BBBB 192.168.201.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg131-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:59:11 (1779976751) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0381912 s, 214 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:59:20 (1779976760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806063 s, 13.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x259:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834432 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-28 09:59:22.000000000 -0400 Modify: 2026-05-28 09:59:22.000000000 -0400 Change: 2026-05-28 09:59:24.000000000 -0400 Birth: 2026-05-28 09:59:22.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:59:30 (1779976770) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:59:33 (1779976773) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.440232 s, 93.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0605681 s, 67.6 kB/s size after second write -rw-r--r-- 1 root root 4096 May 28 09:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 28 09:59 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:59:43 (1779976783) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:59:52 (1779976792) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1846 sec =========== 10:00:03 (1779976803) === sanity-flr: start cleanup 10:00:05 (1779976805) === === sanity-flr: finish cleanup 10:00:13 (1779976813) ===