-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 19:59:21 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg426-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:59:35 (1763341175) === oleg426-client.virtnet: executing check_config_client /mnt/lustre oleg426-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg426-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93ba5195a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93ba5195a800.idle_timeout=debug disable quota as required oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:00:03 (1763341203) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7574] == sanity-flr test 0a: lfs mirror create with -N option == 20:00:04 (1763341204) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:00:17 (1763341217) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:00:19 (1763341219) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:00:22 (1763341222) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:00:43 (1763341243) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:00:46 (1763341246) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:00:49 (1763341249) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93ba5195a800.stats=clear osc.lustre-OST0001-osc-ffff93ba5195a800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:00:56 (1763341256) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:01:06 (1763341266) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:01:13 (1763341273) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:01:15 (1763341275) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:01:23 (1763341283) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:01:29 (1763341289) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:01:33 (1763341293) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:01:40 (1763341300) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:01:48 (1763341308) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:01:54 (1763341314) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.52142 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180722 s, 5.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:02:03 (1763341323) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.328057 s, 32.0 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:02:11 (1763341331) Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:03:16 (1763341396) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.71298 s, 14.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:03:51 (1763341431) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:05:03 (1763341503) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.97573 s, 10.5 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 20:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.234s user 0m0.001s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.218s user 0m0.001s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg426-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg426-server oleg426-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.283s user 0m0.001s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg426-server: oleg426-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg426-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ba5195a800.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:06:30 (1763341590) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:06:33 (1763341593) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:06:37 (1763341597) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:06:40 (1763341600) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:06:47 (1763341607) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.465465 s, 27.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:06:59 (1763341619) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526272 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229353 s, 45.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:07:06 (1763341626) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:07:15 (1763341635) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101289 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00068634 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00221511 s, 462 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00096685 s, 1.1 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 20:07:25 (1763341645) create test files with size 15747288 .. bdc1cb7da532f80b1714b130cd30d178 - 3395a403ef2778aed2599c61cb2f893b - 14aa7b352ab95036b678fce28dbafa8f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 20:07:51 (1763341671) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:08:05 (1763341685) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:08:13 (1763341693) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135825 s, 7.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187819 s, 55.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 20:08:22 (1763341702) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154558 s, 13.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.398248 s, 10.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280486 s, 7.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 20:08:43 (1763341723) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:08:46 (1763341726) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:08:50 (1763341730) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583831 s, 18.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 20:09:00 (1763341740) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:09:03 (1763341743) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:09:15 (1763341755) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.398834 s, 26.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:09:27 (1763341767) llite.lustre-ffff93ba4998a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.187933 s, 55.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg426-client.virtnet /mnt/lustre (opts:) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff93ba58ab4000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:09:42 (1763341782) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:10:20 (1763341820) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:10:58 (1763341858) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:11:02 (1763341862) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241157 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.129354 s, 24.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x266a47d1 (stale) Mirror 2: 0x266a47d1 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 5dff4d539d023f5fc3d8610e4a1e8fe4 - mirror 1 checksum: 293915935dd596245facb7fa60c85fec - mirror 2 checksum: 5dff4d539d023f5fc3d8610e4a1e8fe4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 20:11:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 20:11:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5dff4d539d023f5fc3d8610e4a1e8fe4 - mirror 1 checksum: 293915935dd596245facb7fa60c85fec - mirror 2 checksum: 293915935dd596245facb7fa60c85fec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 20:11:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 20:11:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:11:15 (1763341875) Failing mds1 on oleg426-server Stopping /mnt/lustre-mds1 (opts:) on oleg426-server 20:11:22 (1763341882) shut down facet: mds1 facet_host: oleg426-server facet_failover_host: oleg426-server Failover mds1 to oleg426-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:52 (1763341912) targets are mounted 20:11:52 (1763341912) facet_failover done oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308754 s, 34.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:12:08 (1763341928) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:12:12 (1763341932) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:12:15 (1763341935) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0167124 s, 980 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:12:23 (1763341943) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:12:27 (1763341947) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:12:30 (1763341950) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:12:52 (1763341972) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:13:20 (1763342000) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:13:44 (1763342024) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:14:09 (1763342049) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:14:17 (1763342057) mirror create pid 52304 mirror split pid 52305 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52304 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52305 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:15:24 (1763342124) Seed set to 2126 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1763342126.910050 mapwrite 0x005454 thru 0x00ddf7 (0x89a4 bytes) 000007 1763342127.051385 read 0x003f52 thru 0x00ddf7 (0x9ea6 bytes) 000008 1763342127.052133 write 0x009395 thru 0x0117cd (0x8439 bytes) 000010 1763342127.065965 write 0x003566 thru 0x004b41 (0x15dc bytes) truncating to largest ever: 0x240d2 000011 1763342127.071552 trunc from 0x0117ce to 0x0240d1 (0x12904 bytes) 000012 1763342127.100708 write 0x0086af thru 0x00b9ac (0x32fe bytes) 000016 1763342127.109202 read 0x001e83 thru 0x007749 (0x58c7 bytes) 000018 1763342127.112540 mapwrite 0x030568 thru 0x03c5fe (0xc097 bytes) 000019 1763342127.303426 read 0x026fc2 thru 0x0352b5 (0xe2f4 bytes) 000021 1763342127.306152 mapwrite 0x033b79 thru 0x0382ee (0x4776 bytes) 000023 1763342127.345744 write 0x02b210 thru 0x034a7d (0x986e bytes) truncating to largest ever: 0x3532b 000024 1763342127.348271 trunc from 0x03c5ff to 0x03532a (0x72d4 bytes) 000025 1763342127.367432 mapread 0x00f0c1 thru 0x0141ad (0x50ed bytes) 000027 1763342127.368861 read 0x0013d0 thru 0x00356e (0x219f bytes) 000031 1763342127.372887 mapwrite 0x03ffc5 thru 0x03ffff (0x3b bytes) 000033 1763342127.493593 read 0x00bdda thru 0x00e0ec (0x2313 bytes) 000034 1763342127.494823 write 0x00d680 thru 0x00f200 (0x1b81 bytes) 000036 1763342127.496762 mapwrite 0x02e83f thru 0x033e21 (0x55e3 bytes) 000037 1763342127.589228 trunc from 0x040000 to 0x034725 (0xb8da bytes) 000042 1763342127.637679 trunc from 0x034726 to 0x0277b0 (0xcf75 bytes) 000044 1763342127.686532 trunc from 0x0277b1 to 0x0307bb (0x900b bytes) 000045 1763342127.706968 mapread 0x01cd8c thru 0x01e4eb (0x1760 bytes) 000048 1763342127.708836 read 0x01d224 thru 0x02af6d (0xdd4a bytes) 000050 1763342127.748021 mapwrite 0x014b40 thru 0x01f8d2 (0xad93 bytes) 000051 1763342127.861145 mapread 0x010ec3 thru 0x01a395 (0x94d3 bytes) 000052 1763342127.862028 mapwrite 0x0397ff thru 0x03adc0 (0x15c2 bytes) 000053 1763342128.004574 trunc from 0x03adc1 to 0x010cb5 (0x2a10b bytes) 000054 1763342128.044634 trunc from 0x010cb6 to 0x003f90 (0xcd25 bytes) 000055 1763342128.081672 read 0x00065c thru 0x003f90 (0x3935 bytes) 000056 1763342128.082388 mapread 0x00292b thru 0x003f90 (0x1666 bytes) 000057 1763342128.083184 mapread 0x00010c thru 0x003f90 (0x3e85 bytes) 000058 1763342128.087154 mapwrite 0x036d3f thru 0x038322 (0x15e4 bytes) 000059 1763342128.226630 mapwrite 0x008f97 thru 0x00a367 (0x13d1 bytes) 000060 1763342128.324091 write 0x036dc0 thru 0x03ffff (0x9240 bytes) 000062 1763342128.325971 mapread 0x00275a thru 0x0058b7 (0x315e bytes) 000063 1763342128.330587 mapwrite 0x03f8f3 thru 0x03ffff (0x70d bytes) 000065 1763342128.387872 read 0x035e8d thru 0x0381dd (0x2351 bytes) 000067 1763342128.389067 mapwrite 0x005504 thru 0x00f3e4 (0x9ee1 bytes) 000068 1763342128.433794 write 0x007df5 thru 0x009832 (0x1a3e bytes) 000070 1763342128.435451 read 0x013cc7 thru 0x01905c (0x5396 bytes) 000072 1763342128.437439 mapwrite 0x0335cc thru 0x03c440 (0x8e75 bytes) 000073 1763342128.485723 write 0x00fa71 thru 0x01e971 (0xef01 bytes) 000075 1763342128.488245 trunc from 0x040000 to 0x02a82f (0x157d0 bytes) 000076 1763342128.524087 trunc from 0x02a830 to 0x016c29 (0x13c06 bytes) 000078 1763342128.541944 mapwrite 0x01e29e thru 0x020829 (0x258c bytes) 000082 1763342128.641458 trunc from 0x02082a to 0x01d269 (0x35c0 bytes) 000083 1763342128.671946 mapwrite 0x039948 thru 0x03ffff (0x66b8 bytes) 000085 1763342128.827690 mapread 0x03738c thru 0x03ffff (0x8c74 bytes) 000086 1763342128.828777 mapread 0x016db1 thru 0x017c7f (0xecf bytes) 000087 1763342128.835452 mapwrite 0x03ccb7 thru 0x03ffff (0x3349 bytes) 000088 1763342128.923889 read 0x009a73 thru 0x00c394 (0x2922 bytes) 000089 1763342128.924566 mapread 0x00d75b thru 0x010193 (0x2a39 bytes) 000095 1763342128.934281 mapwrite 0x00b358 thru 0x00d7cc (0x2475 bytes) 000096 1763342128.977332 mapread 0x02b0a7 thru 0x030b73 (0x5acd bytes) 000097 1763342128.978247 trunc from 0x040000 to 0x01310f (0x2cef0 bytes) 000098 1763342129.006756 mapwrite 0x00e6ae thru 0x00f35e (0xcb1 bytes) 000100 1763342129.053523 write 0x037ec2 thru 0x03ffff (0x813e bytes) 000101 1763342129.055612 read 0x009586 thru 0x0128b3 (0x932e bytes) 000102 1763342129.056145 trunc from 0x040000 to 0x0224f8 (0x1db07 bytes) 000103 1763342129.075172 write 0x01c2c1 thru 0x01e198 (0x1ed8 bytes) 000104 1763342129.089056 mapread 0x00cd94 thru 0x00d6d6 (0x943 bytes) 000108 1763342129.091715 read 0x011ae1 thru 0x0176e0 (0x5c00 bytes) 000109 1763342129.092902 write 0x005e28 thru 0x00c2b0 (0x6489 bytes) truncating to largest ever: 0x3c71c 000112 1763342129.097688 trunc from 0x0224f9 to 0x03c71b (0x1a223 bytes) 000113 1763342129.118534 trunc from 0x03c71c to 0x01b3a8 (0x21373 bytes) 000115 1763342129.141194 write 0x025f16 thru 0x02f322 (0x940d bytes) 000116 1763342129.142628 trunc from 0x02f323 to 0x01479e (0x1ab84 bytes) 000123 1763342129.175599 read 0x004129 thru 0x00e3d1 (0xa2a9 bytes) 000126 1763342129.182096 trunc from 0x01479f to 0x02e441 (0x19ca3 bytes) 000127 1763342129.209368 read 0x00119d thru 0x00f89b (0xe6ff bytes) 000128 1763342129.210091 mapwrite 0x03e3af thru 0x03ffff (0x1c51 bytes) 000129 1763342129.290666 write 0x0113d7 thru 0x0116e2 (0x30c bytes) 000133 1763342129.294135 mapwrite 0x0014e8 thru 0x008059 (0x6b72 bytes) 000136 1763342129.424172 mapwrite 0x01f9ea thru 0x021910 (0x1f27 bytes) 000138 1763342129.461967 mapread 0x00c3db thru 0x01062b (0x4251 bytes) 000141 1763342129.463718 mapwrite 0x00446b thru 0x00f4a0 (0xb036 bytes) 000145 1763342129.512052 MIRROR_EXTEND to 2 mirrors 000146 1763342130.139854 read 0x03184e thru 0x03b246 (0x99f9 bytes) 000147 1763342130.158325 read 0x00fe2e thru 0x015226 (0x53f9 bytes) 000148 1763342130.172113 mapread 0x0315ee thru 0x03ec1c (0xd62f bytes) 000151 1763342130.176903 read 0x035a4b thru 0x03c292 (0x6848 bytes) 000152 1763342130.178937 trunc from 0x040000 to 0x00f31c (0x30ce3 bytes) 000154 1763342130.317972 write 0x03f49a thru 0x03ffff (0xb66 bytes) 000155 1763342130.319121 mapwrite 0x015afb thru 0x01dbd7 (0x80dd bytes) 000158 1763342130.390857 mapread 0x02d128 thru 0x033677 (0x6550 bytes) 000159 1763342130.391948 read 0x0123a2 thru 0x01b41f (0x907e bytes) 000160 1763342130.392570 read 0x02241f thru 0x031c16 (0xf7f8 bytes) 000161 1763342130.393917 write 0x02caf0 thru 0x038da4 (0xc2b5 bytes) 000162 1763342130.396444 trunc from 0x040000 to 0x01ceb3 (0x2314c bytes) 000164 1763342130.426463 write 0x03610d thru 0x03ffff (0x9ef3 bytes) 000165 1763342130.430245 read 0x005c22 thru 0x0076a5 (0x1a84 bytes) 000166 1763342130.431869 mapwrite 0x00231b thru 0x002bf8 (0x8de bytes) 000167 1763342130.479859 trunc from 0x040000 to 0x020612 (0x1f9ed bytes) 000168 1763342130.497228 mapwrite 0x00ac67 thru 0x011998 (0x6d32 bytes) 000169 1763342130.559977 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20613): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20613): Mirror 2: 0xfc509099 (stale) CRC-32 checksum value for chunk [0, 0x20613): Mirror 1: 0xcab16244 Mirror 3: 0xcab16244 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20613: skipped 000170 1763342131.387250 mapwrite 0x01934b thru 0x02469b (0xb351 bytes) 000172 1763342131.598202 mapwrite 0x037844 thru 0x03a171 (0x292e bytes) 000176 1763342131.698883 mapwrite 0x01ff2d thru 0x024f75 (0x5049 bytes) 000179 1763342131.800165 trunc from 0x03a172 to 0x039f00 (0x271 bytes) 000180 1763342131.843640 read 0x0122d4 thru 0x021587 (0xf2b4 bytes) 000181 1763342131.849858 mapread 0x02eedb thru 0x039f00 (0xb026 bytes) 000182 1763342131.871809 read 0x006d9a thru 0x01691e (0xfb85 bytes) 000183 1763342131.884473 trunc from 0x039f01 to 0x006057 (0x33ea9 bytes) 000184 1763342131.907856 mapread 0x002260 thru 0x006057 (0x3df8 bytes) 000185 1763342131.908801 trunc from 0x006058 to 0x008982 (0x292b bytes) 000187 1763342131.930716 mapread 0x001bef thru 0x008982 (0x6d94 bytes) 000189 1763342131.945262 trunc from 0x008983 to 0x015325 (0xc9a3 bytes) 000190 1763342131.969943 mapwrite 0x01f93d thru 0x021f54 (0x2618 bytes) 000192 1763342132.090246 write 0x03d823 thru 0x03eb74 (0x1352 bytes) 000194 1763342132.108592 read 0x0161b0 thru 0x01eb75 (0x89c6 bytes) 000195 1763342132.129074 write 0x00cd4e thru 0x00df69 (0x121c bytes) 000197 1763342132.147090 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3eb75): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb75): Mirror 2: 0xa8a003f7 (stale) Mirror 3: 0x38d6aecc (stale) CRC-32 checksum value for chunk [0, 0x3eb75): Mirror 1: 0x79ed3479 Mirror 4: 0x79ed3479 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb75: skipped 000199 1763342133.031925 mapwrite 0x01973e thru 0x01df2d (0x47f0 bytes) 000200 1763342133.307653 mapwrite 0x00cbe5 thru 0x01a9c2 (0xddde bytes) 000206 1763342133.386403 write 0x00d694 thru 0x0137e7 (0x6154 bytes) 000207 1763342133.387402 mapwrite 0x003a43 thru 0x00fdeb (0xc3a9 bytes) 000208 1763342133.501443 read 0x0117ab thru 0x015990 (0x41e6 bytes) 000209 1763342133.503267 write 0x015ab5 thru 0x016417 (0x963 bytes) 000210 1763342133.505793 write 0x034905 thru 0x0365a4 (0x1ca0 bytes) 000212 1763342133.538623 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb75): Mirror 3: 0x38d6aecc (stale) Mirror 4: 0x79ed3479 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb75): Mirror 3: 0x38d6aecc (stale) Mirror 4: 0x79ed3479 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb75: skipped 000213 1763342134.000714 write 0x02ada1 thru 0x03005c (0x52bc bytes) 000214 1763342134.013902 trunc from 0x03eb75 to 0x02837e (0x167f6 bytes) 000215 1763342134.063518 write 0x027900 thru 0x02b851 (0x3f52 bytes) 000218 1763342134.077415 mapwrite 0x0127ea thru 0x01fcc8 (0xd4df bytes) 000219 1763342134.170474 write 0x0196ef thru 0x027c1c (0xe52e bytes) 000220 1763342134.173381 mapread 0x000149 thru 0x000df8 (0xcb0 bytes) 000222 1763342134.199481 mapread 0x022e8f thru 0x024c3c (0x1dae bytes) 000224 1763342134.200832 write 0x035a64 thru 0x03ffff (0xa59c bytes) 000226 1763342134.212123 mapread 0x003321 thru 0x0079ed (0x46cd bytes) 000227 1763342134.218684 write 0x01939d thru 0x01b5c4 (0x2228 bytes) 000230 1763342134.225136 read 0x0201a1 thru 0x02bd05 (0xbb65 bytes) 000231 1763342134.245967 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x884fb7f8 (stale) Mirror 4: 0xd81ae3bc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4defe3af Mirror 5: 0x4defe3af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1763342135.114123 mapread 0x0027d5 thru 0x008123 (0x594f bytes) 000237 1763342135.164845 mapread 0x037116 thru 0x03cf54 (0x5e3f bytes) 000238 1763342135.192524 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x884fb7f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x884fb7f8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4defe3af Mirror 5: 0x4defe3af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000239 1763342135.826986 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x884fb7f8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4defe3af Mirror 5: 0x4defe3af Mirror 6: 0x4defe3af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000241 1763342137.053277 mapwrite 0x010a28 thru 0x0193a9 (0x8982 bytes) 000242 1763342137.263557 read 0x02f845 thru 0x031320 (0x1adc bytes) 000243 1763342137.277615 MIRROR_RESYNC 4 mirrors 000248 1763342138.036805 MIRROR_EXTEND to 5 mirrors truncating to largest ever: 0x3f1cd 000249 1763342138.773205 trunc from 0x040000 to 0x03f1cc (0xe33 bytes) 000251 1763342138.943101 read 0x025378 thru 0x032cc1 (0xd94a bytes) 000253 1763342138.966779 trunc from 0x03f1cd to 0x03ae8a (0x4342 bytes) 000255 1763342138.995392 trunc from 0x03ae8b to 0x03d90a (0x2a80 bytes) 000256 1763342139.014745 trunc from 0x03d90b to 0x02075d (0x1d1ad bytes) 000257 1763342139.044954 read 0x004339 thru 0x0096d6 (0x539e bytes) 000258 1763342139.061446 write 0x000197 thru 0x001315 (0x117f bytes) 000259 1763342139.064367 write 0x034d53 thru 0x03ffff (0xb2ad bytes) 000260 1763342139.066878 mapread 0x00d67e thru 0x0131d5 (0x5b58 bytes) 000265 1763342139.071962 mapwrite 0x03e79d thru 0x03ffff (0x1863 bytes) 000268 1763342139.130351 mapwrite 0x003d6f thru 0x00602c (0x22be bytes) 000271 1763342139.188951 trunc from 0x040000 to 0x022685 (0x1d97a bytes) 000272 1763342139.208250 read 0x0191a7 thru 0x022685 (0x94df bytes) 000273 1763342139.216338 trunc from 0x022686 to 0x0288dd (0x6258 bytes) 000274 1763342139.238950 read 0x02438c thru 0x0288dd (0x4552 bytes) 000275 1763342139.247343 mapread 0x017849 thru 0x01f717 (0x7ecf bytes) 000276 1763342139.252865 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x288de): Mirror 3: 0x35831be5 (stale) Mirror 5: 0x35831be5 (stale) Mirror 6: 0x35831be5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288de): Mirror 3: 0x35831be5 (stale) Mirror 5: 0x35831be5 (stale) Mirror 6: 0x35831be5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288de: skipped 000277 1763342139.669421 read 0x01031a thru 0x01d822 (0xd509 bytes) 000279 1763342139.681714 trunc from 0x0288de to 0x00482c (0x240b1 bytes) 000280 1763342139.724520 mapread 0x003b2b thru 0x00482c (0xd02 bytes) 000281 1763342139.753071 write 0x02de05 thru 0x032251 (0x444d bytes) 000283 1763342139.754795 mapread 0x01a361 thru 0x01deba (0x3b5a bytes) 000284 1763342139.777218 trunc from 0x032252 to 0x01bd0d (0x16544 bytes) 000286 1763342139.801896 trunc from 0x01bd0e to 0x0336b4 (0x179a7 bytes) 000289 1763342139.824186 mapread 0x032ffc thru 0x0336b4 (0x6b9 bytes) 000291 1763342139.841089 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x336b5): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336b5): Mirror 3: 0x2c80d679 (stale) Mirror 5: 0x2c80d679 (stale) Mirror 6: 0x2c80d679 (stale) CRC-32 checksum value for chunk [0, 0x336b5): Mirror 1: 0xee22c2a8 Mirror 7: 0xee22c2a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336b5: skipped 000292 1763342140.634248 mapread 0x018b10 thru 0x022438 (0x9929 bytes) 000295 1763342140.692838 MIRROR_RESYNC 5 mirrors 000297 1763342141.541995 mapread 0x01ccca thru 0x0254bb (0x87f2 bytes) 000298 1763342141.565920 mapwrite 0x02fc23 thru 0x039338 (0x9716 bytes) 000302 1763342141.734279 trunc from 0x039339 to 0x014d4e (0x245ea bytes) 000303 1763342141.763138 mapwrite 0x0293db thru 0x02a3e4 (0x100a bytes) 000304 1763342141.863718 read 0x029e8c thru 0x02a3e4 (0x559 bytes) 000306 1763342141.864861 mapwrite 0x01e051 thru 0x01f501 (0x14b1 bytes) 000307 1763342141.912772 mapwrite 0x01a206 thru 0x01fed0 (0x5ccb bytes) 000309 1763342141.959404 write 0x023c45 thru 0x032da0 (0xf15c bytes) 000311 1763342141.963417 trunc from 0x032da1 to 0x03c3b4 (0x9614 bytes) 000312 1763342141.988125 mapwrite 0x002220 thru 0x011fc4 (0xfda5 bytes) 000313 1763342142.058021 mapwrite 0x011e5c thru 0x01ac36 (0x8ddb bytes) 000314 1763342142.109542 read 0x030994 thru 0x03c3b4 (0xba21 bytes) 000315 1763342142.118814 write 0x03350f thru 0x03ffff (0xcaf1 bytes) 000317 1763342142.120796 write 0x01fbb2 thru 0x0210d4 (0x1523 bytes) 000321 1763342142.136727 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9ac34faa (stale) Mirror 5: 0x9ac34faa (stale) Mirror 6: 0x9ac34faa (stale) Mirror 7: 0x9ac34faa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xabf3e1d2 Mirror 8: 0xabf3e1d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000323 1763342143.076924 mapread 0x018df0 thru 0x026d1b (0xdf2c bytes) 000325 1763342143.159263 trunc from 0x040000 to 0x01d4eb (0x22b14 bytes) 000328 1763342143.298138 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1d4ec): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d4ec): Mirror 3: 0xb22b7480 (stale) Mirror 5: 0xb22b7480 (stale) Mirror 6: 0xb22b7480 (stale) Mirror 7: 0xb22b7480 (stale) Mirror 8: 0xf6cfc879 (stale) CRC-32 checksum value for chunk [0, 0x1d4ec): Mirror 1: 0xf6cfc879 Mirror 9: 0xf6cfc879 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d4ec: skipped 000330 1763342144.277027 write 0x030276 thru 0x031f7a (0x1d05 bytes) 000332 1763342144.373233 mapread 0x02640a thru 0x0289fc (0x25f3 bytes) 000333 1763342144.430582 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31f7b): Mirror 3: 0xb264e850 (stale) Mirror 5: 0xb264e850 (stale) Mirror 6: 0xb264e850 (stale) Mirror 7: 0xb264e850 (stale) Mirror 8: 0xdfc95254 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f7b): Mirror 3: 0xb264e850 (stale) Mirror 5: 0xb264e850 (stale) Mirror 6: 0xb264e850 (stale) Mirror 7: 0xb264e850 (stale) Mirror 8: 0xdfc95254 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f7b: skipped 000336 1763342144.865230 mapwrite 0x01a89c thru 0x01e736 (0x3e9b bytes) 000337 1763342145.018338 read 0x02f8d9 thru 0x031f7a (0x26a2 bytes) 000338 1763342145.034224 read 0x013352 thru 0x0179ba (0x4669 bytes) 000339 1763342145.066750 write 0x03ad30 thru 0x03ffff (0x52d0 bytes) 000340 1763342145.084795 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9ac34faa (stale) Mirror 6: 0x9ac34faa (stale) Mirror 7: 0x9ac34faa (stale) Mirror 8: 0xabf3e1d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9ac34faa (stale) Mirror 6: 0x9ac34faa (stale) Mirror 7: 0x9ac34faa (stale) Mirror 8: 0xabf3e1d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000341 1763342145.528589 MIRROR_RESYNC 5 mirrors 000343 1763342146.571532 mapwrite 0x0143ab thru 0x020521 (0xc177 bytes) 000345 1763342146.759674 read 0x02ce5d thru 0x0341e9 (0x738d bytes) 000347 1763342146.777278 MIRROR_RESYNC 5 mirrors 000348 1763342147.908566 MIRROR_RESYNC 5 mirrors 000350 1763342148.500561 read 0x02787d thru 0x029e8e (0x2612 bytes) 000351 1763342148.509714 write 0x03bf9f thru 0x03ffff (0x4061 bytes) 000352 1763342148.575715 trunc from 0x040000 to 0x010b25 (0x2f4da bytes) 000353 1763342148.609875 write 0x00c13a thru 0x015663 (0x952a bytes) 000356 1763342148.618242 write 0x00bec9 thru 0x01502a (0x9162 bytes) 000357 1763342148.633176 read 0x01106d thru 0x0119da (0x96e bytes) 000358 1763342148.634724 read 0x00f9e0 thru 0x015663 (0x5c84 bytes) 000360 1763342148.635743 trunc from 0x015664 to 0x0094de (0xc185 bytes) 000361 1763342148.654894 mapread 0x008901 thru 0x0094de (0xbde bytes) 000362 1763342148.656511 mapread 0x003ce1 thru 0x0094de (0x57fe bytes) 000364 1763342148.663372 read 0x00137c thru 0x00596f (0x45f4 bytes) 000365 1763342148.664055 read 0x0041f2 thru 0x0050bc (0xecb bytes) 000366 1763342148.668727 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x94df): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94df): Mirror 3: 0x5878aed9 (stale) Mirror 6: 0x5878aed9 (stale) Mirror 7: 0x5878aed9 (stale) Mirror 8: 0x5878aed9 (stale) CRC-32 checksum value for chunk [0, 0x94df): Mirror 1: 0x5878aed9 Mirror 9: 0x5878aed9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94df: skipped 000370 1763342149.471500 MIRROR_RESYNC 6 mirrors 000371 1763342150.647336 read 0x0042bb thru 0x0045af (0x2f5 bytes) 000372 1763342150.662202 MIRROR_EXTEND to 7 mirrors 000374 1763342151.310020 trunc from 0x0094df to 0x014d28 (0xb84a bytes) 000375 1763342151.431078 trunc from 0x014d29 to 0x017150 (0x2428 bytes) 000376 1763342151.455930 write 0x0337a6 thru 0x03b546 (0x7da1 bytes) 000377 1763342151.464800 MIRROR_RESYNC 7 mirrors 000378 1763342152.698733 mapread 0x019ded thru 0x01f70b (0x591f bytes) 000379 1763342152.718672 mapread 0x039d0e thru 0x03b546 (0x1839 bytes) 000380 1763342152.731576 mapread 0x03b519 thru 0x03b546 (0x2e bytes) 000383 1763342152.732774 mapwrite 0x03946e thru 0x03ffff (0x6b92 bytes) 000384 1763342152.925631 mapread 0x010515 thru 0x01579b (0x5287 bytes) 000386 1763342152.939458 read 0x00c306 thru 0x016b3c (0xa837 bytes) 000387 1763342152.940073 trunc from 0x040000 to 0x0082dd (0x37d22 bytes) 000388 1763342152.987623 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x82de): Mirror 6: 0x4d42692b (stale) Mirror 7: 0x4d42692b (stale) Mirror 8: 0x4d42692b (stale) Mirror 9: 0x4d42692b (stale) Mirror 10: 0x4d42692b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x82de): Mirror 6: 0x4d42692b (stale) Mirror 7: 0x4d42692b (stale) Mirror 8: 0x4d42692b (stale) Mirror 9: 0x4d42692b (stale) Mirror 10: 0x4d42692b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x82de: skipped 000392 1763342153.411744 write 0x01134e thru 0x01350e (0x21c1 bytes) 000394 1763342153.418847 mapread 0x00403f thru 0x00915c (0x511e bytes) 000397 1763342153.461969 mapread 0x00b53d thru 0x00bf33 (0x9f7 bytes) 000398 1763342153.468495 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1350f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1350f): Mirror 6: 0xc0bf9856 (stale) Mirror 7: 0xc0bf9856 (stale) Mirror 8: 0xc0bf9856 (stale) Mirror 9: 0xc0bf9856 (stale) Mirror 10: 0xc0bf9856 (stale) CRC-32 checksum value for chunk [0, 0x1350f): Mirror 1: 0x2d0fcc6f Mirror 11: 0x2d0fcc6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1350f: skipped 000399 1763342154.286892 trunc from 0x01350f to 0x00422e (0xf2e0 bytes) 000400 1763342154.453200 write 0x004b69 thru 0x00a89b (0x5d33 bytes) 000404 1763342154.467190 trunc from 0x00a89c to 0x025eef (0x1b654 bytes) 000405 1763342154.489989 read 0x002de0 thru 0x006b77 (0x3d98 bytes) 000406 1763342154.496566 mapwrite 0x019cf8 thru 0x021eb6 (0x81bf bytes) 000408 1763342154.570849 write 0x011975 thru 0x0131ef (0x187b bytes) 000409 1763342154.579586 write 0x034c5e thru 0x03ec19 (0x9fbc bytes) 000411 1763342154.585381 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ec1a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec1a): Mirror 6: 0x3897ac (stale) Mirror 7: 0x3897ac (stale) Mirror 8: 0x3897ac (stale) Mirror 9: 0x3897ac (stale) Mirror 10: 0x3897ac (stale) Mirror 11: 0x67ea0d5a (stale) CRC-32 checksum value for chunk [0, 0x3ec1a): Mirror 1: 0x4cda0091 Mirror 12: 0x4cda0091 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec1a: skipped 000413 1763342155.521736 write 0x011e78 thru 0x021d1f (0xfea8 bytes) 000414 1763342155.680178 MIRROR_RESYNC 8 mirrors 000415 1763342157.301953 MIRROR_RESYNC 8 mirrors 000417 1763342157.905425 mapread 0x023279 thru 0x02d6dd (0xa465 bytes) 000418 1763342157.990585 MIRROR_EXTEND to 9 mirrors 000419 1763342158.821108 write 0x03e192 thru 0x03f62c (0x149b bytes) 000420 1763342158.944091 mapwrite 0x0278f3 thru 0x02fca6 (0x83b4 bytes) 000421 1763342159.020710 trunc from 0x03f62d to 0x0030e2 (0x3c54a bytes) 000423 1763342159.048914 read 0x000129 thru 0x0030e2 (0x2fba bytes) 000424 1763342159.058420 trunc from 0x0030e3 to 0x038023 (0x34f41 bytes) 000428 1763342159.092807 mapwrite 0x0007af thru 0x0018f0 (0x1142 bytes) 000429 1763342159.153936 mapwrite 0x02802a thru 0x02b76c (0x3743 bytes) 000432 1763342159.201473 mapread 0x035a9f thru 0x038023 (0x2585 bytes) 000437 1763342159.215954 trunc from 0x038024 to 0x0069af (0x31674 bytes) 000438 1763342159.239234 trunc from 0x0069b0 to 0x02f9e7 (0x29038 bytes) 000439 1763342159.261428 read 0x0247be thru 0x02a493 (0x5cd6 bytes) 000440 1763342159.274664 read 0x022dd9 thru 0x028ca1 (0x5ec9 bytes) 000441 1763342159.276910 write 0x0309a9 thru 0x032d65 (0x23bd bytes) 000443 1763342159.279242 trunc from 0x032d66 to 0x020bb7 (0x121ae bytes) 000444 1763342159.300865 trunc from 0x020bb8 to 0x01b817 (0x53a0 bytes) 000453 1763342159.323674 write 0x02b110 thru 0x0363fd (0xb2ee bytes) 000455 1763342159.326843 write 0x0309fd thru 0x03dec6 (0xd4ca bytes) 000457 1763342159.330111 mapread 0x014bc7 thru 0x023c37 (0xf071 bytes) 000460 1763342159.341961 mapwrite 0x019c3f thru 0x01d3f2 (0x37b4 bytes) 000461 1763342159.387186 mapwrite 0x0060c0 thru 0x009bce (0x3b0f bytes) 000462 1763342159.420437 mapread 0x001388 thru 0x010b88 (0xf801 bytes) 000464 1763342159.422200 trunc from 0x03dec7 to 0x01dae8 (0x203de bytes) 000466 1763342159.461007 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1dae9): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dae9): Mirror 6: 0x34934810 (stale) Mirror 7: 0x34934810 (stale) Mirror 8: 0x34934810 (stale) Mirror 9: 0x34934810 (stale) Mirror 10: 0x34934810 (stale) Mirror 11: 0x34934810 (stale) Mirror 12: 0x34934810 (stale) Mirror 13: 0x34934810 (stale) CRC-32 checksum value for chunk [0, 0x1dae9): Mirror 1: 0xf97574e4 Mirror 14: 0xf97574e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dae9: skipped 000467 1763342160.638518 mapread 0x01a26c thru 0x01dae8 (0x387d bytes) 000470 1763342160.695014 read 0x0071ec thru 0x014ee1 (0xdcf6 bytes) 000471 1763342160.715057 trunc from 0x01dae9 to 0x030bc2 (0x130da bytes) 000472 1763342160.846406 mapread 0x00854a thru 0x016019 (0xdad0 bytes) 000473 1763342160.866873 trunc from 0x030bc3 to 0x02623c (0xa986 bytes) 000475 1763342160.889016 write 0x03c628 thru 0x03f77f (0x3158 bytes) 000477 1763342160.890564 read 0x00a43e thru 0x018977 (0xe53a bytes) 000479 1763342160.892703 trunc from 0x03f780 to 0x02aae0 (0x14c9f bytes) 000480 1763342160.917513 mapread 0x00691c thru 0x011209 (0xa8ee bytes) 000481 1763342160.919433 mapwrite 0x01f91e thru 0x02c43e (0xcb21 bytes) 000482 1763342161.007456 trunc from 0x02c43f to 0x022872 (0x9bcc bytes) 000485 1763342161.028632 write 0x008274 thru 0x00c4be (0x424b bytes) 000486 1763342161.029664 read 0x00b16a thru 0x01a700 (0xf597 bytes) 000489 1763342161.031794 trunc from 0x022873 to 0x0007e2 (0x22090 bytes) 000490 1763342161.062052 mapwrite 0x010eb0 thru 0x01c3fb (0xb54c bytes) 000491 1763342161.201534 mapread 0x0196bd thru 0x01c3fb (0x2d3f bytes) 000496 1763342161.204102 write 0x02727e thru 0x0371f9 (0xff7c bytes) 000497 1763342161.207524 mapwrite 0x022536 thru 0x025306 (0x2dd1 bytes) 000499 1763342161.314999 trunc from 0x0371fa to 0x02b1ac (0xc04d bytes) 000501 1763342161.358039 mapwrite 0x03abf7 thru 0x03ffff (0x5409 bytes) 000506 1763342161.467999 trunc from 0x040000 to 0x03386c (0xc793 bytes) 000507 1763342161.511818 mapread 0x02c11c thru 0x032c75 (0x6b5a bytes) 000508 1763342161.580364 mapread 0x0027f2 thru 0x003156 (0x965 bytes) 000509 1763342161.585132 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3386d): Mirror 7: 0x87511b8d (stale) Mirror 8: 0x87511b8d (stale) Mirror 9: 0x87511b8d (stale) Mirror 10: 0x87511b8d (stale) Mirror 11: 0x87511b8d (stale) Mirror 12: 0x87511b8d (stale) Mirror 13: 0x87511b8d (stale) Mirror 14: 0x9332627d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3386d): Mirror 7: 0x87511b8d (stale) Mirror 8: 0x87511b8d (stale) Mirror 9: 0x87511b8d (stale) Mirror 10: 0x87511b8d (stale) Mirror 11: 0x87511b8d (stale) Mirror 12: 0x87511b8d (stale) Mirror 13: 0x87511b8d (stale) Mirror 14: 0x9332627d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3386d: skipped 000510 1763342162.175234 read 0x02b98d thru 0x03386c (0x7ee0 bytes) 000511 1763342162.187928 trunc from 0x03386d to 0x01fe00 (0x13a6c bytes) 000512 1763342162.214505 write 0x00cf0b thru 0x018b62 (0xbc58 bytes) 000514 1763342162.227836 write 0x01a8a4 thru 0x022205 (0x7962 bytes) 000515 1763342162.239879 write 0x00042f thru 0x001115 (0xce7 bytes) 000516 1763342162.256120 mapread 0x01c8bd thru 0x01c986 (0xca bytes) 000517 1763342162.256924 trunc from 0x022206 to 0x025f55 (0x3d50 bytes) 000519 1763342162.300455 MIRROR_RESYNC 9 mirrors 000520 1763342163.807555 mapwrite 0x007df8 thru 0x0084fb (0x704 bytes) 000521 1763342164.030128 write 0x02647d thru 0x02e163 (0x7ce7 bytes) 000522 1763342164.031310 mapwrite 0x0123cc thru 0x01e021 (0xbc56 bytes) 000524 1763342164.101557 mapread 0x00df30 thru 0x014407 (0x64d8 bytes) 000526 1763342164.102942 read 0x00a30e thru 0x00bcb5 (0x19a8 bytes) 000528 1763342164.103941 write 0x009dea thru 0x011408 (0x761f bytes) 000531 1763342164.118810 mapwrite 0x01422f thru 0x01c23f (0x8011 bytes) 000532 1763342164.216553 write 0x038684 thru 0x03ffff (0x797c bytes) 000533 1763342164.222859 MIRROR_RESYNC 9 mirrors 000537 1763342165.519720 trunc from 0x040000 to 0x02452e (0x1bad1 bytes) 000539 1763342165.615885 read 0x01b314 thru 0x02452e (0x921b bytes) 000541 1763342165.630954 write 0x001876 thru 0x0094be (0x7c49 bytes) 000542 1763342165.648976 trunc from 0x02452f to 0x000167 (0x243c7 bytes) 000545 1763342165.691696 MIRROR_RESYNC 9 mirrors 000546 1763342166.931633 mapwrite 0x00d5c8 thru 0x016c95 (0x96ce bytes) 000547 1763342167.094919 read 0x00670b thru 0x014be0 (0xe4d6 bytes) 000548 1763342167.111667 write 0x015e57 thru 0x021bbe (0xbd68 bytes) 000551 1763342167.113960 read 0x017189 thru 0x021bbe (0xaa36 bytes) 000552 1763342167.114932 trunc from 0x021bbf to 0x00b8de (0x162e0 bytes) 000557 1763342167.168278 read 0x00406e thru 0x00b8de (0x7871 bytes) 000558 1763342167.193272 mapwrite 0x037e4e thru 0x03ffff (0x81b2 bytes) 000559 1763342167.322828 trunc from 0x040000 to 0x02ec04 (0x113fb bytes) 000560 1763342167.352859 mapread 0x015f06 thru 0x024267 (0xe362 bytes) 000562 1763342167.364833 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ec05): Mirror 8: 0xbf98b21a (stale) Mirror 9: 0xbf98b21a (stale) Mirror 10: 0xbf98b21a (stale) Mirror 11: 0xbf98b21a (stale) Mirror 12: 0xbf98b21a (stale) Mirror 13: 0xbf98b21a (stale) Mirror 14: 0xbf98b21a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec05): Mirror 8: 0xbf98b21a (stale) Mirror 9: 0xbf98b21a (stale) Mirror 10: 0xbf98b21a (stale) Mirror 11: 0xbf98b21a (stale) Mirror 12: 0xbf98b21a (stale) Mirror 13: 0xbf98b21a (stale) Mirror 14: 0xbf98b21a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec05: skipped 000563 1763342168.023505 write 0x008684 thru 0x0106c2 (0x803f bytes) 000564 1763342168.034784 write 0x02542c thru 0x029841 (0x4416 bytes) 000566 1763342168.044751 mapread 0x01dc1c thru 0x023d52 (0x6137 bytes) 000567 1763342168.069815 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ec05): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec05): Mirror 8: 0xbf98b21a (stale) Mirror 9: 0xbf98b21a (stale) Mirror 10: 0xbf98b21a (stale) Mirror 11: 0xbf98b21a (stale) Mirror 12: 0xbf98b21a (stale) Mirror 13: 0xbf98b21a (stale) Mirror 14: 0xbf98b21a (stale) CRC-32 checksum value for chunk [0, 0x2ec05): Mirror 1: 0xc24ffaaf Mirror 15: 0xc24ffaaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec05: skipped 000568 1763342168.892949 trunc from 0x02ec05 to 0x01b73a (0x134ca bytes) 000570 1763342169.044822 trunc from 0x01b73b to 0x03c898 (0x2115e bytes) 000573 1763342169.074202 mapread 0x000838 thru 0x00ed58 (0xe521 bytes) 000574 1763342169.103728 trunc from 0x03c899 to 0x03a2a9 (0x25ef bytes) 000577 1763342169.126370 mapwrite 0x014d96 thru 0x02013f (0xb3aa bytes) 000578 1763342169.221447 mapread 0x00e122 thru 0x018917 (0xa7f6 bytes) 000581 1763342169.230244 read 0x0223a6 thru 0x0242fb (0x1f56 bytes) 000584 1763342169.233345 write 0x0340de thru 0x03ffff (0xbf22 bytes) 000585 1763342169.257377 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0x74d5b405 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf88d4645 Mirror 16: 0xf88d4645 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1763342170.113426 mapread 0x0069e3 thru 0x0088a8 (0x1ec6 bytes) 000590 1763342170.144191 trunc from 0x040000 to 0x035f13 (0xa0ec bytes) 000591 1763342170.282402 mapwrite 0x02324f thru 0x02d488 (0xa23a bytes) 000594 1763342170.407542 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x35f14): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f14): Mirror 8: 0x35b19e8e (stale) Mirror 9: 0x35b19e8e (stale) Mirror 10: 0x35b19e8e (stale) Mirror 11: 0x35b19e8e (stale) Mirror 12: 0x35b19e8e (stale) Mirror 13: 0x35b19e8e (stale) Mirror 14: 0x35b19e8e (stale) Mirror 15: 0x5395e638 (stale) Mirror 16: 0x8e96b3f5 (stale) CRC-32 checksum value for chunk [0, 0x35f14): Mirror 1: 0x9d14c3b Mirror 17: 0x9d14c3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f14: skipped 000595 1763342171.313006 mapread 0x0157d4 thru 0x017522 (0x1d4f bytes) 000596 1763342171.348700 MIRROR_RESYNC 11 mirrors 000599 1763342172.893766 MIRROR_EXTEND to 12 mirrors 000600 1763342173.756431 trunc from 0x035f14 to 0x02d75e (0x87b5 bytes) 000601 1763342173.849810 mapwrite 0x022e39 thru 0x029831 (0x69f9 bytes) 000603 1763342173.953525 trunc from 0x02d75f to 0x018dd5 (0x14989 bytes) 000605 1763342173.998528 mapwrite 0x02ec09 thru 0x03685f (0x7c57 bytes) 000607 1763342174.096366 trunc from 0x036860 to 0x021ca4 (0x14bbb bytes) 000608 1763342174.124633 mapwrite 0x01f3df thru 0x02443d (0x505f bytes) 000610 1763342174.207862 trunc from 0x02443e to 0x00f3a0 (0x1509d bytes) 000611 1763342174.238497 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf3a1): Mirror 8: 0x260a9fd8 (stale) Mirror 9: 0x260a9fd8 (stale) Mirror 10: 0x260a9fd8 (stale) Mirror 11: 0x260a9fd8 (stale) Mirror 12: 0x260a9fd8 (stale) Mirror 13: 0x260a9fd8 (stale) Mirror 14: 0x260a9fd8 (stale) Mirror 15: 0x260a9fd8 (stale) Mirror 17: 0x260a9fd8 (stale) Mirror 18: 0x260a9fd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf3a1): Mirror 8: 0x260a9fd8 (stale) Mirror 9: 0x260a9fd8 (stale) Mirror 10: 0x260a9fd8 (stale) Mirror 11: 0x260a9fd8 (stale) Mirror 12: 0x260a9fd8 (stale) Mirror 13: 0x260a9fd8 (stale) Mirror 14: 0x260a9fd8 (stale) Mirror 15: 0x260a9fd8 (stale) Mirror 17: 0x260a9fd8 (stale) Mirror 18: 0x260a9fd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf3a1: skipped 000613 1763342174.779857 write 0x02bd64 thru 0x036e5c (0xb0f9 bytes) 000614 1763342174.781409 write 0x021184 thru 0x02ca0e (0xb88b bytes) 000615 1763342174.790001 mapread 0x028bac thru 0x029c5e (0x10b3 bytes) 000616 1763342174.796408 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36e5d): Mirror 8: 0x2ac892e (stale) Mirror 9: 0x2ac892e (stale) Mirror 10: 0x2ac892e (stale) Mirror 11: 0x2ac892e (stale) Mirror 13: 0x2ac892e (stale) Mirror 14: 0x2ac892e (stale) Mirror 15: 0x2ac892e (stale) Mirror 17: 0x2ac892e (stale) Mirror 18: 0x2ac892e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e5d): Mirror 8: 0x2ac892e (stale) Mirror 9: 0x2ac892e (stale) Mirror 10: 0x2ac892e (stale) Mirror 11: 0x2ac892e (stale) Mirror 13: 0x2ac892e (stale) Mirror 14: 0x2ac892e (stale) Mirror 15: 0x2ac892e (stale) Mirror 17: 0x2ac892e (stale) Mirror 18: 0x2ac892e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e5d: skipped 000617 1763342175.449520 mapwrite 0x02cd0f thru 0x02f9c9 (0x2cbb bytes) 000618 1763342175.554590 MIRROR_RESYNC 10 mirrors 000619 1763342176.963324 write 0x0151cc thru 0x0186fd (0x3532 bytes) 000624 1763342177.049166 read 0x0347e8 thru 0x03514f (0x968 bytes) 000625 1763342177.059926 trunc from 0x036e5d to 0x03ee2c (0x7fd0 bytes) 000628 1763342177.086580 write 0x001038 thru 0x00cce6 (0xbcaf bytes) 000629 1763342177.095251 mapwrite 0x01c7a2 thru 0x027c43 (0xb4a2 bytes) 000631 1763342177.166244 mapread 0x037686 thru 0x03ee2c (0x77a7 bytes) 000632 1763342177.177224 write 0x0368a3 thru 0x03e610 (0x7d6e bytes) 000633 1763342177.178569 mapread 0x00e778 thru 0x0141cb (0x5a54 bytes) 000635 1763342177.187934 write 0x013424 thru 0x020196 (0xcd73 bytes) 000636 1763342177.189694 read 0x011811 thru 0x0180d6 (0x68c6 bytes) 000637 1763342177.195995 MIRROR_RESYNC 10 mirrors 000639 1763342178.973663 mapread 0x03b4ac thru 0x03ee2c (0x3981 bytes) 000641 1763342178.997629 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) 000642 1763342179.651429 trunc from 0x03ee2d to 0x0025d1 (0x3c85b bytes) 000643 1763342179.761407 write 0x00b7d2 thru 0x018cca (0xd4f9 bytes) 000645 1763342179.763809 trunc from 0x018ccb to 0x005ed6 (0x12df4 bytes) 000646 1763342179.803358 trunc from 0x005ed7 to 0x008b1b (0x2c45 bytes) 000647 1763342179.826998 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x8b1c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b1c): Mirror 8: 0x9e965b8f (stale) Mirror 9: 0x9e965b8f (stale) Mirror 10: 0x9e965b8f (stale) Mirror 11: 0x9e965b8f (stale) Mirror 13: 0x9e965b8f (stale) Mirror 15: 0x9e965b8f (stale) Mirror 17: 0x9e965b8f (stale) Mirror 18: 0x9e965b8f (stale) CRC-32 checksum value for chunk [0, 0x8b1c): Mirror 1: 0xf2114858 Mirror 19: 0xf2114858 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b1c: skipped 000649 1763342180.864676 trunc from 0x008b1c to 0x0319b4 (0x28e99 bytes) 000650 1763342180.962572 mapwrite 0x01424e thru 0x019406 (0x51b9 bytes) 000651 1763342181.062039 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x319b5): Mirror 9: 0xdd4088bd (stale) Mirror 10: 0xdd4088bd (stale) Mirror 11: 0xdd4088bd (stale) Mirror 13: 0xdd4088bd (stale) Mirror 15: 0xdd4088bd (stale) Mirror 17: 0xdd4088bd (stale) Mirror 18: 0xdd4088bd (stale) Mirror 19: 0xa16e38fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x319b5): Mirror 9: 0xdd4088bd (stale) Mirror 10: 0xdd4088bd (stale) Mirror 11: 0xdd4088bd (stale) Mirror 13: 0xdd4088bd (stale) Mirror 15: 0xdd4088bd (stale) Mirror 17: 0xdd4088bd (stale) Mirror 18: 0xdd4088bd (stale) Mirror 19: 0xa16e38fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x319b5: skipped 000653 1763342181.579513 trunc from 0x0319b5 to 0x0128b6 (0x1f0fe bytes) 000654 1763342181.602978 read 0x008d82 thru 0x0128b6 (0x9b35 bytes) 000655 1763342181.615065 mapwrite 0x034553 thru 0x03a44d (0x5efb bytes) 000656 1763342181.727258 trunc from 0x03a44e to 0x038fd4 (0x1479 bytes) 000657 1763342181.756771 write 0x02793b thru 0x032835 (0xaefb bytes) 000658 1763342181.768475 mapread 0x01cc1a thru 0x02bb12 (0xeef9 bytes) 000659 1763342181.777630 read 0x038e7b thru 0x038fd4 (0x15a bytes) 000662 1763342181.779275 write 0x010ccb thru 0x016b65 (0x5e9b bytes) 000663 1763342181.781432 trunc from 0x038fd5 to 0x00d2a8 (0x2bd2c bytes) 000666 1763342181.807109 MIRROR_RESYNC 9 mirrors 000672 1763342183.140953 mapwrite 0x02cad7 thru 0x034975 (0x7e9f bytes) 000673 1763342183.332649 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34976): Mirror 9: 0x646a5c98 (stale) Mirror 10: 0x646a5c98 (stale) Mirror 11: 0x646a5c98 (stale) Mirror 15: 0x646a5c98 (stale) Mirror 17: 0x646a5c98 (stale) Mirror 18: 0x646a5c98 (stale) Mirror 19: 0x646a5c98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34976): Mirror 9: 0x646a5c98 (stale) Mirror 10: 0x646a5c98 (stale) Mirror 11: 0x646a5c98 (stale) Mirror 15: 0x646a5c98 (stale) Mirror 17: 0x646a5c98 (stale) Mirror 18: 0x646a5c98 (stale) Mirror 19: 0x646a5c98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34976: skipped 000675 1763342183.830565 mapread 0x028980 thru 0x032471 (0x9af2 bytes) 000676 1763342183.858186 write 0x01bc94 thru 0x02a35d (0xe6ca bytes) 000677 1763342183.866821 mapwrite 0x027433 thru 0x027eac (0xa7a bytes) 000678 1763342183.919813 trunc from 0x034976 to 0x0061cf (0x2e7a6 bytes) 000679 1763342183.938231 trunc from 0x0061d0 to 0x018981 (0x127b2 bytes) 000682 1763342183.961180 write 0x02e701 thru 0x035b1e (0x741e bytes) 000683 1763342183.967444 MIRROR_RESYNC 8 mirrors 000686 1763342185.209024 read 0x01927f thru 0x01e4fb (0x527d bytes) 000687 1763342185.223930 mapwrite 0x02eca6 thru 0x032e73 (0x41ce bytes) 000689 1763342185.445119 mapread 0x016909 thru 0x01adf7 (0x44ef bytes) 000692 1763342185.456069 trunc from 0x035b1f to 0x03d679 (0x7b5b bytes) 000693 1763342185.488048 mapread 0x03967e thru 0x03d679 (0x3ffc bytes) 000694 1763342185.500006 trunc from 0x03d67a to 0x00219c (0x3b4dd bytes) 000695 1763342185.521141 mapwrite 0x034e2b thru 0x03ffff (0xb1d5 bytes) 000696 1763342185.597525 write 0x01862f thru 0x022091 (0x9a63 bytes) 000697 1763342185.612171 mapwrite 0x03fee1 thru 0x03ffff (0x11f bytes) 000698 1763342185.660250 read 0x03b570 thru 0x03e9b4 (0x3445 bytes) 000699 1763342185.660941 read 0x00fb71 thru 0x012669 (0x2af9 bytes) 000700 1763342185.671228 read 0x03f900 thru 0x03ffff (0x700 bytes) 000701 1763342185.672711 trunc from 0x040000 to 0x02dc4e (0x123b1 bytes) 000707 1763342185.694342 read 0x004626 thru 0x011b15 (0xd4f0 bytes) 000708 1763342185.702066 MIRROR_RESYNC 8 mirrors 000711 1763342187.296607 mapread 0x024d09 thru 0x02d309 (0x8601 bytes) 000712 1763342187.315575 write 0x0066ab thru 0x014c28 (0xe57e bytes) 000716 1763342187.409749 read 0x02b4e7 thru 0x02dc4e (0x2768 bytes) 000717 1763342187.416831 read 0x02867f thru 0x02dc4e (0x55d0 bytes) 000718 1763342187.429601 mapread 0x0023e1 thru 0x003165 (0xd85 bytes) 000719 1763342187.430552 mapread 0x02b121 thru 0x02dc4e (0x2b2e bytes) 000720 1763342187.431379 mapwrite 0x01821d thru 0x01b7a4 (0x3588 bytes) 000721 1763342187.489309 write 0x02165b thru 0x02913f (0x7ae5 bytes) 000722 1763342187.491649 write 0x034066 thru 0x03ffff (0xbf9a bytes) 000724 1763342187.493635 write 0x02d52d thru 0x031d73 (0x4847 bytes) 000725 1763342187.501805 write 0x0037ee thru 0x012e07 (0xf61a bytes) 000726 1763342187.503192 mapwrite 0x01f095 thru 0x02cac8 (0xda34 bytes) 000727 1763342187.567368 trunc from 0x040000 to 0x037543 (0x8abc bytes) 000729 1763342187.592928 trunc from 0x037544 to 0x019296 (0x1e2ad bytes) 000732 1763342187.620774 read 0x0053f0 thru 0x011508 (0xc119 bytes) 000733 1763342187.623392 write 0x013df1 thru 0x020e10 (0xd020 bytes) 000735 1763342187.626134 trunc from 0x020e11 to 0x00486f (0x1c5a1 bytes) 000736 1763342187.650169 mapread 0x00407e thru 0x00486f (0x7f2 bytes) 000737 1763342187.651187 mapwrite 0x01aa12 thru 0x0255eb (0xabda bytes) 000739 1763342187.776390 write 0x01b6de thru 0x01e898 (0x31bb bytes) 000740 1763342187.777520 trunc from 0x0255ec to 0x01e03f (0x75ac bytes) 000742 1763342187.804528 mapwrite 0x003bf4 thru 0x00a85f (0x6c6c bytes) 000743 1763342187.866128 read 0x00e0ca thru 0x01b803 (0xd73a bytes) 000745 1763342187.872082 mapread 0x0094f6 thru 0x00cb83 (0x368e bytes) 000748 1763342187.883566 MIRROR_RESYNC 8 mirrors 000749 1763342189.461774 read 0x00cd2f thru 0x010dd6 (0x40a8 bytes) 000750 1763342189.471268 write 0x0265fe thru 0x033819 (0xd21c bytes) 000754 1763342189.553071 mapwrite 0x00a26b thru 0x00be0d (0x1ba3 bytes) 000757 1763342189.620400 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3381a): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3381a): Mirror 9: 0x74e1114d (stale) Mirror 10: 0x74e1114d (stale) Mirror 11: 0x74e1114d (stale) Mirror 15: 0x74e1114d (stale) Mirror 17: 0x74e1114d (stale) Mirror 18: 0x74e1114d (stale) Mirror 19: 0x74e1114d (stale) CRC-32 checksum value for chunk [0, 0x3381a): Mirror 1: 0xdcde41b7 Mirror 20: 0xdcde41b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3381a: skipped 000759 1763342190.810856 write 0x01c79f thru 0x01edf7 (0x2659 bytes) 000760 1763342190.943703 mapread 0x00f3e9 thru 0x0136bb (0x42d3 bytes) 000762 1763342190.974982 MIRROR_RESYNC 9 mirrors 000764 1763342192.631176 mapwrite 0x03a5a9 thru 0x03ffff (0x5a57 bytes) 000765 1763342192.859809 read 0x02b79a thru 0x034d03 (0x956a bytes) 000766 1763342192.868174 trunc from 0x040000 to 0x0212c4 (0x1ed3b bytes) 000768 1763342192.907911 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x212c5): Mirror 9: 0x632c4206 (stale) Mirror 10: 0x632c4206 (stale) Mirror 11: 0x632c4206 (stale) Mirror 15: 0x632c4206 (stale) Mirror 18: 0x632c4206 (stale) Mirror 19: 0x632c4206 (stale) Mirror 20: 0x632c4206 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x212c5): Mirror 9: 0x632c4206 (stale) Mirror 10: 0x632c4206 (stale) Mirror 11: 0x632c4206 (stale) Mirror 15: 0x632c4206 (stale) Mirror 18: 0x632c4206 (stale) Mirror 19: 0x632c4206 (stale) Mirror 20: 0x632c4206 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x212c5: skipped 000770 1763342193.466437 mapread 0x00c6dc thru 0x012ca7 (0x65cc bytes) 000771 1763342193.488728 mapwrite 0x01f12b thru 0x02be2d (0xcd03 bytes) 000772 1763342193.598517 mapread 0x01a085 thru 0x01b906 (0x1882 bytes) 000773 1763342193.608754 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 9: 0x6c1539fb (stale) Mirror 10: 0x6c1539fb (stale) Mirror 11: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 9: 0x6c1539fb (stale) Mirror 10: 0x6c1539fb (stale) Mirror 11: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be2e: skipped 000776 1763342194.157255 read 0x0225d3 thru 0x025079 (0x2aa7 bytes) 000778 1763342194.170917 read 0x01d1c3 thru 0x0226c7 (0x5505 bytes) 000779 1763342194.193609 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 10: 0x6c1539fb (stale) Mirror 11: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 10: 0x6c1539fb (stale) Mirror 11: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be2e: skipped 000780 1763342194.634913 mapread 0x011e80 thru 0x016ca1 (0x4e22 bytes) 000782 1763342194.657525 read 0x019a44 thru 0x019f46 (0x503 bytes) 000784 1763342194.662866 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 10: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be2e): Mirror 10: 0x6c1539fb (stale) Mirror 15: 0x6c1539fb (stale) Mirror 18: 0x6c1539fb (stale) Mirror 19: 0x6c1539fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be2e: skipped 000786 1763342195.071190 mapwrite 0x02bebc thru 0x032f9f (0x70e4 bytes) 000791 1763342195.251572 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32fa0): Mirror 10: 0xe6e6154d (stale) Mirror 15: 0xe6e6154d (stale) Mirror 19: 0xe6e6154d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fa0): Mirror 10: 0xe6e6154d (stale) Mirror 15: 0xe6e6154d (stale) Mirror 19: 0xe6e6154d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fa0: skipped 000794 1763342195.718095 trunc from 0x032fa0 to 0x0264db (0xcac4 bytes) 000795 1763342195.749997 write 0x038dfa thru 0x03ffff (0x7206 bytes) 000796 1763342195.756494 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa272be1a (stale) Mirror 15: 0xa272be1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa272be1a (stale) Mirror 15: 0xa272be1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000797 1763342196.129201 trunc from 0x040000 to 0x00c9b8 (0x33647 bytes) 000798 1763342196.152688 mapread 0x001ff0 thru 0x004c11 (0x2c22 bytes) 000799 1763342196.187005 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc9b9): Mirror 15: 0x1d35c7a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9b9): Mirror 15: 0x1d35c7a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9b9: skipped 000800 1763342196.559173 write 0x006d8d thru 0x015392 (0xe606 bytes) 000801 1763342196.566865 write 0x021001 thru 0x022f1e (0x1f1e bytes) 000803 1763342196.568365 read 0x017d8d thru 0x022f1e (0xb192 bytes) 000804 1763342196.583772 mapwrite 0x0246a8 thru 0x033f0b (0xf864 bytes) 000805 1763342196.791777 write 0x031ac1 thru 0x03ffff (0xe53f bytes) 000806 1763342196.796941 trunc from 0x040000 to 0x00d341 (0x32cbe bytes) 000807 1763342196.849275 mapwrite 0x037635 thru 0x039af0 (0x24bc bytes) 000808 1763342196.997481 mapwrite 0x01e4a9 thru 0x02b9ca (0xd522 bytes) 000810 1763342197.125819 write 0x0201fa thru 0x029229 (0x9030 bytes) 000813 1763342197.127559 trunc from 0x039af1 to 0x032813 (0x72dd bytes) 000814 1763342197.159893 mapwrite 0x01c362 thru 0x0254b6 (0x9155 bytes) 000815 1763342197.249498 write 0x0359bd thru 0x0386db (0x2d1f bytes) 000816 1763342197.250533 mapread 0x0273d8 thru 0x034478 (0xd0a1 bytes) 000817 1763342197.269162 MIRROR_RESYNC 2 mirrors 000818 1763342198.063218 mapread 0x004ddc thru 0x008bbc (0x3de1 bytes) 000820 1763342198.087105 trunc from 0x0386dc to 0x006552 (0x32189 bytes) 000822 1763342198.237337 trunc from 0x006553 to 0x00020c (0x6346 bytes) 000824 1763342198.291319 trunc from 0x00020d to 0x032ff8 (0x32dec bytes) 000825 1763342198.369964 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32ff9): Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ff9): Mirror 15: 0x19adb422 (stale) CRC-32 checksum value for chunk [0, 0x32ff9): Mirror 1: 0xfb6c936c Mirror 16: 0xfb6c936c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ff9: skipped 000826 1763342199.142475 write 0x033c70 thru 0x03a2a8 (0x6639 bytes) 000828 1763342199.233723 read 0x00188e thru 0x002912 (0x1085 bytes) 000830 1763342199.240245 trunc from 0x03a2a9 to 0x0138ac (0x269fc bytes) 000831 1763342199.290676 read 0x0016ea thru 0x008136 (0x6a4d bytes) 000832 1763342199.310594 mapread 0x007a0d thru 0x010cd3 (0x92c7 bytes) 000833 1763342199.311546 trunc from 0x0138ad to 0x03d8e9 (0x2a03d bytes) 000834 1763342199.346936 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d8ea): Mirror 15: 0x28522d90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8ea): Mirror 15: 0x28522d90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8ea: skipped 000836 1763342199.649059 trunc from 0x03d8ea to 0x013052 (0x2a897 bytes) 000837 1763342199.680376 read 0x00c9d7 thru 0x013052 (0x667c bytes) 000839 1763342199.707659 mapwrite 0x02c099 thru 0x0353af (0x9317 bytes) 000840 1763342199.866849 write 0x0371e4 thru 0x03ffff (0x8e1c bytes) 000841 1763342199.868176 trunc from 0x040000 to 0x012f69 (0x2d096 bytes) 000842 1763342199.913106 mapread 0x00bd12 thru 0x00f994 (0x3c83 bytes) 000843 1763342199.932542 mapwrite 0x00c510 thru 0x013df1 (0x78e2 bytes) 000844 1763342200.017903 mapread 0x00792e thru 0x013df1 (0xc4c4 bytes) 000845 1763342200.019138 read 0x0031bf thru 0x010dd7 (0xdc19 bytes) 000846 1763342200.025765 write 0x03f983 thru 0x03ffff (0x67d bytes) 000847 1763342200.031427 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x653d2ce5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x227f281 Mirror 16: 0x227f281 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1763342200.699945 write 0x02c950 thru 0x03a942 (0xdff3 bytes) 000851 1763342200.848570 mapwrite 0x030a62 thru 0x03b970 (0xaf0f bytes) 000852 1763342200.934928 read 0x011005 thru 0x016099 (0x5095 bytes) 000853 1763342200.944661 read 0x007cdb thru 0x011f31 (0xa257 bytes) 000856 1763342200.956641 mapwrite 0x005f83 thru 0x00e9cc (0x8a4a bytes) 000857 1763342201.014720 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x227f281 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x227f281 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000858 1763342201.348882 mapwrite 0x02b249 thru 0x03a528 (0xf2e0 bytes) 000859 1763342201.467793 mapwrite 0x036920 thru 0x03e915 (0x7ff6 bytes) 000860 1763342201.561452 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x227f281 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6edc655e Mirror 17: 0x6edc655e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000861 1763342202.806016 write 0x030aed thru 0x037813 (0x6d27 bytes) 000862 1763342202.912736 trunc from 0x040000 to 0x001687 (0x3e978 bytes) 000864 1763342202.960929 mapread 0x001392 thru 0x001687 (0x2f6 bytes) 000865 1763342202.969640 trunc from 0x001688 to 0x03a8a7 (0x39220 bytes) 000866 1763342203.025941 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8a8): Mirror 17: 0x32b35dd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8a8): Mirror 17: 0x32b35dd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8a8: skipped 000867 1763342203.430328 read 0x001eec thru 0x007f3f (0x6054 bytes) 000873 1763342203.444842 trunc from 0x03a8a8 to 0x0391de (0x16c9 bytes) 000874 1763342203.465867 trunc from 0x0391df to 0x008fc6 (0x30218 bytes) 000879 1763342203.497733 mapwrite 0x0091fc thru 0x00adc7 (0x1bcc bytes) 000880 1763342203.598950 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xadc8): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xadc8): Mirror 17: 0xe29482d (stale) CRC-32 checksum value for chunk [0, 0xadc8): Mirror 1: 0x2e31f19b Mirror 18: 0x2e31f19b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xadc8: skipped 000881 1763342204.605651 read 0x0094fd thru 0x00adc7 (0x18cb bytes) 000882 1763342204.643632 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xadc8): Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xadc8): Mirror 17: 0xe29482d (stale) CRC-32 checksum value for chunk [0, 0xadc8): Mirror 1: 0x2e31f19b Mirror 18: 0x2e31f19b Mirror 19: 0x2e31f19b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xadc8: skipped 000883 1763342205.518734 write 0x01a628 thru 0x01c782 (0x215b bytes) 000885 1763342205.603741 mapread 0x0059ca thru 0x0134b4 (0xdaeb bytes) 000888 1763342205.649091 mapread 0x000d10 thru 0x0047e4 (0x3ad5 bytes) 000889 1763342205.660105 write 0x020eeb thru 0x028a32 (0x7b48 bytes) 000892 1763342205.662409 mapread 0x00e8e2 thru 0x01719b (0x88ba bytes) 000893 1763342205.665334 mapwrite 0x022d6e thru 0x028196 (0x5429 bytes) 000894 1763342205.727076 trunc from 0x028a33 to 0x032813 (0x9de1 bytes) 000896 1763342205.798279 trunc from 0x032814 to 0x03a89e (0x808b bytes) 000897 1763342205.838129 MIRROR_RESYNC 4 mirrors 000898 1763342206.716724 mapread 0x00d0d4 thru 0x012958 (0x5885 bytes) 000901 1763342206.760370 mapread 0x028a64 thru 0x03745f (0xe9fc bytes) 000902 1763342206.809572 MIRROR_EXTEND to 5 mirrors 000904 1763342207.492928 write 0x010e5c thru 0x01fee4 (0xf089 bytes) 000906 1763342207.584636 write 0x015748 thru 0x0238de (0xe197 bytes) 000907 1763342207.591409 mapread 0x02cfcf thru 0x02e234 (0x1266 bytes) 000908 1763342207.600634 trunc from 0x03a89f to 0x014790 (0x2610e bytes) 000909 1763342207.625804 mapread 0x006d3c thru 0x009368 (0x262d bytes) 000911 1763342207.637910 read 0x007040 thru 0x0101d6 (0x9197 bytes) 000912 1763342207.639206 mapread 0x0088ff thru 0x012683 (0x9d85 bytes) 000913 1763342207.640218 read 0x006734 thru 0x014790 (0xe05d bytes) 000914 1763342207.641669 trunc from 0x014791 to 0x02b450 (0x16cc0 bytes) 000915 1763342207.668035 write 0x03b873 thru 0x03d240 (0x19ce bytes) 000918 1763342207.670765 mapwrite 0x03c714 thru 0x03ffff (0x38ec bytes) 000919 1763342207.766143 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xda3568fe (stale) Mirror 18: 0xda3568fe (stale) Mirror 19: 0xda3568fe (stale) Mirror 20: 0xda3568fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2826962b Mirror 21: 0x2826962b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000920 1763342208.754638 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xda3568fe (stale) Mirror 18: 0xda3568fe (stale) Mirror 19: 0xda3568fe (stale) Mirror 20: 0xda3568fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2826962b Mirror 21: 0x2826962b Mirror 22: 0x2826962b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000921 1763342209.634535 mapread 0x025fe5 thru 0x02de12 (0x7e2e bytes) 000922 1763342209.669142 write 0x000982 thru 0x009d6a (0x93e9 bytes) 000923 1763342209.778492 mapwrite 0x00bcd9 thru 0x0108b3 (0x4bdb bytes) 000924 1763342209.859147 mapread 0x0086ca thru 0x011046 (0x897d bytes) 000925 1763342209.860731 write 0x0175e5 thru 0x02179e (0xa1ba bytes) 000926 1763342209.876116 trunc from 0x040000 to 0x024e9a (0x1b165 bytes) 000927 1763342209.911421 trunc from 0x024e9b to 0x032f05 (0xe06b bytes) 000930 1763342209.944160 mapwrite 0x031f83 thru 0x03505c (0x30da bytes) 000931 1763342210.073273 MIRROR_RESYNC 7 mirrors 000932 1763342211.399279 trunc from 0x03505d to 0x02704b (0xe011 bytes) 000933 1763342211.550221 mapread 0x0108a3 thru 0x01e30a (0xda68 bytes) 000934 1763342211.590773 write 0x0212e3 thru 0x0255a9 (0x42c7 bytes) 000935 1763342211.600538 mapread 0x00c4b9 thru 0x010a32 (0x457a bytes) 000938 1763342211.610352 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2704c): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2704c): Mirror 17: 0xc525fef0 (stale) Mirror 18: 0xc525fef0 (stale) Mirror 19: 0xc525fef0 (stale) Mirror 20: 0xc525fef0 (stale) Mirror 21: 0xc525fef0 (stale) Mirror 22: 0xc525fef0 (stale) CRC-32 checksum value for chunk [0, 0x2704c): Mirror 1: 0x5b287f9f Mirror 23: 0x5b287f9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2704c: skipped 000941 1763342212.496946 trunc from 0x02704c to 0x0022fc (0x24d4f bytes) 000944 1763342212.614220 trunc from 0x0022fd to 0x009a55 (0x7759 bytes) 000945 1763342212.651564 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x9a56): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a56): Mirror 17: 0x4fc1d035 (stale) Mirror 18: 0x4fc1d035 (stale) Mirror 19: 0x4fc1d035 (stale) Mirror 20: 0x4fc1d035 (stale) Mirror 21: 0x4fc1d035 (stale) Mirror 22: 0x4fc1d035 (stale) Mirror 23: 0x4fc1d035 (stale) CRC-32 checksum value for chunk [0, 0x9a56): Mirror 1: 0xb25bafe4 Mirror 24: 0xb25bafe4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a56: skipped 000947 1763342213.591912 trunc from 0x009a56 to 0x03095c (0x26f07 bytes) 000948 1763342213.697340 mapwrite 0x010dfb thru 0x01d378 (0xc57e bytes) 000949 1763342213.778791 write 0x017c30 thru 0x018e5b (0x122c bytes) 000950 1763342213.779784 write 0x03870a thru 0x03f236 (0x6b2d bytes) 000951 1763342213.780956 write 0x031254 thru 0x03a524 (0x92d1 bytes) 000952 1763342213.790852 write 0x018b4e thru 0x01c66f (0x3b22 bytes) 000953 1763342213.792923 mapwrite 0x02f06d thru 0x03bcd8 (0xcc6c bytes) 000954 1763342213.851347 write 0x01f515 thru 0x02039d (0xe89 bytes) 000955 1763342213.869075 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f237): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f237): Mirror 17: 0x99808c65 (stale) Mirror 18: 0x99808c65 (stale) Mirror 19: 0x99808c65 (stale) Mirror 20: 0x99808c65 (stale) Mirror 21: 0x99808c65 (stale) Mirror 22: 0x99808c65 (stale) Mirror 23: 0xce2f037d (stale) Mirror 24: 0x6764c7d5 (stale) CRC-32 checksum value for chunk [0, 0x3f237): Mirror 1: 0x6b43b084 Mirror 25: 0x6b43b084 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f237: skipped 000956 1763342214.863992 write 0x02cb06 thru 0x033c48 (0x7143 bytes) 000957 1763342214.956566 mapread 0x030fae thru 0x03791f (0x6972 bytes) 000961 1763342214.966760 trunc from 0x03f237 to 0x024ca9 (0x1a58d bytes) 000964 1763342214.993947 mapread 0x013536 thru 0x023335 (0xfe00 bytes) 000969 1763342215.014157 mapread 0x01f349 thru 0x024ca9 (0x5961 bytes) 000970 1763342215.015171 write 0x009dac thru 0x00a5cd (0x822 bytes) 000971 1763342215.035047 MIRROR_RESYNC 10 mirrors 000972 1763342216.642206 write 0x03f4f7 thru 0x03ffff (0xb09 bytes) 000973 1763342216.697610 mapwrite 0x0331de thru 0x03ffff (0xce22 bytes) 000974 1763342216.800756 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xf3764b6 (stale) Mirror 18: 0xf3764b6 (stale) Mirror 19: 0xf3764b6 (stale) Mirror 20: 0xf3764b6 (stale) Mirror 21: 0xf3764b6 (stale) Mirror 22: 0xf3764b6 (stale) Mirror 23: 0xf3764b6 (stale) Mirror 24: 0xf3764b6 (stale) Mirror 25: 0xf3764b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x89195951 Mirror 26: 0x89195951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000975 1763342217.955613 MIRROR_RESYNC 11 mirrors 000976 1763342219.487955 trunc from 0x040000 to 0x02fab9 (0x10546 bytes) 000980 1763342219.609976 mapread 0x02116e thru 0x0246d6 (0x3569 bytes) 000983 1763342219.639972 mapread 0x02f6ca thru 0x02fab9 (0x3f0 bytes) 000984 1763342219.646902 trunc from 0x02faba to 0x02c14e (0x396b bytes) 000985 1763342219.708552 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2c14f): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c14f): Mirror 17: 0x94833628 (stale) Mirror 18: 0x94833628 (stale) Mirror 19: 0x94833628 (stale) Mirror 20: 0x94833628 (stale) Mirror 21: 0x94833628 (stale) Mirror 22: 0x94833628 (stale) Mirror 23: 0x94833628 (stale) Mirror 24: 0x94833628 (stale) Mirror 25: 0x94833628 (stale) Mirror 26: 0x94833628 (stale) CRC-32 checksum value for chunk [0, 0x2c14f): Mirror 1: 0x94833628 Mirror 27: 0x94833628 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c14f: skipped 000986 1763342220.832510 trunc from 0x02c14f to 0x015c62 (0x164ec bytes) 000988 1763342220.919701 MIRROR_RESYNC 12 mirrors 000989 1763342222.531788 mapread 0x012ca4 thru 0x014c44 (0x1fa1 bytes) 000990 1763342222.565432 read 0x010dd8 thru 0x013437 (0x2660 bytes) 000991 1763342222.570617 write 0x0016a7 thru 0x00f790 (0xe0ea bytes) 000993 1763342222.679885 read 0x0107bf thru 0x015c62 (0x54a4 bytes) 000995 1763342222.708808 MIRROR_RESYNC 12 mirrors 000996 1763342224.419441 mapwrite 0x00a86c thru 0x013534 (0x8cc9 bytes) 000997 1763342224.638240 write 0x032479 thru 0x036041 (0x3bc9 bytes) 000999 1763342224.639791 mapread 0x035ee0 thru 0x036041 (0x162 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 20:17:09 (1763342229) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 20:17:13 (1763342233) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6657344 .. Extending file size to 4758848 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8273856 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6991424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8500768 .. Extending file size to 7147968 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7731264 .. Extending file size to 8639808 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8985472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9030432 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 54859 54860 54861 54862 54864 done 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) 192.168.204.126@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:18:32 (1763342312) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7710016 .. Extending file size to 5479904 .. done Extending file size to 8006240 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7435552 .. Extending file size to 8403008 .. Extending file size to 8411104 .. Extending file size to 8705792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8541760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9340000 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8725920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56557 56558 56559 56564 56565 failed 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) 192.168.204.126@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:19:53 (1763342393) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123813 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.380134 s, 5.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x97ab54e0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:20:13 (1763342413) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141266 s, 14.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 20:20:20 (1763342420) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x26d:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:20:26 (1763342426) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191938 s, 10.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:20:34 (1763342434) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30445 s, 6.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x271:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:20:42 (1763342442) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.332361 s, 6.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x273:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:21:13 (1763342473) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143011 s, 7.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x278:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x278:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x274:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:22:08 (1763342528) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194153 s, 10.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:23:08 (1763342588) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg426-server mds-ost sync done. Creating new pool test_204f oleg426-server: Pool lustre.test_204f created Adding targets to pool oleg426-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg426-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344688 s, 6.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg426-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg426-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg426-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:24:24 (1763342664) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:24:31 (1763342671) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:24:38 (1763342678) oleg426-server: Pool lustre.test_206 created Adding targets to pool oleg426-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg426-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg426-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg426-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg426-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:25:03 (1763342703) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367706 s, 28.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036615 s, 28.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2711dafc Mirror 3: 0x2711dafc PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:25:13 (1763342713) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:25:16 (1763342716) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:25:19 (1763342719) Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.129553 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2110000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1df0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1df0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1df0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9a0000 Read this (4 bytes): BBBB 192.168.204.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg426-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:25:51 (1763342751) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0295967 s, 277 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:25:58 (1763342758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585976 s, 17.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20b lmm_fid: [0x200000404:0x20b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20b lmm_fid: [0x200000404:0x20b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834443 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 20:26:00.000000000 -0500 Modify: 2025-11-16 20:26:00.000000000 -0500 Change: 2025-11-16 20:26:01.000000000 -0500 Birth: 2025-11-16 20:25:59.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:26:07 (1763342767) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:26:09 (1763342769) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.472228 s, 86.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0511465 s, 80.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 20:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 20:26 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:26:17 (1763342777) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:26:24 (1763342784) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1629 sec =========== 20:26:31 (1763342791) === sanity-flr: start cleanup 20:26:33 (1763342793) === === sanity-flr: finish cleanup 20:26:39 (1763342799) ===