-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 21:15:15 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg205-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:15:34 (1763345734) === oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cf94308d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cf94308d800.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:16:03 (1763345763) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7574] == sanity-flr test 0a: lfs mirror create with -N option == 21:16:05 (1763345765) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:16:20 (1763345780) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:16:23 (1763345783) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:16:26 (1763345786) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:16:52 (1763345812) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:16:55 (1763345815) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:16:59 (1763345819) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cf94308d800.stats=clear osc.lustre-OST0001-osc-ffff8cf94308d800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:17:07 (1763345827) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:17:18 (1763345838) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:17:27 (1763345847) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:17:30 (1763345850) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 21:17:39 (1763345859) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:17:45 (1763345865) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:17:49 (1763345869) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:17:56 (1763345876) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:18:07 (1763345887) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:18:13 (1763345893) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.514654 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132611 s, 7.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:18:23 (1763345903) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.4408 s, 38.1 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:18:31 (1763345911) Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:19:40 (1763345980) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.126669 s, 24.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:20:16 (1763346016) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (63s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:21:18 (1763346078) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.40335 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 21:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.771s user 0m0.000s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.952s user 0m0.000s sys 0m0.167s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.999s user 0m0.000s sys 0m0.201s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cf94308d800.ost_server_uuid in FULL state after 0 sec PASS 33b (75s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:22:33 (1763346153) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:22:36 (1763346156) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:22:39 (1763346159) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:22:42 (1763346162) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 21:22:51 (1763346171) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.607083 s, 20.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:23:05 (1763346185) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032659 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397723 s, 26.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:23:13 (1763346193) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:23:21 (1763346201) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00178644 s, 573 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000911084 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000682818 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00280432 s, 365 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 21:23:30 (1763346210) create test files with size 15759434 .. 431ca92fabd89ab519bd2b448ebcd80d - 0f2df30dc7d7912461d15ec42dd2abcc - 08a4324cb9ce7e35a88b14ccc30ccbad - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 21:23:51 (1763346231) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:24:02 (1763346242) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:24:07 (1763346247) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130422 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227654 s, 46.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 21:24:15 (1763346255) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111072 s, 18.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.379787 s, 11.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204207 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 21:24:30 (1763346270) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:24:32 (1763346272) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:24:35 (1763346275) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369416 s, 28.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 21:24:41 (1763346281) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:24:44 (1763346284) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:24:52 (1763346292) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.186228 s, 56.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:25:01 (1763346301) llite.lustre-ffff8cf9435e7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.199985 s, 52.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8cf944d93000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:25:11 (1763346311) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:25:52 (1763346352) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:26:28 (1763346388) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:26:31 (1763346391) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.236569 s, 13.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121905 s, 25.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbc33f21b (stale) Mirror 2: 0xbc33f21b (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 6acd15d16b07066ccd910b60ef896ddb - mirror 1 checksum: 2a162ca09b2e996803650a8ae4177186 - mirror 2 checksum: 6acd15d16b07066ccd910b60ef896ddb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 21:26:34 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 21:26:34 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6acd15d16b07066ccd910b60ef896ddb - mirror 1 checksum: 2a162ca09b2e996803650a8ae4177186 - mirror 2 checksum: 2a162ca09b2e996803650a8ae4177186 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 21:26:34 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 21:26:34 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:26:43 (1763346403) Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server 21:26:48 (1763346408) shut down facet: mds1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover mds1 to oleg205-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 21:27:08 (1763346428) targets are mounted 21:27:08 (1763346428) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391735 s, 26.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:27:21 (1763346441) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:27:24 (1763346444) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:27:28 (1763346448) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.013037 s, 1.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:27:36 (1763346456) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:27:38 (1763346458) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:27:41 (1763346461) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:28:02 (1763346482) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:28:30 (1763346510) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:28:54 (1763346534) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:29:18 (1763346558) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 21:29:25 (1763346565) mirror create pid 52296 mirror split pid 52298 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52296 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52298 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 21:30:32 (1763346632) Seed set to 6633 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002 1763346634.032022 MIRROR_EXTEND to 2 mirrors 000005 1763346634.467746 MIRROR_RESYNC 2 mirrors 000007 1763346634.625378 mapwrite 0x01a17f thru 0x0296ac (0xf52e bytes) 000008 1763346634.844475 mapread 0x010000 thru 0x019075 (0x9076 bytes) 000010 1763346634.855163 write 0x0275c7 thru 0x034771 (0xd1ab bytes) 000011 1763346634.856531 mapread 0x000b79 thru 0x00470b (0x3b93 bytes) truncating to largest ever: 0x175f5 000012 1763346634.867266 trunc from 0x034772 to 0x0175f4 (0x1d17d bytes) truncating to largest ever: 0x35dd4 000014 1763346634.912825 trunc from 0x0175f5 to 0x035dd3 (0x1e7df bytes) 000020 1763346634.957169 trunc from 0x035dd4 to 0x024414 (0x119bf bytes) 000026 1763346634.983505 read 0x021320 thru 0x024414 (0x30f5 bytes) 000027 1763346634.996468 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24415): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24415): Mirror 2: 0x56d6e1bb (stale) CRC-32 checksum value for chunk [0, 0x24415): Mirror 1: 0x56d6e1bb Mirror 3: 0x56d6e1bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24415: skipped 000028 1763346635.959763 mapread 0x01f5ec thru 0x024414 (0x4e29 bytes) 000029 1763346635.994969 write 0x020bc3 thru 0x022dc9 (0x2207 bytes) 000030 1763346636.110746 MIRROR_RESYNC 3 mirrors 000033 1763346637.057536 MIRROR_EXTEND to 4 mirrors 000036 1763346637.861063 mapwrite 0x01e962 thru 0x02aa16 (0xc0b5 bytes) 000037 1763346638.042972 write 0x02a188 thru 0x02bb9b (0x1a14 bytes) 000039 1763346638.044481 write 0x012699 thru 0x0172ec (0x4c54 bytes) 000043 1763346638.057906 read 0x020b99 thru 0x02bb9b (0xb003 bytes) 000047 1763346638.060210 mapread 0x00c7d2 thru 0x01b684 (0xeeb3 bytes) 000048 1763346638.062559 mapwrite 0x02bc03 thru 0x02df32 (0x2330 bytes) 000049 1763346638.176791 mapread 0x024fb9 thru 0x028a65 (0x3aad bytes) 000052 1763346638.206824 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2df33): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df33): Mirror 2: 0x9e241a47 (stale) Mirror 3: 0x9e241a47 (stale) Mirror 4: 0x9e241a47 (stale) CRC-32 checksum value for chunk [0, 0x2df33): Mirror 1: 0x312a5229 Mirror 5: 0x312a5229 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df33: skipped 000054 1763346639.145651 mapwrite 0x004019 thru 0x0101da (0xc1c2 bytes) 000055 1763346639.331697 mapwrite 0x015244 thru 0x01af93 (0x5d50 bytes) 000056 1763346639.396079 read 0x022e00 thru 0x02df32 (0xb133 bytes) 000057 1763346639.406511 mapwrite 0x00ba72 thru 0x013c0c (0x819b bytes) 000058 1763346639.471628 read 0x005ed0 thru 0x010c23 (0xad54 bytes) 000060 1763346639.481550 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df33): Mirror 2: 0x9e241a47 (stale) Mirror 4: 0x9e241a47 (stale) Mirror 5: 0x312a5229 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df33): Mirror 2: 0x9e241a47 (stale) Mirror 4: 0x9e241a47 (stale) Mirror 5: 0x312a5229 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df33: skipped 000061 1763346639.839406 read 0x02d94f thru 0x02df32 (0x5e4 bytes) 000062 1763346639.851588 mapread 0x011dd8 thru 0x018299 (0x64c2 bytes) 000067 1763346639.894785 trunc from 0x02df33 to 0x001433 (0x2caff bytes) 000068 1763346639.979750 mapwrite 0x00fa25 thru 0x01902e (0x960a bytes) 000069 1763346640.158841 mapwrite 0x015902 thru 0x01905d (0x375c bytes) 000070 1763346640.224545 read 0x016b0e thru 0x01905d (0x2550 bytes) 000071 1763346640.225190 write 0x02deb8 thru 0x02e13a (0x283 bytes) 000073 1763346640.228003 read 0x00728d thru 0x008953 (0x16c7 bytes) 000074 1763346640.249385 MIRROR_RESYNC 4 mirrors 000076 1763346641.061659 mapwrite 0x00c1da thru 0x00f010 (0x2e37 bytes) 000077 1763346641.223635 trunc from 0x02e13b to 0x00b49a (0x22ca0 bytes) 000078 1763346641.281235 read 0x003eac thru 0x007b76 (0x3ccb bytes) 000079 1763346641.292456 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb49b): Mirror 4: 0xf7e2bd18 (stale) Mirror 5: 0xf7e2bd18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb49b): Mirror 4: 0xf7e2bd18 (stale) Mirror 5: 0xf7e2bd18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb49b: skipped 000080 1763346641.606775 mapwrite 0x024fa7 thru 0x026ce0 (0x1d3a bytes) truncating to largest ever: 0x3e99a 000083 1763346641.700725 trunc from 0x026ce1 to 0x03e999 (0x17cb9 bytes) 000084 1763346641.725185 mapwrite 0x013859 thru 0x022a49 (0xf1f1 bytes) 000085 1763346641.795128 trunc from 0x03e99a to 0x020093 (0x1e906 bytes) 000087 1763346641.817437 read 0x000432 thru 0x000f1b (0xaea bytes) 000088 1763346641.824189 trunc from 0x020094 to 0x0265ab (0x6518 bytes) 000089 1763346641.839458 read 0x00a708 thru 0x00d565 (0x2e5e bytes) 000090 1763346641.855676 trunc from 0x0265ac to 0x03208e (0xbae3 bytes) 000091 1763346641.872335 trunc from 0x03208f to 0x013937 (0x1e757 bytes) 000092 1763346641.890671 write 0x01b7f3 thru 0x02b583 (0xfd91 bytes) 000093 1763346641.893160 mapread 0x0074a6 thru 0x0088eb (0x1446 bytes) 000094 1763346641.894820 write 0x0213f6 thru 0x02cfeb (0xbbf6 bytes) 000096 1763346641.896567 read 0x003fff thru 0x00f17a (0xb17c bytes) 000097 1763346641.897241 trunc from 0x02cfec to 0x01be91 (0x1115a bytes) 000099 1763346641.925769 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1be92): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1be92): Mirror 4: 0xc61dcdad (stale) Mirror 5: 0xc61dcdad (stale) CRC-32 checksum value for chunk [0, 0x1be92): Mirror 1: 0xf6453951 Mirror 6: 0xf6453951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1be92: skipped 000100 1763346643.004427 mapread 0x011cfa thru 0x01925e (0x7565 bytes) 000102 1763346643.065961 mapread 0x00e257 thru 0x01a95e (0xc708 bytes) 000103 1763346643.097778 mapread 0x015478 thru 0x01be91 (0x6a1a bytes) 000104 1763346643.143568 read 0x01261b thru 0x018765 (0x614b bytes) 000105 1763346643.165896 mapwrite 0x0382d1 thru 0x03ffff (0x7d2f bytes) 000107 1763346643.435923 trunc from 0x040000 to 0x01c7e2 (0x2381d bytes) truncating to largest ever: 0x3f28b 000108 1763346643.468031 trunc from 0x01c7e3 to 0x03f28a (0x22aa8 bytes) 000109 1763346643.492815 mapwrite 0x00bfd0 thru 0x0116b5 (0x56e6 bytes) 000110 1763346643.610636 mapread 0x02f9a8 thru 0x02fef5 (0x54e bytes) 000111 1763346643.627835 mapread 0x0142e3 thru 0x017e44 (0x3b62 bytes) 000113 1763346643.644058 mapread 0x00f5ae thru 0x015f50 (0x69a3 bytes) 000114 1763346643.644917 read 0x031cdc thru 0x03f28a (0xd5af bytes) 000116 1763346643.660093 trunc from 0x03f28b to 0x02e5c9 (0x10cc1 bytes) 000117 1763346643.691444 read 0x00a1a5 thru 0x00f5b0 (0x540c bytes) 000118 1763346643.692075 mapwrite 0x01d4f7 thru 0x01ea58 (0x1562 bytes) 000119 1763346643.758648 mapread 0x02c5c0 thru 0x02e5c9 (0x200a bytes) 000120 1763346643.760406 write 0x018283 thru 0x01ba30 (0x37ae bytes) 000123 1763346643.762312 trunc from 0x02e5ca to 0x017dd5 (0x167f4 bytes) 000126 1763346643.792757 write 0x027138 thru 0x030b43 (0x9a0c bytes) 000127 1763346643.794526 trunc from 0x030b44 to 0x026a7c (0xa0c7 bytes) 000129 1763346643.814842 trunc from 0x026a7d to 0x034dd3 (0xe357 bytes) 000130 1763346643.835040 read 0x00298e thru 0x009ced (0x7360 bytes) 000132 1763346643.841946 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34dd4): Mirror 5: 0xa9c749cf (stale) Mirror 6: 0x2f1067ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dd4): Mirror 5: 0xa9c749cf (stale) Mirror 6: 0x2f1067ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dd4: skipped 000136 1763346644.307342 mapread 0x0272bf thru 0x02a703 (0x3445 bytes) 000137 1763346644.323776 mapread 0x01325e thru 0x016e6f (0x3c12 bytes) 000138 1763346644.333068 read 0x02906c thru 0x02d5d4 (0x4569 bytes) 000140 1763346644.334013 read 0x0293b5 thru 0x0342c0 (0xaf0c bytes) 000145 1763346644.344748 mapread 0x0331c5 thru 0x034dd3 (0x1c0f bytes) 000147 1763346644.345950 mapwrite 0x033e8c thru 0x037f8f (0x4104 bytes) 000148 1763346644.425407 mapwrite 0x00cb71 thru 0x018c9e (0xc12e bytes) 000149 1763346644.490764 write 0x03d55c thru 0x03ffff (0x2aa4 bytes) 000151 1763346644.492275 write 0x01e1d0 thru 0x0205a5 (0x23d6 bytes) 000153 1763346644.499671 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfbb3331f (stale) Mirror 6: 0xaa88a53f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e318fc2 Mirror 7: 0x3e318fc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000154 1763346645.353344 mapwrite 0x0059dd thru 0x011ae9 (0xc10d bytes) 000155 1763346645.599787 write 0x036123 thru 0x03ffff (0x9edd bytes) 000156 1763346645.616018 mapwrite 0x035061 thru 0x039162 (0x4102 bytes) 000159 1763346645.672682 write 0x009d6e thru 0x01617e (0xc411 bytes) 000160 1763346645.673889 mapread 0x03acdb thru 0x03d220 (0x2546 bytes) 000161 1763346645.674632 mapwrite 0x00c703 thru 0x00e62f (0x1f2d bytes) 000162 1763346645.739128 trunc from 0x040000 to 0x03aa60 (0x559f bytes) 000166 1763346645.806799 read 0x01e0e8 thru 0x02cabc (0xe9d5 bytes) 000167 1763346645.822668 write 0x00cba0 thru 0x01bb81 (0xefe2 bytes) 000172 1763346645.832279 mapread 0x00d37f thru 0x0178ce (0xa550 bytes) 000173 1763346645.833448 mapread 0x031a3b thru 0x032e09 (0x13cf bytes) 000174 1763346645.834235 mapwrite 0x00830e thru 0x0150a0 (0xcd93 bytes) 000175 1763346645.903724 write 0x025ec6 thru 0x03210a (0xc245 bytes) 000177 1763346645.905659 mapwrite 0x005cca thru 0x00b21a (0x5551 bytes) 000178 1763346645.965385 mapwrite 0x0140c0 thru 0x02184b (0xd78c bytes) 000182 1763346646.038053 read 0x0156cb thru 0x018559 (0x2e8f bytes) 000183 1763346646.038663 mapread 0x018083 thru 0x018813 (0x791 bytes) 000185 1763346646.047972 MIRROR_RESYNC 4 mirrors 000186 1763346647.047638 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000188 1763346647.586179 read 0x030a7b thru 0x032179 (0x16ff bytes) 000190 1763346647.608676 read 0x02b8e2 thru 0x02f8ff (0x401e bytes) 000192 1763346647.638606 MIRROR_EXTEND to 4 mirrors 000193 1763346648.409722 mapwrite 0x013cb9 thru 0x017d75 (0x40bd bytes) 000194 1763346648.585123 write 0x00cf62 thru 0x019b97 (0xcc36 bytes) 000197 1763346648.595696 mapread 0x02437f thru 0x02df5a (0x9bdc bytes) 000200 1763346648.621991 mapwrite 0x010678 thru 0x0154ac (0x4e35 bytes) 000201 1763346648.679664 mapread 0x004bc5 thru 0x00dd60 (0x919c bytes) 000205 1763346648.689115 trunc from 0x03aa61 to 0x008d80 (0x31ce0 bytes) 000206 1763346648.717175 trunc from 0x008d81 to 0x03beb5 (0x33135 bytes) 000208 1763346648.753096 mapread 0x029242 thru 0x032a1f (0x97de bytes) 000209 1763346648.786878 write 0x0283b3 thru 0x02961e (0x126c bytes) 000210 1763346648.787733 mapwrite 0x014bd1 thru 0x0239b6 (0xede6 bytes) 000214 1763346648.876141 mapread 0x01036f thru 0x016e44 (0x6ad6 bytes) 000215 1763346648.886030 MIRROR_RESYNC 4 mirrors 000217 1763346650.002931 trunc from 0x03beb6 to 0x013151 (0x28d64 bytes) 000218 1763346650.118055 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13152): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13152): Mirror 6: 0x7bdc6634 (stale) Mirror 7: 0x7bdc6634 (stale) Mirror 8: 0x7bdc6634 (stale) CRC-32 checksum value for chunk [0, 0x13152): Mirror 1: 0x7bdc6634 Mirror 9: 0x7bdc6634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13152: skipped 000220 1763346650.938848 read 0x0070c5 thru 0x00bfc4 (0x4f00 bytes) 000221 1763346650.988553 read 0x00e9dd thru 0x013151 (0x4775 bytes) 000222 1763346651.047629 mapread 0x0089bd thru 0x011dbf (0x9403 bytes) 000223 1763346651.098669 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13152): Mirror 7: 0x7bdc6634 (stale) Mirror 8: 0x7bdc6634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13152): Mirror 7: 0x7bdc6634 (stale) Mirror 8: 0x7bdc6634 (stale) CRC-32 checksum value for chunk [0, 0x13152): Mirror 1: 0x7bdc6634 Mirror 9: 0x7bdc6634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13152: skipped 000227 1763346651.874088 mapwrite 0x00ce4b thru 0x00cfdb (0x191 bytes) 000229 1763346652.056632 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13152): Mirror 8: 0x7bdc6634 (stale) Mirror 9: 0x7bdc6634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13152): Mirror 8: 0x7bdc6634 (stale) Mirror 9: 0x7bdc6634 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13152: skipped 000231 1763346652.401702 trunc from 0x013152 to 0x023d9d (0x10c4c bytes) 000234 1763346652.424562 mapread 0x00d3ee thru 0x00e99c (0x15af bytes) 000235 1763346652.436903 mapread 0x0159e0 thru 0x023d9d (0xe3be bytes) 000237 1763346652.454083 trunc from 0x023d9e to 0x029040 (0x52a3 bytes) 000238 1763346652.469458 trunc from 0x029041 to 0x01fbd2 (0x946e bytes) 000239 1763346652.488995 read 0x00e317 thru 0x0126a2 (0x438c bytes) 000240 1763346652.489634 mapread 0x0053d8 thru 0x013a49 (0xe672 bytes) 000241 1763346652.501278 mapread 0x017f28 thru 0x01b0a4 (0x317d bytes) 000245 1763346652.507363 mapwrite 0x02faec thru 0x036fd1 (0x74e6 bytes) 000247 1763346652.589805 read 0x02f2ac thru 0x031bae (0x2903 bytes) 000248 1763346652.590402 mapwrite 0x02edde thru 0x02f7f0 (0xa13 bytes) 000251 1763346652.642434 trunc from 0x036fd2 to 0x005ce1 (0x312f0 bytes) 000252 1763346652.663415 read 0x00236c thru 0x005ce1 (0x3976 bytes) 000253 1763346652.664006 read 0x0036c0 thru 0x003b21 (0x462 bytes) 000256 1763346652.665305 trunc from 0x005ce2 to 0x011586 (0xb8a5 bytes) 000258 1763346652.700551 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11587): Mirror 8: 0xacce4b35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11587): Mirror 8: 0xacce4b35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11587: skipped 000259 1763346652.973634 read 0x009aad thru 0x00aad1 (0x1025 bytes) 000261 1763346652.986732 trunc from 0x011587 to 0x01872e (0x71a8 bytes) 000262 1763346653.032654 mapwrite 0x036310 thru 0x0363c9 (0xba bytes) 000263 1763346653.107689 write 0x02cea2 thru 0x03b381 (0xe4e0 bytes) 000266 1763346653.115564 read 0x00fcf6 thru 0x015a73 (0x5d7e bytes) 000269 1763346653.126308 read 0x02dd58 thru 0x03b381 (0xd62a bytes) 000270 1763346653.127334 trunc from 0x03b382 to 0x010bb0 (0x2a7d1 bytes) 000272 1763346653.151778 trunc from 0x010bb1 to 0x02172e (0x10b7e bytes) 000273 1763346653.177738 trunc from 0x02172f to 0x02020e (0x1520 bytes) 000275 1763346653.191883 write 0x01ce07 thru 0x0232a2 (0x649c bytes) 000276 1763346653.203263 write 0x02719b thru 0x028ce0 (0x1b46 bytes) 000278 1763346653.204687 mapwrite 0x03f101 thru 0x03ffff (0xeff bytes) 000283 1763346653.284026 trunc from 0x040000 to 0x02d43c (0x12bc3 bytes) 000284 1763346653.306008 read 0x014da0 thru 0x016238 (0x1499 bytes) 000287 1763346653.309670 read 0x01e7af thru 0x024744 (0x5f96 bytes) 000292 1763346653.311929 read 0x00611e thru 0x00ebf0 (0x8ad3 bytes) 000293 1763346653.313310 mapread 0x00af94 thru 0x01824a (0xd2b7 bytes) 000295 1763346653.315585 mapread 0x02b0b4 thru 0x02d43c (0x2389 bytes) 000298 1763346653.324968 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000299 1763346653.486069 mapread 0x0066cb thru 0x0096da (0x3010 bytes) 000300 1763346653.522337 mapread 0x001f96 thru 0x006156 (0x41c1 bytes) 000304 1763346653.524600 read 0x0104de thru 0x01c780 (0xc2a3 bytes) 000309 1763346653.547736 MIRROR_EXTEND to 2 mirrors 000312 1763346654.131374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000313 1763346654.303289 mapread 0x01696d thru 0x024509 (0xdb9d bytes) 000314 1763346654.335395 write 0x020dc8 thru 0x029d18 (0x8f51 bytes) 000316 1763346654.361020 mapwrite 0x0315f5 thru 0x03fba5 (0xe5b1 bytes) 000317 1763346654.485882 write 0x010ada thru 0x016764 (0x5c8b bytes) 000318 1763346654.505459 write 0x02d363 thru 0x031457 (0x40f5 bytes) 000319 1763346654.506719 trunc from 0x03fba6 to 0x00210c (0x3da99 bytes) 000321 1763346654.531229 trunc from 0x00210d to 0x03b6db (0x395cf bytes) 000322 1763346654.550527 mapread 0x00100e thru 0x00fe25 (0xee18 bytes) 000326 1763346654.566071 trunc from 0x03b6dc to 0x03cd15 (0x163a bytes) 000327 1763346654.585137 mapread 0x02af92 thru 0x034281 (0x92f0 bytes) 000328 1763346654.613557 read 0x01a30d thru 0x02450b (0xa1ff bytes) 000331 1763346654.625643 write 0x014328 thru 0x015a74 (0x174d bytes) 000333 1763346654.641565 write 0x02561b thru 0x032f28 (0xd90e bytes) 000334 1763346654.642943 mapwrite 0x00890b thru 0x015f2b (0xd621 bytes) 000338 1763346654.716635 write 0x02277e thru 0x02731f (0x4ba2 bytes) 000340 1763346654.738536 MIRROR_EXTEND to 2 mirrors 000342 1763346655.446660 mapread 0x00eca8 thru 0x01eba9 (0xff02 bytes) 000343 1763346655.531764 read 0x016554 thru 0x02273f (0xc1ec bytes) 000344 1763346655.548082 trunc from 0x03cd16 to 0x005e49 (0x36ecc bytes) 000345 1763346655.658230 mapread 0x003458 thru 0x005e49 (0x29f2 bytes) 000347 1763346655.674810 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x5e4a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5e4a): Mirror 2: 0x396fc99d (stale) CRC-32 checksum value for chunk [0, 0x5e4a): Mirror 1: 0x396fc99d Mirror 3: 0x396fc99d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5e4a: skipped 000348 1763346656.557273 mapwrite 0x035b46 thru 0x037c32 (0x20ed bytes) 000349 1763346656.716578 trunc from 0x037c33 to 0x01a23d (0x1d9f5 bytes) 000350 1763346656.743888 trunc from 0x01a23e to 0x03a4e4 (0x202a7 bytes) 000351 1763346656.768674 mapread 0x033d3a thru 0x03a4e4 (0x67ab bytes) 000353 1763346656.779289 write 0x01ea59 thru 0x02361a (0x4bc2 bytes) 000354 1763346656.792792 mapread 0x0275f0 thru 0x03516d (0xdb7e bytes) 000355 1763346656.802127 mapwrite 0x02154a thru 0x024168 (0x2c1f bytes) 000356 1763346656.861063 mapread 0x0365df thru 0x03a4e4 (0x3f06 bytes) 000358 1763346656.862353 mapread 0x0381b8 thru 0x03a4e4 (0x232d bytes) 000360 1763346656.863523 mapwrite 0x03449a thru 0x03abcf (0x6736 bytes) 000361 1763346656.955349 trunc from 0x03abd0 to 0x026307 (0x148c8 bytes) 000362 1763346656.988304 mapread 0x01d792 thru 0x026307 (0x8b76 bytes) 000363 1763346656.991823 trunc from 0x026308 to 0x027194 (0xe8d bytes) 000369 1763346657.027990 mapread 0x005475 thru 0x009e2e (0x49ba bytes) 000370 1763346657.045053 mapwrite 0x00ca7b thru 0x00e7c3 (0x1d49 bytes) 000374 1763346657.107220 write 0x036034 thru 0x036c5a (0xc27 bytes) 000375 1763346657.109079 read 0x005932 thru 0x007ea1 (0x2570 bytes) 000376 1763346657.110424 mapwrite 0x015c6a thru 0x01787c (0x1c13 bytes) 000378 1763346657.173066 write 0x0071f5 thru 0x00e6b8 (0x74c4 bytes) 000380 1763346657.174496 mapread 0x002d0a thru 0x00ad1d (0x8014 bytes) 000382 1763346657.177186 mapwrite 0x00e0c7 thru 0x014e93 (0x6dcd bytes) 000387 1763346657.237674 mapwrite 0x00fdc9 thru 0x01bf6a (0xc1a2 bytes) 000388 1763346657.283625 write 0x02880f thru 0x02b02c (0x281e bytes) 000389 1763346657.296345 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36c5b): Mirror 2: 0xbe79ec4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c5b): Mirror 2: 0xbe79ec4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c5b: skipped 000390 1763346657.634045 write 0x008684 thru 0x00c566 (0x3ee3 bytes) 000394 1763346657.660429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000395 1763346657.806088 trunc from 0x036c5b to 0x012703 (0x24557 bytes) 000397 1763346657.826541 mapwrite 0x0326fe thru 0x03a831 (0x8134 bytes) 000398 1763346657.965358 mapwrite 0x025588 thru 0x034fee (0xfa67 bytes) 000399 1763346658.022137 mapread 0x030a0c thru 0x030cac (0x2a1 bytes) 000401 1763346658.023297 read 0x02c42f thru 0x02f256 (0x2e28 bytes) 000402 1763346658.023754 write 0x025abc thru 0x0298cc (0x3e11 bytes) 000403 1763346658.024568 mapwrite 0x002a4c thru 0x00f2bb (0xc870 bytes) 000406 1763346658.115136 write 0x006840 thru 0x008e28 (0x25e9 bytes) 000408 1763346658.117534 mapread 0x00dc88 thru 0x01d678 (0xf9f1 bytes) 000409 1763346658.134470 trunc from 0x03a832 to 0x00c398 (0x2e499 bytes) 000411 1763346658.153834 write 0x003c4a thru 0x00b0e4 (0x749b bytes) 000413 1763346658.155470 mapread 0x009b06 thru 0x00c04e (0x2549 bytes) 000415 1763346658.157562 read 0x00b78a thru 0x00c398 (0xc0f bytes) 000418 1763346658.159608 mapwrite 0x03b847 thru 0x03ffff (0x47b9 bytes) 000419 1763346658.250110 write 0x0032e3 thru 0x006f41 (0x3c5f bytes) 000422 1763346658.251807 write 0x024cd8 thru 0x02be3c (0x7165 bytes) 000423 1763346658.252726 write 0x0194e2 thru 0x02531f (0xbe3e bytes) 000425 1763346658.254672 trunc from 0x040000 to 0x005c89 (0x3a376 bytes) 000426 1763346658.283968 mapread 0x0010e5 thru 0x005c89 (0x4ba5 bytes) 000430 1763346658.287604 read 0x005620 thru 0x005c89 (0x66a bytes) 000433 1763346658.289390 trunc from 0x005c8a to 0x0395d3 (0x3394a bytes) 000434 1763346658.316098 MIRROR_EXTEND to 2 mirrors 000437 1763346658.998259 mapwrite 0x00266f thru 0x01130f (0xeca1 bytes) 000438 1763346659.267489 read 0x028a27 thru 0x02bb8d (0x3167 bytes) 000440 1763346659.281012 MIRROR_RESYNC 2 mirrors 000441 1763346659.871811 mapwrite 0x013f55 thru 0x01a632 (0x66de bytes) 000442 1763346660.049523 trunc from 0x0395d4 to 0x03b330 (0x1d5d bytes) 000443 1763346660.111374 trunc from 0x03b331 to 0x0379e3 (0x394d bytes) 000444 1763346660.150742 mapread 0x02dc10 thru 0x0379e3 (0x9dd4 bytes) 000446 1763346660.177125 mapread 0x0050f4 thru 0x006af9 (0x1a06 bytes) 000447 1763346660.209062 mapread 0x025a1d thru 0x0351dc (0xf7c0 bytes) 000448 1763346660.234967 write 0x00dc3a thru 0x01bf8d (0xe354 bytes) 000449 1763346660.236304 mapread 0x01cb52 thru 0x01ddb3 (0x1262 bytes) 000450 1763346660.248617 write 0x0025db thru 0x00862e (0x6054 bytes) 000451 1763346660.249601 write 0x016a5f thru 0x01df0a (0x74ac bytes) 000455 1763346660.257005 read 0x027eeb thru 0x02ab70 (0x2c86 bytes) 000457 1763346660.258466 trunc from 0x0379e4 to 0x00a7dc (0x2d207 bytes) 000459 1763346660.295536 trunc from 0x00a7dd to 0x031d4e (0x27572 bytes) 000461 1763346660.349136 MIRROR_RESYNC 2 mirrors 000464 1763346660.985051 write 0x00e267 thru 0x0123c3 (0x415d bytes) 000467 1763346661.082588 write 0x02644c thru 0x0276dc (0x1291 bytes) 000470 1763346661.094578 mapread 0x00c4ad thru 0x01540d (0x8f61 bytes) 000471 1763346661.125928 read 0x003280 thru 0x011cd7 (0xea58 bytes) 000472 1763346661.152802 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000473 1763346661.319348 trunc from 0x031d4f to 0x0359a2 (0x3c54 bytes) 000475 1763346661.340766 write 0x03fdd0 thru 0x03ffff (0x230 bytes) 000477 1763346661.343644 mapread 0x03df5e thru 0x03ffff (0x20a2 bytes) 000478 1763346661.358391 mapwrite 0x02092b thru 0x027897 (0x6f6d bytes) 000479 1763346661.440570 mapread 0x023715 thru 0x024810 (0x10fc bytes) 000480 1763346661.441423 read 0x004e5a thru 0x014e2d (0xffd4 bytes) 000483 1763346661.454428 trunc from 0x040000 to 0x0190f8 (0x26f07 bytes) 000484 1763346661.486845 trunc from 0x0190f9 to 0x028d54 (0xfc5c bytes) 000485 1763346661.518383 mapwrite 0x001c60 thru 0x0108e6 (0xec87 bytes) 000487 1763346661.613520 read 0x015f0f thru 0x01ff4a (0xa03c bytes) 000492 1763346661.624954 mapread 0x014441 thru 0x0214fd (0xd0bd bytes) 000500 1763346661.629723 write 0x01dd8a thru 0x029dcc (0xc043 bytes) 000501 1763346661.631705 trunc from 0x029dcd to 0x01a4f3 (0xf8d9 bytes) 000502 1763346661.662357 write 0x03fbc0 thru 0x03ffff (0x440 bytes) 000503 1763346661.668419 read 0x037a8f thru 0x03ffff (0x8571 bytes) 000504 1763346661.680536 mapread 0x007c1c thru 0x017b47 (0xff2c bytes) 000508 1763346661.683514 read 0x0032a0 thru 0x009142 (0x5ea3 bytes) 000509 1763346661.684888 read 0x000149 thru 0x005a05 (0x58bd bytes) 000510 1763346661.686530 mapread 0x025068 thru 0x02a143 (0x50dc bytes) 000511 1763346661.687377 read 0x02b4cf thru 0x0326be (0x71f0 bytes) 000514 1763346661.688720 mapwrite 0x033c60 thru 0x035709 (0x1aaa bytes) 000519 1763346661.757491 MIRROR_EXTEND to 2 mirrors 000521 1763346662.444578 write 0x028c58 thru 0x032bc7 (0x9f70 bytes) 000522 1763346662.525568 MIRROR_RESYNC 2 mirrors 000526 1763346663.099564 write 0x02d390 thru 0x03448b (0x70fc bytes) 000527 1763346663.161259 mapread 0x01bbe8 thru 0x021b4d (0x5f66 bytes) 000529 1763346663.176301 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf243fb16 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1e9f2f0e Mirror 3: 0x1e9f2f0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000530 1763346663.897008 trunc from 0x040000 to 0x03cd76 (0x3289 bytes) 000533 1763346664.026650 trunc from 0x03cd77 to 0x030e7e (0xbef8 bytes) 000534 1763346664.055339 read 0x001235 thru 0x00d6ca (0xc496 bytes) 000536 1763346664.065567 read 0x02db3f thru 0x030e7e (0x3340 bytes) 000537 1763346664.080818 read 0x0265ff thru 0x030e7e (0xa880 bytes) 000538 1763346664.081499 trunc from 0x030e7f to 0x02a0fe (0x6d80 bytes) 000540 1763346664.101653 mapwrite 0x031cd0 thru 0x03628b (0x45bc bytes) 000542 1763346664.232073 mapwrite 0x03435e thru 0x0387b2 (0x4455 bytes) 000543 1763346664.304539 read 0x01661a thru 0x022336 (0xbd1d bytes) 000546 1763346664.307006 trunc from 0x0387b3 to 0x0275cf (0x111e3 bytes) 000552 1763346664.340209 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x275d0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275d0): Mirror 2: 0xacf4aba2 (stale) Mirror 3: 0xacf4aba2 (stale) CRC-32 checksum value for chunk [0, 0x275d0): Mirror 1: 0xacf4aba2 Mirror 4: 0xacf4aba2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275d0: skipped 000554 1763346665.192890 write 0x00fb03 thru 0x0192de (0x97dc bytes) 000555 1763346665.292757 trunc from 0x0275d0 to 0x03d8f8 (0x16329 bytes) 000556 1763346665.374178 read 0x031cb0 thru 0x03d8f8 (0xbc49 bytes) 000560 1763346665.393011 mapread 0x010ca8 thru 0x014814 (0x3b6d bytes) 000561 1763346665.396736 mapread 0x036cf7 thru 0x03d8f8 (0x6c02 bytes) 000563 1763346665.398064 mapread 0x00c236 thru 0x010aab (0x4876 bytes) 000568 1763346665.418490 mapread 0x0375ef thru 0x03d8f8 (0x630a bytes) 000570 1763346665.422276 mapread 0x024524 thru 0x030a60 (0xc53d bytes) 000571 1763346665.444713 read 0x00e73c thru 0x011ebd (0x3782 bytes) 000575 1763346665.449521 mapread 0x00b06b thru 0x013c37 (0x8bcd bytes) 000577 1763346665.451901 write 0x0141fd thru 0x0231c5 (0xefc9 bytes) 000579 1763346665.455680 trunc from 0x03d8f9 to 0x03a3bb (0x353d bytes) 000580 1763346665.510157 MIRROR_RESYNC 4 mirrors 000586 1763346666.312541 write 0x0279f0 thru 0x032d1b (0xb32c bytes) 000587 1763346666.437646 read 0x031610 thru 0x03a3bb (0x8dac bytes) 000588 1763346666.454710 read 0x0123fe thru 0x01479b (0x239e bytes) 000589 1763346666.468467 write 0x024762 thru 0x02f16b (0xaa0a bytes) 000591 1763346666.470699 read 0x01a0be thru 0x02716f (0xd0b2 bytes) 000592 1763346666.478447 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a3bc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3bc): Mirror 2: 0xf54d3dd5 (stale) Mirror 3: 0xf54d3dd5 (stale) Mirror 4: 0xf54d3dd5 (stale) CRC-32 checksum value for chunk [0, 0x3a3bc): Mirror 1: 0x97ec714d Mirror 5: 0x97ec714d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3bc: skipped 000593 1763346667.327885 trunc from 0x03a3bc to 0x03e05b (0x3ca0 bytes) 000595 1763346667.462191 mapwrite 0x03323c thru 0x03c421 (0x91e6 bytes) 000596 1763346667.557402 read 0x016976 thru 0x02246c (0xbaf7 bytes) 000597 1763346667.569794 mapread 0x01f19f thru 0x02ce03 (0xdc65 bytes) 000599 1763346667.587275 trunc from 0x03e05c to 0x0196ad (0x249ae bytes) 000600 1763346667.617512 read 0x018f6d thru 0x0196ad (0x741 bytes) 000601 1763346667.618086 mapread 0x00db19 thru 0x01484b (0x6d33 bytes) 000603 1763346667.627590 trunc from 0x0196ae to 0x020742 (0x7095 bytes) 000605 1763346667.648756 mapread 0x016893 thru 0x020742 (0x9eb0 bytes) 000606 1763346667.663970 mapwrite 0x018632 thru 0x025877 (0xd246 bytes) 000607 1763346667.759553 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25878): Mirror 2: 0xa21a26e3 (stale) Mirror 4: 0xa21a26e3 (stale) Mirror 5: 0x363c8ca8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25878): Mirror 2: 0xa21a26e3 (stale) Mirror 4: 0xa21a26e3 (stale) Mirror 5: 0x363c8ca8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25878: skipped 000608 1763346668.133118 write 0x010bb7 thru 0x01e2f3 (0xd73d bytes) 000610 1763346668.143974 write 0x0250ef thru 0x02c9e8 (0x78fa bytes) 000611 1763346668.155574 read 0x008cc5 thru 0x011e2b (0x9167 bytes) 000612 1763346668.167821 trunc from 0x02c9e9 to 0x0004b8 (0x2c530 bytes) 000613 1763346668.201225 MIRROR_RESYNC 4 mirrors 000614 1763346669.010711 mapwrite 0x01a4cb thru 0x021225 (0x6d5b bytes) 000615 1763346669.206781 read 0x010ac6 thru 0x017c3d (0x7178 bytes) 000616 1763346669.220485 trunc from 0x021226 to 0x03006f (0xee4a bytes) 000617 1763346669.258305 mapwrite 0x00acf6 thru 0x01595c (0xac67 bytes) 000618 1763346669.326837 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30070): Mirror 2: 0x499e51f1 (stale) Mirror 5: 0x499e51f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30070): Mirror 2: 0x499e51f1 (stale) Mirror 5: 0x499e51f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30070: skipped 000619 1763346669.699989 write 0x03110d thru 0x03928f (0x8183 bytes) 000622 1763346669.708529 MIRROR_RESYNC 3 mirrors 000623 1763346670.549165 MIRROR_RESYNC 3 mirrors 000624 1763346670.965774 read 0x033e9e thru 0x03928f (0x53f2 bytes) 000625 1763346670.977060 trunc from 0x039290 to 0x002e0f (0x36480 bytes) 000627 1763346671.081804 mapwrite 0x03e3d8 thru 0x03ffff (0x1c28 bytes) 000628 1763346671.189209 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdeaa7f80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdeaa7f80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1763346671.479491 mapwrite 0x03082c thru 0x03f4ce (0xeca3 bytes) 000631 1763346671.553592 trunc from 0x040000 to 0x01daf8 (0x22507 bytes) 000633 1763346671.570666 mapwrite 0x014c21 thru 0x02025c (0xb63c bytes) 000635 1763346671.676294 write 0x02fff4 thru 0x034bba (0x4bc7 bytes) 000640 1763346671.685705 write 0x029a3e thru 0x0354a4 (0xba67 bytes) 000643 1763346671.714585 write 0x010a2d thru 0x015fc6 (0x559a bytes) 000645 1763346671.724188 write 0x0106e9 thru 0x012440 (0x1d58 bytes) 000646 1763346671.726396 read 0x0291a6 thru 0x02e284 (0x50df bytes) 000647 1763346671.727317 trunc from 0x0354a5 to 0x0175af (0x1def5 bytes) 000648 1763346671.759186 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000649 1763346671.977897 trunc from 0x0175b0 to 0x02b3bd (0x13e0e bytes) 000653 1763346672.008190 write 0x01c3e3 thru 0x020923 (0x4541 bytes) 000654 1763346672.027233 read 0x012831 thru 0x014c17 (0x23e7 bytes) 000656 1763346672.041024 mapwrite 0x00599c thru 0x00c509 (0x6b6e bytes) 000664 1763346672.100428 mapread 0x024d79 thru 0x02b3bd (0x6645 bytes) 000666 1763346672.115147 write 0x039df3 thru 0x03ffff (0x620d bytes) 000674 1763346672.127521 MIRROR_EXTEND to 2 mirrors 000676 1763346672.637961 read 0x027d64 thru 0x031129 (0x93c6 bytes) 000677 1763346672.651764 trunc from 0x040000 to 0x0013e2 (0x3ec1d bytes) 000678 1763346672.784958 read 0x0009c4 thru 0x0013e2 (0xa1f bytes) 000681 1763346672.796625 trunc from 0x0013e3 to 0x02cd25 (0x2b943 bytes) 000683 1763346672.838123 read 0x01ed45 thru 0x02925e (0xa51a bytes) 000686 1763346672.855488 trunc from 0x02cd26 to 0x01f2aa (0xda7b bytes) 000687 1763346672.881600 write 0x002d48 thru 0x00cb3f (0x9df8 bytes) 000688 1763346672.882705 mapwrite 0x00189c thru 0x011654 (0xfdb9 bytes) 000689 1763346672.945714 mapread 0x009e9d thru 0x015656 (0xb7ba bytes) 000690 1763346672.953404 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f2ab): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2ab): Mirror 2: 0xdbb3fd82 (stale) CRC-32 checksum value for chunk [0, 0x1f2ab): Mirror 1: 0x8fd4d6ee Mirror 3: 0x8fd4d6ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2ab: skipped 000691 1763346673.856205 write 0x013650 thru 0x01ea43 (0xb3f4 bytes) 000694 1763346673.947754 trunc from 0x01f2ab to 0x02db70 (0xe8c6 bytes) 000698 1763346673.968046 write 0x00eb61 thru 0x0166c5 (0x7b65 bytes) 000699 1763346673.982105 mapwrite 0x0386fd thru 0x03baa2 (0x33a6 bytes) 000701 1763346674.087761 read 0x025c23 thru 0x02daef (0x7ecd bytes) 000703 1763346674.111314 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baa3): Mirror 2: 0xbd12e4b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baa3): Mirror 2: 0xbd12e4b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baa3: skipped 000704 1763346674.476259 trunc from 0x03baa3 to 0x03e99d (0x2efb bytes) 000705 1763346674.493477 trunc from 0x03e99e to 0x03a6bf (0x42de bytes) 000706 1763346674.516223 read 0x008b26 thru 0x009ef9 (0x13d4 bytes) 000709 1763346674.529138 mapwrite 0x02cfe7 thru 0x03c5a9 (0xf5c3 bytes) 000710 1763346674.650951 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c5aa): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5aa): Mirror 2: 0xadb21692 (stale) CRC-32 checksum value for chunk [0, 0x3c5aa): Mirror 1: 0xc1fa49d4 Mirror 3: 0xc1fa49d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5aa: skipped 000711 1763346675.308580 read 0x014060 thru 0x020607 (0xc5a8 bytes) 000712 1763346675.327214 write 0x019cae thru 0x0252e1 (0xb634 bytes) 000713 1763346675.394235 mapread 0x00ae05 thru 0x019c8a (0xee86 bytes) 000714 1763346675.420034 write 0x0039df thru 0x005332 (0x1954 bytes) 000715 1763346675.421079 read 0x033da6 thru 0x03b9de (0x7c39 bytes) 000716 1763346675.429459 mapread 0x03119d thru 0x031e82 (0xce6 bytes) 000718 1763346675.434633 MIRROR_RESYNC 3 mirrors 000719 1763346676.119300 mapwrite 0x0059fb thru 0x006a2c (0x1032 bytes) 000720 1763346676.301048 mapwrite 0x003096 thru 0x00d3e0 (0xa34b bytes) 000724 1763346676.362081 read 0x034a31 thru 0x03c5a9 (0x7b79 bytes) 000725 1763346676.375813 trunc from 0x03c5aa to 0x003adf (0x38aca bytes) 000726 1763346676.426229 mapread 0x003724 thru 0x003adf (0x3bc bytes) 000727 1763346676.427537 mapwrite 0x011ea0 thru 0x01bc90 (0x9df1 bytes) 000729 1763346676.531892 write 0x029c88 thru 0x0339f7 (0x9d70 bytes) 000730 1763346676.535130 mapwrite 0x02b5b8 thru 0x034954 (0x939d bytes) 000732 1763346676.615170 write 0x017195 thru 0x0261b9 (0xf025 bytes) 000736 1763346676.622940 mapwrite 0x01fecb thru 0x02da64 (0xdb9a bytes) 000737 1763346676.684771 write 0x0379e4 thru 0x03ffff (0x861c bytes) 000739 1763346676.687616 read 0x034b87 thru 0x03ffff (0xb479 bytes) 000742 1763346676.705314 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cfba121 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cfba121 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1763346677.015910 read 0x01c843 thru 0x02bc6a (0xf428 bytes) 000748 1763346677.030459 mapwrite 0x0109c1 thru 0x01e1ea (0xd82a bytes) 000749 1763346677.169021 mapread 0x033141 thru 0x03c9f8 (0x98b8 bytes) 000751 1763346677.214121 mapread 0x013b7d thru 0x019acc (0x5f50 bytes) 000753 1763346677.215358 mapwrite 0x033877 thru 0x0390f0 (0x587a bytes) 000754 1763346677.289533 write 0x017364 thru 0x0226c7 (0xb364 bytes) 000756 1763346677.291259 mapread 0x039f78 thru 0x03d63d (0x36c6 bytes) 000757 1763346677.292052 read 0x01bdfb thru 0x0289e1 (0xcbe7 bytes) 000760 1763346677.304015 mapwrite 0x01c3bb thru 0x023b21 (0x7767 bytes) 000763 1763346677.392195 trunc from 0x040000 to 0x001017 (0x3efe8 bytes) 000764 1763346677.438374 write 0x00f6f9 thru 0x010b48 (0x1450 bytes) 000765 1763346677.441706 write 0x028431 thru 0x02dd6f (0x593f bytes) 000766 1763346677.443763 mapread 0x023532 thru 0x025f0d (0x29dc bytes) 000767 1763346677.490535 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) 000769 1763346677.701132 trunc from 0x02dd70 to 0x017da6 (0x15fc9 bytes) 000771 1763346677.734367 mapwrite 0x02f769 thru 0x0347b7 (0x504f bytes) 000773 1763346677.836546 write 0x02d654 thru 0x02fb4f (0x24fc bytes) 000774 1763346677.837476 read 0x00458e thru 0x00c1a1 (0x7c14 bytes) 000775 1763346677.861564 trunc from 0x0347b8 to 0x0144a7 (0x20310 bytes) 000776 1763346677.901098 read 0x00af46 thru 0x0144a7 (0x9562 bytes) 000777 1763346677.913781 mapwrite 0x038859 thru 0x03e0ba (0x5862 bytes) 000780 1763346678.093545 mapread 0x02b14f thru 0x0374c1 (0xc373 bytes) 000783 1763346678.097828 trunc from 0x03e0bb to 0x0202d6 (0x1dde4 bytes) 000784 1763346678.128948 write 0x02489c thru 0x02e87c (0x9fe1 bytes) 000786 1763346678.133034 mapwrite 0x012180 thru 0x01ea19 (0xc89a bytes) 000787 1763346678.229800 mapwrite 0x02b469 thru 0x03aee2 (0xfa7a bytes) 000788 1763346678.428129 mapread 0x031bfb thru 0x03aee2 (0x92e8 bytes) 000790 1763346678.430621 write 0x00257a thru 0x006039 (0x3ac0 bytes) 000792 1763346678.433057 mapread 0x023978 thru 0x024f38 (0x15c1 bytes) 000793 1763346678.441679 mapread 0x033cd4 thru 0x03aee2 (0x720f bytes) 000796 1763346678.448368 MIRROR_EXTEND to 2 mirrors 000798 1763346679.132686 read 0x025018 thru 0x02501e (0x7 bytes) 000800 1763346679.159365 mapwrite 0x036aa5 thru 0x03ffff (0x955b bytes) 000802 1763346679.524182 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb40ff5ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88b71ca2 Mirror 3: 0x88b71ca2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000806 1763346680.492035 mapread 0x02222e thru 0x031e71 (0xfc44 bytes) 000807 1763346680.539510 trunc from 0x040000 to 0x00d44b (0x32bb4 bytes) 000808 1763346680.698052 mapwrite 0x02616e thru 0x033920 (0xd7b3 bytes) 000810 1763346680.847408 trunc from 0x033921 to 0x029450 (0xa4d0 bytes) 000811 1763346680.876612 read 0x00140a thru 0x00553e (0x4135 bytes) 000812 1763346680.887738 read 0x010067 thru 0x01e11a (0xe0b4 bytes) 000813 1763346680.908997 read 0x01dc23 thru 0x029450 (0xb82e bytes) 000816 1763346680.918698 trunc from 0x029451 to 0x02fb93 (0x6743 bytes) 000820 1763346680.967199 mapread 0x010d20 thru 0x019ea8 (0x9189 bytes) 000822 1763346680.984048 read 0x02792a thru 0x02fb93 (0x826a bytes) 000823 1763346681.012738 mapread 0x01261b thru 0x01bc62 (0x9648 bytes) 000824 1763346681.022647 write 0x00736c thru 0x00ae37 (0x3acc bytes) 000827 1763346681.040330 trunc from 0x02fb94 to 0x00f5a1 (0x205f2 bytes) 000831 1763346681.090426 MIRROR_RESYNC 3 mirrors 000832 1763346682.058202 trunc from 0x00f5a2 to 0x01848c (0x8eeb bytes) 000835 1763346682.198544 read 0x00f32e thru 0x017d65 (0x8a38 bytes) 000836 1763346682.206594 read 0x0072b1 thru 0x00841f (0x116f bytes) 000837 1763346682.233532 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1848d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1848d): Mirror 2: 0xcbe941f2 (stale) Mirror 3: 0xcbe941f2 (stale) CRC-32 checksum value for chunk [0, 0x1848d): Mirror 1: 0xcbe941f2 Mirror 4: 0xcbe941f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1848d: skipped 000838 1763346683.017414 trunc from 0x01848d to 0x03dfc8 (0x25b3c bytes) 000841 1763346683.157261 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 3: 0x6d385242 (stale) Mirror 4: 0x6d385242 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 3: 0x6d385242 (stale) Mirror 4: 0x6d385242 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfc9: skipped 000844 1763346683.600780 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 3: 0x6d385242 (stale) Mirror 4: 0x6d385242 (stale) CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 1: 0x6d385242 Mirror 5: 0x6d385242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfc9: skipped 000845 1763346684.900721 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 4: 0x6d385242 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 4: 0x6d385242 (stale) CRC-32 checksum value for chunk [0, 0x3dfc9): Mirror 1: 0x6d385242 Mirror 5: 0x6d385242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfc9: skipped 000847 1763346685.581771 trunc from 0x03dfc9 to 0x01c8dc (0x216ec bytes) 000848 1763346685.688223 mapwrite 0x021089 thru 0x0234b2 (0x242a bytes) 000849 1763346685.804425 mapwrite 0x0244ea thru 0x02ef5f (0xaa76 bytes) 000850 1763346685.915344 write 0x008ca9 thru 0x01028e (0x75e6 bytes) 000851 1763346685.942728 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 4: 0x6f5bc719 (stale) Mirror 5: 0x6f5bc719 (stale) CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 1: 0xd74f8474 Mirror 6: 0xd74f8474 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef60: skipped 000853 1763346687.209248 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 4: 0x6f5bc719 (stale) Mirror 5: 0x6f5bc719 (stale) CRC-32 checksum value for chunk [0, 0x2ef60): Mirror 1: 0xd74f8474 Mirror 6: 0xd74f8474 Mirror 7: 0xd74f8474 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef60: skipped 000856 1763346688.540273 MIRROR_RESYNC 5 mirrors 000857 1763346689.948375 write 0x02b7ea thru 0x03b6d2 (0xfee9 bytes) 000859 1763346690.065336 trunc from 0x03b6d3 to 0x02ba75 (0xfc5d bytes) 000860 1763346690.111271 mapwrite 0x035c7c thru 0x036d8d (0x1112 bytes) 000861 1763346690.259647 write 0x01ab15 thru 0x01f327 (0x4813 bytes) 000863 1763346690.281604 trunc from 0x036d8e to 0x0383e2 (0x1655 bytes) 000866 1763346690.318570 write 0x000141 thru 0x008d79 (0x8c39 bytes) 000869 1763346690.343222 read 0x035039 thru 0x035b61 (0xb29 bytes) 000870 1763346690.351503 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x383e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383e3): Mirror 4: 0x68337339 (stale) Mirror 5: 0x68337339 (stale) Mirror 6: 0x68337339 (stale) Mirror 7: 0x68337339 (stale) CRC-32 checksum value for chunk [0, 0x383e3): Mirror 1: 0x96fe1aa1 Mirror 8: 0x96fe1aa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383e3: skipped 000871 1763346691.239551 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x383e3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383e3): Mirror 4: 0x68337339 (stale) Mirror 5: 0x68337339 (stale) Mirror 6: 0x68337339 (stale) Mirror 7: 0x68337339 (stale) CRC-32 checksum value for chunk [0, 0x383e3): Mirror 1: 0x96fe1aa1 Mirror 8: 0x96fe1aa1 Mirror 9: 0x96fe1aa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383e3: skipped 000872 1763346692.476755 read 0x020b67 thru 0x0211b0 (0x64a bytes) 000875 1763346692.482914 read 0x02a66d thru 0x03677c (0xc110 bytes) 000876 1763346692.506657 trunc from 0x0383e3 to 0x017188 (0x2125a bytes) 000879 1763346692.690826 trunc from 0x017189 to 0x01e78f (0x7607 bytes) 000882 1763346692.729763 read 0x00962f thru 0x011a88 (0x845a bytes) 000884 1763346692.752000 read 0x018e6d thru 0x01e78f (0x5923 bytes) 000885 1763346692.754199 mapwrite 0x01f765 thru 0x028a03 (0x929f bytes) 000888 1763346692.905988 mapwrite 0x009085 thru 0x00e516 (0x5492 bytes) 000889 1763346693.000148 trunc from 0x028a04 to 0x007637 (0x213cc bytes) 000893 1763346693.052514 mapread 0x002f05 thru 0x007637 (0x4733 bytes) 000896 1763346693.079202 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x7638): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7638): Mirror 4: 0x3a06e651 (stale) Mirror 5: 0x3a06e651 (stale) Mirror 6: 0x3a06e651 (stale) Mirror 7: 0x3a06e651 (stale) Mirror 8: 0xfcc5b3f0 (stale) Mirror 9: 0xfcc5b3f0 (stale) CRC-32 checksum value for chunk [0, 0x7638): Mirror 1: 0xfcc5b3f0 Mirror 10: 0xfcc5b3f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7638: skipped 000899 1763346694.542270 read 0x0066c9 thru 0x007637 (0xf6f bytes) 000901 1763346694.567352 mapwrite 0x036b8d thru 0x03ffff (0x9473 bytes) 000902 1763346694.877187 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf5d8b149 (stale) Mirror 5: 0xf5d8b149 (stale) Mirror 6: 0xf5d8b149 (stale) Mirror 7: 0xf5d8b149 (stale) Mirror 8: 0x8bc36975 (stale) Mirror 9: 0x8bc36975 (stale) Mirror 10: 0xcf1fdfc0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x81b30e4a Mirror 11: 0x81b30e4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000903 1763346696.582628 trunc from 0x040000 to 0x033882 (0xc77d bytes) 000906 1763346696.691440 trunc from 0x033883 to 0x02a515 (0x936d bytes) 000908 1763346696.727757 mapwrite 0x0376dd thru 0x03bcf2 (0x4616 bytes) 000910 1763346696.914743 trunc from 0x03bcf3 to 0x0351ac (0x6b46 bytes) 000911 1763346696.975517 mapwrite 0x021adb thru 0x025a33 (0x3f59 bytes) 000912 1763346697.124362 read 0x01f0aa thru 0x02865d (0x95b4 bytes) 000913 1763346697.125103 write 0x026134 thru 0x033466 (0xd333 bytes) 000915 1763346697.127954 read 0x006a4f thru 0x010cd9 (0xa28b bytes) 000917 1763346697.176285 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: 0xaa2c0a1d (stale) Mirror 5: 0xaa2c0a1d (stale) Mirror 6: 0xaa2c0a1d (stale) Mirror 7: 0xaa2c0a1d (stale) Mirror 9: 0x29338fae (stale) Mirror 10: 0xbbedd3ac (stale) Mirror 11: 0xbbedd3ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: 0xaa2c0a1d (stale) Mirror 5: 0xaa2c0a1d (stale) Mirror 6: 0xaa2c0a1d (stale) Mirror 7: 0xaa2c0a1d (stale) Mirror 9: 0x29338fae (stale) Mirror 10: 0xbbedd3ac (stale) Mirror 11: 0xbbedd3ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351ad: skipped 000919 1763346698.177504 write 0x020c2f thru 0x022c8e (0x2060 bytes) 000920 1763346698.201500 mapwrite 0x01d13e thru 0x0270c0 (0x9f83 bytes) 000921 1763346698.298548 read 0x0028e0 thru 0x011322 (0xea43 bytes) 000924 1763346698.313350 mapwrite 0x0122ed thru 0x01a67e (0x8392 bytes) 000925 1763346698.373165 read 0x027104 thru 0x0293af (0x22ac bytes) 000926 1763346698.376140 mapwrite 0x0082e1 thru 0x013ac5 (0xb7e5 bytes) 000929 1763346698.459599 mapread 0x010262 thru 0x01294f (0x26ee bytes) 000930 1763346698.461393 mapread 0x01ccd7 thru 0x028082 (0xb3ac bytes) 000931 1763346698.469543 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: 0xaa2c0a1d (stale) Mirror 5: 0xaa2c0a1d (stale) Mirror 6: 0xaa2c0a1d (stale) Mirror 7: 0xaa2c0a1d (stale) Mirror 9: 0x29338fae (stale) Mirror 10: 0xbbedd3ac (stale) Mirror 11: 0xbbedd3ac (stale) CRC-32 checksum value for chunk [0, 0x351ad): Mirror 1: 0x1931458 Mirror 12: 0x1931458 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351ad: skipped 000932 1763346700.203759 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: 0xaa2c0a1d (stale) Mirror 5: 0xaa2c0a1d (stale) Mirror 6: 0xaa2c0a1d (stale) Mirror 7: 0xaa2c0a1d (stale) Mirror 9: 0x29338fae (stale) Mirror 11: 0xbbedd3ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351ad): Mirror 4: 0xaa2c0a1d (stale) Mirror 5: 0xaa2c0a1d (stale) Mirror 6: 0xaa2c0a1d (stale) Mirror 7: 0xaa2c0a1d (stale) Mirror 9: 0x29338fae (stale) Mirror 11: 0xbbedd3ac (stale) CRC-32 checksum value for chunk [0, 0x351ad): Mirror 1: 0x1931458 Mirror 12: 0x1931458 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351ad: skipped 000934 1763346701.549128 trunc from 0x0351ad to 0x029b42 (0xb66a bytes) 000936 1763346701.702726 read 0x014aaf thru 0x02465b (0xfbad bytes) 000938 1763346701.718998 write 0x007a59 thru 0x00cfa1 (0x5549 bytes) 000939 1763346701.742031 read 0x022102 thru 0x0240da (0x1fd9 bytes) 000940 1763346701.760913 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29b43): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b43): Mirror 4: 0x17d37c45 (stale) Mirror 5: 0x17d37c45 (stale) Mirror 6: 0x17d37c45 (stale) Mirror 7: 0x17d37c45 (stale) Mirror 9: 0x4a7d63a8 (stale) Mirror 11: 0x11edaea8 (stale) Mirror 12: 0xed9b8072 (stale) CRC-32 checksum value for chunk [0, 0x29b43): Mirror 1: 0x8b97f8b7 Mirror 13: 0x8b97f8b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b43: skipped 000941 1763346703.180227 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b43): Mirror 4: 0x17d37c45 (stale) Mirror 5: 0x17d37c45 (stale) Mirror 6: 0x17d37c45 (stale) Mirror 7: 0x17d37c45 (stale) Mirror 9: 0x4a7d63a8 (stale) Mirror 11: 0x11edaea8 (stale) Mirror 12: 0xed9b8072 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b43): Mirror 4: 0x17d37c45 (stale) Mirror 5: 0x17d37c45 (stale) Mirror 6: 0x17d37c45 (stale) Mirror 7: 0x17d37c45 (stale) Mirror 9: 0x4a7d63a8 (stale) Mirror 11: 0x11edaea8 (stale) Mirror 12: 0xed9b8072 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b43: skipped 000942 1763346703.860541 write 0x02413b thru 0x02b45d (0x7323 bytes) 000943 1763346703.923264 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b45e): Mirror 4: 0xc84e8760 (stale) Mirror 5: 0xc84e8760 (stale) Mirror 6: 0xc84e8760 (stale) Mirror 7: 0xc84e8760 (stale) Mirror 9: 0xcfc87d75 (stale) Mirror 12: 0xe74a5c68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b45e): Mirror 4: 0xc84e8760 (stale) Mirror 5: 0xc84e8760 (stale) Mirror 6: 0xc84e8760 (stale) Mirror 7: 0xc84e8760 (stale) Mirror 9: 0xcfc87d75 (stale) Mirror 12: 0xe74a5c68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b45e: skipped 000944 1763346704.390433 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b45e): Mirror 5: 0xc84e8760 (stale) Mirror 6: 0xc84e8760 (stale) Mirror 7: 0xc84e8760 (stale) Mirror 9: 0xcfc87d75 (stale) Mirror 12: 0xe74a5c68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b45e): Mirror 5: 0xc84e8760 (stale) Mirror 6: 0xc84e8760 (stale) Mirror 7: 0xc84e8760 (stale) Mirror 9: 0xcfc87d75 (stale) Mirror 12: 0xe74a5c68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b45e: skipped 000945 1763346704.806990 mapread 0x008aa9 thru 0x00aa7f (0x1fd7 bytes) 000946 1763346704.844763 read 0x011bcc thru 0x01e27e (0xc6b3 bytes) 000947 1763346704.858722 trunc from 0x02b45e to 0x0206e6 (0xad77 bytes) 000951 1763346704.889766 trunc from 0x0206e7 to 0x01e1e2 (0x2504 bytes) 000953 1763346704.916349 read 0x00fa2a thru 0x01c945 (0xcf1c bytes) 000955 1763346704.928042 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e1e3): Mirror 5: 0x34baa98b (stale) Mirror 7: 0x34baa98b (stale) Mirror 9: 0xae57bc60 (stale) Mirror 12: 0xa172922c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e1e3): Mirror 5: 0x34baa98b (stale) Mirror 7: 0x34baa98b (stale) Mirror 9: 0xae57bc60 (stale) Mirror 12: 0xa172922c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e1e3: skipped 000958 1763346705.276554 trunc from 0x01e1e3 to 0x0268d3 (0x86f1 bytes) 000967 1763346705.303388 mapwrite 0x01108c thru 0x01eea9 (0xde1e bytes) 000968 1763346705.502030 mapwrite 0x016240 thru 0x0249ab (0xe76c bytes) 000971 1763346705.621122 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x268d4): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268d4): Mirror 5: 0x5aff2763 (stale) Mirror 7: 0x5aff2763 (stale) Mirror 9: 0x4695a668 (stale) Mirror 12: 0xf30178a1 (stale) CRC-32 checksum value for chunk [0, 0x268d4): Mirror 1: 0x5340664a Mirror 13: 0x5340664a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268d4: skipped 000973 1763346706.546287 mapwrite 0x0171e0 thru 0x01d229 (0x604a bytes) 000976 1763346706.809837 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x268d4): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268d4): Mirror 5: 0x5aff2763 (stale) Mirror 7: 0x5aff2763 (stale) Mirror 9: 0x4695a668 (stale) Mirror 12: 0xf30178a1 (stale) Mirror 13: 0x5340664a (stale) CRC-32 checksum value for chunk [0, 0x268d4): Mirror 1: 0x1bad1bd7 Mirror 14: 0x1bad1bd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268d4: skipped 000977 1763346707.859001 read 0x013760 thru 0x017fe6 (0x4887 bytes) 000979 1763346707.884248 read 0x01604d thru 0x01acc8 (0x4c7c bytes) 000980 1763346707.899763 read 0x01881d thru 0x021fb5 (0x9799 bytes) 000981 1763346707.911730 read 0x00caa2 thru 0x016f02 (0xa461 bytes) 000982 1763346707.919367 trunc from 0x0268d4 to 0x027824 (0xf51 bytes) 000985 1763346708.038473 write 0x03b800 thru 0x03ffff (0x4800 bytes) 000988 1763346708.040540 read 0x01525e thru 0x016821 (0x15c4 bytes) 000990 1763346708.086934 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf5d8b149 (stale) Mirror 7: 0xf5d8b149 (stale) Mirror 12: 0x1d1d86ef (stale) Mirror 13: 0xc1daced (stale) Mirror 14: 0x4c5b0577 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf5d8b149 (stale) Mirror 7: 0xf5d8b149 (stale) Mirror 12: 0x1d1d86ef (stale) Mirror 13: 0xc1daced (stale) Mirror 14: 0x4c5b0577 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1763346708.527384 mapread 0x00fe9c thru 0x01858b (0x86f0 bytes) 000992 1763346708.563435 mapwrite 0x0227fb thru 0x02821a (0x5a20 bytes) 000997 1763346708.660692 MIRROR_RESYNC 6 mirrors 000999 1763346709.929954 write 0x00de6a thru 0x016e04 (0x8f9b bytes) All operations completed A-OK! PASS 70a (83s) == sanity-flr test 71: check mirror extend parent fid ==== 21:31:54 (1763346714) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 21:31:58 (1763346718) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2141344 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 510176 .. failed Extending file size to 6188512 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6480032 .. Extending file size to 8840448 .. Extending file size to 8762240 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9067552 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9160672 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8906016 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 54831 54832 54833 54837 54839 done 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:33:19 (1763346799) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 6678976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7678720 .. Extending file size to 7930912 .. Extending file size to 7628256 .. Extending file size to 8160992 .. Extending file size to 8482816 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8325984 .. Extending file size to 8906400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8769440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9049248 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56762 56766 56768 56769 56770 failed 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:34:36 (1763346876) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175539 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.463676 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb5e6ad8f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:34:58 (1763346898) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0955373 s, 22.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:35:05 (1763346905) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x27e:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:35:13 (1763346913) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233496 s, 9.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:35:22 (1763346922) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25203 s, 8.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x282:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:35:31 (1763346931) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30317 s, 6.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:36:05 (1763346965) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962695 s, 10.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:37:07 (1763347027) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292886 s, 7.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:38:07 (1763347087) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. Creating new pool test_204f oleg205-server: Pool lustre.test_204f created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.297713 s, 7.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg205-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:39:30 (1763347170) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:39:38 (1763347178) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:39:45 (1763347185) oleg205-server: Pool lustre.test_206 created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg205-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:40:08 (1763347208) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632959 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458691 s, 22.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x83243a55 Mirror 3: 0x83243a55 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:40:18 (1763347218) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:40:22 (1763347222) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:40:26 (1763347226) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.104921 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1970000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2020000 Read this (4 bytes): BBBB 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:41:05 (1763347265) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0269524 s, 304 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:41:13 (1763347273) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671602 s, 15.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x220 lmm_fid: [0x200000404:0x220:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x220 lmm_fid: [0x200000404:0x220:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834464 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 21:41:14.000000000 -0500 Modify: 2025-11-16 21:41:14.000000000 -0500 Change: 2025-11-16 21:41:16.000000000 -0500 Birth: 2025-11-16 21:41:14.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:41:21 (1763347281) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:41:25 (1763347285) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.959547 s, 42.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0741774 s, 55.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 21:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x222:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 21:41 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:41:34 (1763347294) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:41:42 (1763347302) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1596 sec =========== 21:41:53 (1763347313) === sanity-flr: start cleanup 21:41:55 (1763347315) === === sanity-flr: finish cleanup 21:42:02 (1763347322) ===