-----============= acceptance-small: sanity-flr ============----- Wed Oct 22 09:53:50 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg305-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:53:57 (1761141237) === oleg305-client.virtnet: executing check_config_client /mnt/lustre oleg305-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg305-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff944f4a20d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff944f4a20d000.idle_timeout=debug disable quota as required oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:54:10 (1761141250) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7539] == sanity-flr test 0a: lfs mirror create with -N option == 09:54:11 (1761141251) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:54:17 (1761141257) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:54:18 (1761141258) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:54:20 (1761141260) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:54:30 (1761141270) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:54:31 (1761141271) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:54:33 (1761141273) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff944f4a20d000.stats=clear osc.lustre-OST0001-osc-ffff944f4a20d000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:54:36 (1761141276) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:54:42 (1761141282) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:54:45 (1761141285) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:54:47 (1761141287) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 09:54:51 (1761141291) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:54:55 (1761141295) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:54:56 (1761141296) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:55:00 (1761141300) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:55:04 (1761141304) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:55:08 (1761141308) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144212 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747002 s, 14.0 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:55:12 (1761141312) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.102635 s, 91.9 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:55:15 (1761141315) Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:55:48 (1761141348) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.157909 s, 46.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:56:02 (1761141362) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:56:27 (1761141387) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37802 s, 38.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 22 09:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.273s user 0m0.000s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.299s user 0m0.000s sys 0m0.159s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg305-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg305-server oleg305-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.332s user 0m0.002s sys 0m0.165s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg305-server: oleg305-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg305-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff944f4a20d000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:56:56 (1761141416) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:56:57 (1761141417) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:56:58 (1761141418) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:56:59 (1761141419) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 09:57:02 (1761141422) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.17392 s, 72.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:57:07 (1761141427) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110506 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136679 s, 76.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:57:10 (1761141430) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:57:13 (1761141433) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000728551 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000687538 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000789841 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000740343 s, 1.4 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 09:57:17 (1761141437) create test files with size 15729002 .. 9041d6d16851577794a17e5510c54972 - d1f8e03692754888f6d1fcee6f641348 - 1cfe84adc624411df0b3ebd561317fe6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 09:57:25 (1761141445) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:57:31 (1761141451) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:57:34 (1761141454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484763 s, 21.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139934 s, 74.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:57:38 (1761141458) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0524179 s, 40.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.124199 s, 33.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0752692 s, 27.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 09:57:43 (1761141463) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:57:44 (1761141464) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:57:45 (1761141465) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123768 s, 84.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 09:57:49 (1761141469) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:57:50 (1761141470) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:57:54 (1761141474) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.124088 s, 84.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:58:13 (1761141493) llite.lustre-ffff944f4a20f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.134865 s, 77.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg305-client.virtnet /mnt/lustre (opts:) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff944f47c37800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:58:18 (1761141498) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:58:33 (1761141513) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:58:46 (1761141526) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:58:47 (1761141527) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0983817 s, 32.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0723705 s, 43.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 5679aef0c3c072b798b8159ba0a52936 - mirror 1 checksum: cb666bda95b2a3e9e79179dd1937fae8 - mirror 2 checksum: 5679aef0c3c072b798b8159ba0a52936 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Oct 22 09:58:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Oct 22 09:58:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5679aef0c3c072b798b8159ba0a52936 - mirror 1 checksum: cb666bda95b2a3e9e79179dd1937fae8 - mirror 2 checksum: cb666bda95b2a3e9e79179dd1937fae8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Oct 22 09:58:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Oct 22 09:58:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:58:52 (1761141532) Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 09:58:54 (1761141534) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 09:59:09 (1761141549) targets are mounted 09:59:09 (1761141549) facet_failover done oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134536 s, 77.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (23s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:59:15 (1761141555) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:59:16 (1761141556) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:59:18 (1761141558) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00254463 s, 6.4 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:59:20 (1761141560) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:59:21 (1761141561) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:59:22 (1761141562) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:59:28 (1761141568) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:59:48 (1761141588) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:00:06 (1761141606) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:00:24 (1761141624) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 10:00:26 (1761141626) mirror create pid 52180 mirror split pid 52181 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52180 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52181 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 10:01:29 (1761141689) Seed set to 1689 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x28f5d 000003 1761141689.921910 trunc from 00000000 to 0x028f5c (0x28f5d bytes) 000004 1761141689.927434 trunc from 0x028f5d to 0x01f0f3 (0x9e69 bytes) 000011 1761141689.935245 mapread 0x00a99b thru 0x00e08b (0x36f1 bytes) 000013 1761141689.941736 mapread 0x0161cb thru 0x01f0f3 (0x8f29 bytes) 000017 1761141689.947602 trunc from 0x01f0f4 to 0x0263af (0x72bc bytes) 000018 1761141689.952735 mapwrite 0x020b22 thru 0x024439 (0x3918 bytes) truncating to largest ever: 0x2b8e0 000019 1761141689.975369 trunc from 0x0263b0 to 0x02b8df (0x5530 bytes) 000021 1761141689.980903 read 0x00618a thru 0x011a97 (0xb90e bytes) 000022 1761141689.981482 mapread 0x017a39 thru 0x025014 (0xd5dc bytes) 000023 1761141689.982356 trunc from 0x02b8e0 to 0x0152b7 (0x16628 bytes) 000024 1761141689.988084 mapread 0x00e422 thru 0x01360f (0x51ee bytes) 000025 1761141689.988924 mapwrite 0x00f6c9 thru 0x018cab (0x95e3 bytes) 000032 1761141690.017707 mapread 0x006e5b thru 0x011f28 (0xb0ce bytes) 000035 1761141690.019493 write 0x0150f5 thru 0x01ec65 (0x9b71 bytes) 000036 1761141690.020530 mapread 0x014457 thru 0x0159fa (0x15a4 bytes) 000037 1761141690.021225 mapread 0x002c81 thru 0x0065d3 (0x3953 bytes) 000038 1761141690.024261 mapwrite 0x01df4b thru 0x0205fc (0x26b2 bytes) 000039 1761141690.045102 mapread 0x01817e thru 0x019549 (0x13cc bytes) 000040 1761141690.045908 mapread 0x007034 thru 0x01377a (0xc747 bytes) 000041 1761141690.046698 mapread 0x000b86 thru 0x006e52 (0x62cd bytes) 000045 1761141690.051608 MIRROR_EXTEND to 2 mirrors 000046 1761141690.220628 mapwrite 0x009885 thru 0x012eac (0x9628 bytes) 000047 1761141690.284468 write 0x02801b thru 0x02e5f6 (0x65dc bytes) 000055 1761141690.288495 trunc from 0x02e5f7 to 0x024b95 (0x9a61 bytes) 000056 1761141690.305243 mapwrite 0x0206cf thru 0x02e34d (0xdc7f bytes) 000058 1761141690.342369 mapwrite 0x008ec0 thru 0x00fa35 (0x6b76 bytes) 000059 1761141690.359205 read 0x010eb8 thru 0x01300b (0x2154 bytes) 000061 1761141690.362140 write 0x018274 thru 0x0206be (0x844b bytes) 000063 1761141690.366057 mapwrite 0x01db0e thru 0x02bc6a (0xe15d bytes) 000065 1761141690.385026 MIRROR_EXTEND to 3 mirrors 000066 1761141690.543373 write 0x021d3f thru 0x02c9ec (0xacae bytes) 000067 1761141690.568122 write 0x035b74 thru 0x03cd1e (0x71ab bytes) 000068 1761141690.569170 write 0x020f1b thru 0x022b1f (0x1c05 bytes) 000069 1761141690.572658 mapwrite 0x02ebd5 thru 0x03371e (0x4b4a bytes) 000070 1761141690.593213 mapread 0x009ad0 thru 0x009eb8 (0x3e9 bytes) 000072 1761141690.597500 read 0x00bc52 thru 0x01a789 (0xeb38 bytes) 000073 1761141690.600801 mapwrite 0x03d49c thru 0x03ffff (0x2b64 bytes) 000074 1761141690.627058 write 0x0168f2 thru 0x01919e (0x28ad bytes) 000075 1761141690.629707 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000079 1761141690.694660 mapread 0x0187ce thru 0x01fef6 (0x7729 bytes) 000080 1761141690.700171 mapread 0x00e1c8 thru 0x01d80b (0xf644 bytes) 000081 1761141690.703519 read 0x03db7c thru 0x03ffff (0x2484 bytes) 000083 1761141690.706146 mapwrite 0x037ca7 thru 0x03ffff (0x8359 bytes) 000086 1761141690.724694 MIRROR_RESYNC 2 mirrors 000090 1761141690.891846 mapwrite 0x02626b thru 0x035f21 (0xfcb7 bytes) 000093 1761141690.957817 read 0x012c96 thru 0x02251e (0xf889 bytes) 000094 1761141690.960901 mapwrite 0x02a969 thru 0x02b624 (0xcbc bytes) 000095 1761141690.973378 mapwrite 0x016a0e thru 0x025524 (0xeb17 bytes) 000096 1761141690.991585 trunc from 0x040000 to 0x023072 (0x1cf8d bytes) 000098 1761141691.005142 read 0x00db4a thru 0x01af92 (0xd449 bytes) 000102 1761141691.009718 trunc from 0x023073 to 0x014488 (0xebea bytes) 000104 1761141691.016329 mapread 0x01065a thru 0x011850 (0x11f7 bytes) 000107 1761141691.017958 read 0x004f80 thru 0x009b8d (0x4c0e bytes) 000109 1761141691.021515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000110 1761141691.067984 write 0x01da8c thru 0x02a9d4 (0xcf49 bytes) 000111 1761141691.069009 write 0x004bcd thru 0x00bb13 (0x6f47 bytes) 000112 1761141691.072586 mapwrite 0x02fa82 thru 0x03ede6 (0xf365 bytes) 000113 1761141691.101103 mapwrite 0x00c7ee thru 0x016e35 (0xa648 bytes) 000114 1761141691.125608 mapwrite 0x00f1ff thru 0x01f11b (0xff1d bytes) 000115 1761141691.148825 mapread 0x03af4b thru 0x03ede6 (0x3e9c bytes) 000116 1761141691.149549 mapread 0x0258dd thru 0x02e9e6 (0x910a bytes) truncating to largest ever: 0x3d94a 000118 1761141691.150652 trunc from 0x03ede7 to 0x03d949 (0x149d bytes) 000119 1761141691.159218 read 0x026192 thru 0x034243 (0xe0b2 bytes) 000121 1761141691.162241 MIRROR_EXTEND to 2 mirrors 000125 1761141691.341132 mapread 0x036713 thru 0x0381de (0x1acc bytes) 000128 1761141691.351711 write 0x0276d8 thru 0x02da52 (0x637b bytes) 000131 1761141691.375920 mapwrite 0x0157e2 thru 0x018c14 (0x3433 bytes) 000132 1761141691.394685 read 0x008a00 thru 0x0153a6 (0xc9a7 bytes) 000133 1761141691.399586 MIRROR_RESYNC 2 mirrors 000134 1761141691.587157 mapwrite 0x0081ae thru 0x01165f (0x94b2 bytes) 000140 1761141691.647450 mapread 0x01ae0f thru 0x01d94c (0x2b3e bytes) 000142 1761141691.652037 write 0x0283ce thru 0x02def3 (0x5b26 bytes) 000144 1761141691.657065 mapwrite 0x0202ab thru 0x02b90b (0xb661 bytes) 000145 1761141691.677155 MIRROR_RESYNC 2 mirrors 000146 1761141691.843945 mapwrite 0x00ab5a thru 0x0121c4 (0x766b bytes) 000147 1761141691.902645 MIRROR_EXTEND to 3 mirrors 000148 1761141692.097366 MIRROR_EXTEND to 4 mirrors 000149 1761141692.297717 mapread 0x02a0c2 thru 0x037483 (0xd3c2 bytes) 000150 1761141692.307515 trunc from 0x03d94a to 0x012c04 (0x2ad45 bytes) 000152 1761141692.329442 mapread 0x009976 thru 0x00c220 (0x28ab bytes) 000154 1761141692.333657 mapwrite 0x00ed79 thru 0x012e56 (0x40de bytes) 000161 1761141692.360085 mapread 0x0091c5 thru 0x00b7dd (0x2619 bytes) 000163 1761141692.361260 read 0x007777 thru 0x00a75e (0x2fe8 bytes) 000164 1761141692.361826 mapwrite 0x00b161 thru 0x019660 (0xe500 bytes) 000165 1761141692.392426 write 0x012100 thru 0x01d3d8 (0xb2d9 bytes) 000166 1761141692.393596 mapread 0x00e09e thru 0x018bcb (0xab2e bytes) 000167 1761141692.394400 mapwrite 0x0345c2 thru 0x03ffff (0xba3e bytes) 000168 1761141692.425011 write 0x01bde9 thru 0x02a8ee (0xeb06 bytes) 000171 1761141692.428995 trunc from 0x040000 to 0x039549 (0x6ab6 bytes) 000172 1761141692.441426 read 0x00689c thru 0x00903d (0x27a2 bytes) 000173 1761141692.442034 mapread 0x022b3e thru 0x025192 (0x2655 bytes) 000175 1761141692.443111 write 0x036720 thru 0x03ffff (0x98e0 bytes) 000176 1761141692.444293 write 0x0035ec thru 0x005b00 (0x2515 bytes) 000178 1761141692.445425 read 0x03e70d thru 0x03ffff (0x18f3 bytes) 000179 1761141692.445912 trunc from 0x040000 to 0x006e74 (0x3918b bytes) 000180 1761141692.454542 write 0x00206e thru 0x002ec3 (0xe56 bytes) 000181 1761141692.455430 mapread 0x003ad2 thru 0x006e74 (0x33a3 bytes) 000184 1761141692.457068 trunc from 0x006e75 to 0x02d94e (0x26ada bytes) 000188 1761141692.464903 read 0x02b1b5 thru 0x02d94e (0x279a bytes) 000189 1761141692.470320 mapread 0x018dea thru 0x0224be (0x96d5 bytes) 000190 1761141692.471222 mapread 0x003a4c thru 0x00e7b4 (0xad69 bytes) 000192 1761141692.472300 write 0x03c2b2 thru 0x03e290 (0x1fdf bytes) 000193 1761141692.475369 MIRROR_RESYNC 4 mirrors 000194 1761141692.722051 mapread 0x03b69a thru 0x03dfde (0x2945 bytes) 000195 1761141692.731521 read 0x032f71 thru 0x03e290 (0xb320 bytes) 000196 1761141692.734023 write 0x03edf5 thru 0x03ef0a (0x116 bytes) 000198 1761141692.758984 trunc from 0x03ef0b to 0x007dcc (0x3713e bytes) 000199 1761141692.772576 write 0x006177 thru 0x00bf53 (0x5ddd bytes) 000200 1761141692.776158 mapread 0x0073cd thru 0x00bf53 (0x4b87 bytes) 000202 1761141692.777287 mapread 0x002d02 thru 0x004955 (0x1c54 bytes) 000203 1761141692.779966 MIRROR_RESYNC 4 mirrors 000207 1761141693.014844 write 0x02e578 thru 0x032d3d (0x47c6 bytes) 000208 1761141693.035473 read 0x0168a3 thru 0x017395 (0xaf3 bytes) 000209 1761141693.041143 trunc from 0x032d3e to 0x015001 (0x1dd3c bytes) 000210 1761141693.060923 MIRROR_EXTEND to 5 mirrors 000212 1761141693.249030 mapwrite 0x03a716 thru 0x03ffff (0x58ea bytes) 000213 1761141693.315188 read 0x00b74a thru 0x017509 (0xbdc0 bytes) 000214 1761141693.320837 write 0x0104a2 thru 0x0151b6 (0x4d15 bytes) 000215 1761141693.321808 write 0x007aca thru 0x0139f5 (0xbf2c bytes) 000218 1761141693.327349 write 0x0103ea thru 0x0115ab (0x11c2 bytes) 000221 1761141693.329068 mapread 0x0292ef thru 0x02a3d4 (0x10e6 bytes) 000222 1761141693.335981 mapwrite 0x039291 thru 0x03fc32 (0x69a2 bytes) 000223 1761141693.363693 read 0x00548e thru 0x00faad (0xa620 bytes) 000224 1761141693.364308 read 0x022707 thru 0x02bd85 (0x967f bytes) 000225 1761141693.364811 write 0x02eaae thru 0x0331da (0x472d bytes) 000229 1761141693.373022 mapwrite 0x010bfd thru 0x01af94 (0xa398 bytes) 000231 1761141693.396584 read 0x024d46 thru 0x029869 (0x4b24 bytes) 000233 1761141693.397612 mapwrite 0x0205ca thru 0x02842e (0x7e65 bytes) 000235 1761141693.421774 mapwrite 0x02a342 thru 0x039cb2 (0xf971 bytes) 000238 1761141693.453701 mapread 0x027267 thru 0x033452 (0xc1ec bytes) 000240 1761141693.455004 trunc from 0x040000 to 0x025f06 (0x1a0f9 bytes) 000241 1761141693.465718 write 0x01b8f8 thru 0x028db4 (0xd4bd bytes) 000243 1761141693.467473 read 0x01f8a2 thru 0x028db4 (0x9513 bytes) 000246 1761141693.468871 read 0x0255a4 thru 0x028db4 (0x3811 bytes) 000248 1761141693.469803 trunc from 0x028db5 to 0x003a96 (0x2531e bytes) 000250 1761141693.476398 write 0x02f9c0 thru 0x034cfc (0x533d bytes) 000251 1761141693.477487 mapwrite 0x025409 thru 0x0286fc (0x32f4 bytes) 000252 1761141693.498270 read 0x003c66 thru 0x00b11b (0x74b6 bytes) 000253 1761141693.498775 mapread 0x00bcda thru 0x010297 (0x45be bytes) 000254 1761141693.499466 trunc from 0x034cfd to 0x0066d4 (0x2e628 bytes) 000255 1761141693.505861 mapread 0x005899 thru 0x0066d4 (0xe3c bytes) 000257 1761141693.507059 mapread 0x000baa thru 0x0066d4 (0x5b2b bytes) 000258 1761141693.507742 read 0x000ab5 thru 0x001bc2 (0x110e bytes) 000259 1761141693.508307 trunc from 0x0066d5 to 0x01bce6 (0x15612 bytes) 000263 1761141693.516735 MIRROR_EXTEND to 6 mirrors 000264 1761141693.700040 write 0x01af33 thru 0x0203f2 (0x54c0 bytes) 000266 1761141693.726094 trunc from 0x0203f3 to 0x00357a (0x1ce78 bytes) 000268 1761141693.739993 mapread 0x001bd5 thru 0x00357a (0x19a6 bytes) 000269 1761141693.744220 mapwrite 0x03db51 thru 0x03f0aa (0x155a bytes) 000271 1761141693.768040 trunc from 0x03f0ab to 0x028b69 (0x16541 bytes) 000277 1761141693.775486 read 0x008f88 thru 0x00f855 (0x68ce bytes) 000278 1761141693.779528 read 0x00ab5d thru 0x0147c5 (0x9c69 bytes) 000280 1761141693.785698 MIRROR_EXTEND to 7 mirrors 000281 1761141693.955304 write 0x020d54 thru 0x02c739 (0xb9e6 bytes) 000282 1761141693.987659 mapwrite 0x0144b2 thru 0x024067 (0xfbb6 bytes) 000283 1761141694.015287 trunc from 0x02c73a to 0x009eb9 (0x22880 bytes) 000285 1761141694.029974 mapread 0x00579a thru 0x009eb9 (0x4720 bytes) 000286 1761141694.036130 MIRROR_EXTEND to 8 mirrors 000289 1761141694.233501 mapwrite 0x01277e thru 0x01a155 (0x79d8 bytes) 000292 1761141694.290519 mapwrite 0x017a2e thru 0x01adad (0x3380 bytes) 000293 1761141694.309104 mapread 0x01a791 thru 0x01adad (0x61d bytes) 000294 1761141694.309813 mapread 0x0136c7 thru 0x01adad (0x76e7 bytes) 000296 1761141694.310593 mapwrite 0x001309 thru 0x00350e (0x2206 bytes) 000297 1761141694.326369 mapread 0x012bb2 thru 0x018cb1 (0x6100 bytes) 000298 1761141694.327093 read 0x003405 thru 0x00b25a (0x7e56 bytes) 000299 1761141694.327634 mapwrite 0x00b9bb thru 0x016636 (0xac7c bytes) 000300 1761141694.345601 MIRROR_EXTEND to 9 mirrors 000301 1761141694.528414 write 0x020931 thru 0x02d54e (0xcc1e bytes) 000304 1761141694.553752 mapread 0x01d711 thru 0x029a5d (0xc34d bytes) 000305 1761141694.560211 trunc from 0x02d54f to 0x02a2dd (0x3271 bytes) 000308 1761141694.574844 mapwrite 0x0009f6 thru 0x0035e8 (0x2bf3 bytes) 000309 1761141694.594443 trunc from 0x02a2de to 0x01abcd (0xf710 bytes) 000310 1761141694.600662 write 0x03fc51 thru 0x03ffff (0x3af bytes) 000311 1761141694.601610 mapread 0x00a014 thru 0x0143a9 (0xa396 bytes) 000312 1761141694.606317 mapwrite 0x0278c7 thru 0x033147 (0xb881 bytes) 000313 1761141694.630596 trunc from 0x040000 to 0x0073b0 (0x38c4f bytes) 000314 1761141694.638243 trunc from 0x0073b1 to 0x026b81 (0x1f7d1 bytes) 000316 1761141694.646120 read 0x02402e thru 0x026b81 (0x2b54 bytes) 000318 1761141694.650070 write 0x034b60 thru 0x03ffff (0xb4a0 bytes) 000319 1761141694.651456 mapwrite 0x03b638 thru 0x03e698 (0x3061 bytes) 000321 1761141694.671978 mapread 0x028e0d thru 0x030b98 (0x7d8c bytes) 000322 1761141694.677473 write 0x030cb3 thru 0x03e420 (0xd76e bytes) 000323 1761141694.678759 read 0x0180f8 thru 0x019453 (0x135c bytes) 000325 1761141694.684114 trunc from 0x040000 to 0x03aad7 (0x5528 bytes) 000328 1761141694.692006 read 0x005150 thru 0x0145fb (0xf4ac bytes) 000329 1761141694.694824 MIRROR_EXTEND to 10 mirrors 000330 1761141694.885872 MIRROR_RESYNC 10 mirrors 000331 1761141695.312519 trunc from 0x03aad8 to 0x009e7d (0x30c5a bytes) 000334 1761141695.347762 read 0x00784f thru 0x009e7d (0x262f bytes) 000335 1761141695.352093 trunc from 0x009e7e to 0x013c5f (0x9de2 bytes) 000336 1761141695.363526 MIRROR_EXTEND to 11 mirrors 000338 1761141695.548979 write 0x00c493 thru 0x019236 (0xcda4 bytes) 000339 1761141695.574128 mapread 0x002a94 thru 0x00d989 (0xaef6 bytes) 000340 1761141695.580399 write 0x00f503 thru 0x016794 (0x7292 bytes) 000341 1761141695.581137 trunc from 0x019237 to 0x019b3b (0x905 bytes) 000343 1761141695.593872 read 0x00c302 thru 0x015634 (0x9333 bytes) 000344 1761141695.594353 mapread 0x01151c thru 0x019b3b (0x8620 bytes) 000346 1761141695.597430 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000347 1761141695.704078 MIRROR_RESYNC 10 mirrors 000350 1761141696.177043 trunc from 0x019b3c to 0x005ff7 (0x13b44 bytes) 000351 1761141696.203613 write 0x02dbfe thru 0x03a2db (0xc6de bytes) 000352 1761141696.205136 trunc from 0x03a2dc to 0x00ccbf (0x2d61c bytes) 000356 1761141696.216173 write 0x01cc61 thru 0x02a0dc (0xd47c bytes) 000357 1761141696.217481 trunc from 0x02a0dd to 0x0331e6 (0x910a bytes) 000358 1761141696.227614 MIRROR_EXTEND to 11 mirrors 000359 1761141696.480644 mapwrite 0x03c82d thru 0x03ffff (0x37d3 bytes) 000360 1761141696.529221 mapread 0x036ece thru 0x03a715 (0x3848 bytes) 000362 1761141696.533415 trunc from 0x040000 to 0x019597 (0x26a68 bytes) 000363 1761141696.540051 mapread 0x00bd0f thru 0x019597 (0xd889 bytes) 000365 1761141696.546602 write 0x029c17 thru 0x02dbaf (0x3f99 bytes) 000366 1761141696.549043 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000367 1761141696.618174 mapwrite 0x03e191 thru 0x03ffff (0x1e6f bytes) 000368 1761141696.641259 mapwrite 0x027a17 thru 0x036954 (0xef3e bytes) 000369 1761141696.666542 read 0x023bae thru 0x02c349 (0x879c bytes) 000371 1761141696.667674 read 0x03c23e thru 0x03ffff (0x3dc2 bytes) 000372 1761141696.673522 write 0x00bb0c thru 0x010308 (0x47fd bytes) 000373 1761141696.676825 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000374 1761141696.753155 mapwrite 0x023905 thru 0x0313ef (0xdaeb bytes) 000376 1761141696.783122 read 0x0066c5 thru 0x0087a5 (0x20e1 bytes) 000378 1761141696.786542 write 0x0193d4 thru 0x0194b7 (0xe4 bytes) 000379 1761141696.790997 write 0x02e8db thru 0x03e7e2 (0xff08 bytes) 000382 1761141696.795355 mapwrite 0x0221b7 thru 0x02cd65 (0xabaf bytes) 000383 1761141696.818295 mapwrite 0x034a51 thru 0x03ffff (0xb5af bytes) 000384 1761141696.844103 mapwrite 0x03b45c thru 0x03ffff (0x4ba4 bytes) 000385 1761141696.865195 write 0x01aa3b thru 0x026e74 (0xc43a bytes) 000386 1761141696.866384 trunc from 0x040000 to 0x028367 (0x17c98 bytes) 000387 1761141696.873853 write 0x00c290 thru 0x01188b (0x55fc bytes) 000388 1761141696.879324 mapread 0x01fefb thru 0x020070 (0x176 bytes) 000390 1761141696.882984 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000392 1761141696.972423 mapread 0x014c65 thru 0x018800 (0x3b9c bytes) 000393 1761141696.980148 mapwrite 0x03e13f thru 0x03ffff (0x1ec1 bytes) 000394 1761141697.005279 mapread 0x001571 thru 0x00eea8 (0xd938 bytes) 000395 1761141697.010094 mapwrite 0x014128 thru 0x01cacd (0x89a6 bytes) 000397 1761141697.026890 write 0x021199 thru 0x0230ac (0x1f14 bytes) 000398 1761141697.030284 mapwrite 0x00d460 thru 0x01a067 (0xcc08 bytes) 000399 1761141697.049887 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000400 1761141697.121159 write 0x01b1b8 thru 0x01bf57 (0xda0 bytes) 000401 1761141697.123701 mapread 0x008981 thru 0x00ce4b (0x44cb bytes) 000402 1761141697.130245 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000403 1761141697.200742 read 0x0018f6 thru 0x0089b9 (0x70c4 bytes) 000405 1761141697.204964 trunc from 0x040000 to 0x02ccfb (0x13304 bytes) 000406 1761141697.210768 read 0x013a9d thru 0x01ba85 (0x7fe9 bytes) 000409 1761141697.216366 mapread 0x00db3a thru 0x0153b3 (0x787a bytes) 000410 1761141697.217273 trunc from 0x02ccfc to 0x016788 (0x16573 bytes) 000411 1761141697.223348 write 0x00967b thru 0x015769 (0xc0ef bytes) 000413 1761141697.225056 read 0x00027b thru 0x00abbb (0xa941 bytes) 000414 1761141697.225567 trunc from 0x016789 to 0x012017 (0x4771 bytes) 000415 1761141697.234021 MIRROR_EXTEND to 7 mirrors 000416 1761141697.429658 read 0x006d1f thru 0x012017 (0xb2f9 bytes) 000417 1761141697.435683 mapwrite 0x01460c thru 0x018f36 (0x492b bytes) 000419 1761141697.499662 trunc from 0x018f37 to 0x026e7c (0xdf46 bytes) 000427 1761141697.512938 mapwrite 0x034221 thru 0x034cc2 (0xaa2 bytes) 000428 1761141697.541358 read 0x007d69 thru 0x00f693 (0x792b bytes) 000431 1761141697.547525 trunc from 0x034cc3 to 0x00d58c (0x27736 bytes) 000433 1761141697.557570 MIRROR_RESYNC 7 mirrors 000434 1761141697.893098 read 0x0028a1 thru 0x00addc (0x853c bytes) 000437 1761141697.896247 trunc from 0x00d58d to 0x0058ba (0x7cd2 bytes) 000439 1761141697.925723 read 0x0018a4 thru 0x0058ba (0x4017 bytes) 000441 1761141697.931025 mapwrite 0x0139e8 thru 0x018640 (0x4c59 bytes) 000443 1761141697.962315 trunc from 0x018641 to 0x01bf67 (0x3927 bytes) 000444 1761141697.969183 mapwrite 0x03d69f thru 0x03ea8f (0x13f1 bytes) 000445 1761141697.996938 mapread 0x028bd1 thru 0x036775 (0xdba5 bytes) 000446 1761141697.997776 mapwrite 0x010db8 thru 0x01eb8a (0xddd3 bytes) 000447 1761141698.015961 mapwrite 0x00b70e thru 0x011558 (0x5e4b bytes) 000449 1761141698.033720 MIRROR_RESYNC 7 mirrors 000452 1761141698.345044 mapwrite 0x02f64b thru 0x035b7c (0x6532 bytes) truncating to largest ever: 0x3fe73 000455 1761141698.416183 trunc from 0x03ea90 to 0x03fe72 (0x13e3 bytes) 000457 1761141698.436664 read 0x01740b thru 0x01d002 (0x5bf8 bytes) 000458 1761141698.440764 read 0x03e835 thru 0x03fe72 (0x163e bytes) 000459 1761141698.447891 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000460 1761141698.533206 write 0x01fb3e thru 0x01fc1c (0xdf bytes) 000461 1761141698.536130 trunc from 0x03fe73 to 0x00798d (0x384e5 bytes) 000462 1761141698.542499 read 0x0015f1 thru 0x00798d (0x639d bytes) 000464 1761141698.549323 MIRROR_RESYNC 6 mirrors 000466 1761141698.873425 mapwrite 0x00da95 thru 0x012927 (0x4e93 bytes) 000468 1761141698.935657 mapread 0x00478a thru 0x00eb4f (0xa3c6 bytes) 000469 1761141698.941063 write 0x0307db thru 0x0377f9 (0x701f bytes) 000470 1761141698.942065 mapwrite 0x020b3d thru 0x025b36 (0x4ffa bytes) 000472 1761141698.968561 MIRROR_RESYNC 6 mirrors 000473 1761141699.262601 mapwrite 0x0345af thru 0x03ac38 (0x668a bytes) 000474 1761141699.309504 trunc from 0x03ac39 to 0x03cfa6 (0x236e bytes) 000475 1761141699.316917 trunc from 0x03cfa7 to 0x002e64 (0x3a142 bytes) 000476 1761141699.323100 read 0x000148 thru 0x002e64 (0x2d1d bytes) 000478 1761141699.326197 mapread 0x002ada thru 0x002e64 (0x38b bytes) 000479 1761141699.326914 write 0x026f96 thru 0x027c25 (0xc90 bytes) 000481 1761141699.328132 read 0x010db0 thru 0x01a3bd (0x960e bytes) 000483 1761141699.332666 read 0x0094a4 thru 0x01368c (0xa1e9 bytes) 000484 1761141699.333283 mapread 0x00e8c0 thru 0x019424 (0xab65 bytes) 000486 1761141699.334212 mapread 0x003e0e thru 0x005ff9 (0x21ec bytes) 000487 1761141699.334944 mapwrite 0x00718e thru 0x01630a (0xf17d bytes) 000488 1761141699.356244 mapwrite 0x002f97 thru 0x0123f8 (0xf462 bytes) 000490 1761141699.375888 mapwrite 0x00fa0e thru 0x0172c1 (0x78b4 bytes) 000491 1761141699.392597 trunc from 0x027c26 to 0x03cab4 (0x14e8f bytes) 000493 1761141699.400362 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000494 1761141699.468192 read 0x005ad5 thru 0x0078b3 (0x1ddf bytes) 000496 1761141699.470943 mapwrite 0x00c50a thru 0x00d497 (0xf8e bytes) 000497 1761141699.489359 trunc from 0x03cab5 to 0x00ab20 (0x31f94 bytes) 000498 1761141699.498543 MIRROR_RESYNC 5 mirrors 000499 1761141699.741312 mapread 0x004001 thru 0x009713 (0x5713 bytes) 000500 1761141699.746745 trunc from 0x00ab21 to 0x00b72d (0xc0d bytes) 000503 1761141699.773945 read 0x000234 thru 0x005e86 (0x5c53 bytes) 000504 1761141699.777618 read 0x0006ca thru 0x008d53 (0x868a bytes) 000505 1761141699.781197 trunc from 0x00b72e to 0x018c7b (0xd54e bytes) 000506 1761141699.787933 trunc from 0x018c7c to 0x0028be (0x163bd bytes) 000509 1761141699.794293 mapwrite 0x033acf thru 0x035b23 (0x2055 bytes) 000510 1761141699.821154 mapwrite 0x0097ea thru 0x00fdbd (0x65d4 bytes) 000511 1761141699.846369 MIRROR_RESYNC 5 mirrors 000513 1761141700.132169 mapwrite 0x019ce2 thru 0x01ba1c (0x1d3b bytes) 000515 1761141700.183924 write 0x03aef5 thru 0x03ffff (0x510b bytes) 000516 1761141700.184575 mapread 0x006604 thru 0x00c015 (0x5a12 bytes) 000517 1761141700.187977 mapread 0x01075b thru 0x016285 (0x5b2b bytes) 000519 1761141700.190657 trunc from 0x040000 to 0x03124c (0xedb3 bytes) 000524 1761141700.198179 read 0x028586 thru 0x02b291 (0x2d0c bytes) 000525 1761141700.200706 trunc from 0x03124d to 0x009d32 (0x2751a bytes) 000529 1761141700.207051 write 0x018a56 thru 0x0210d3 (0x867e bytes) 000530 1761141700.209710 MIRROR_EXTEND to 6 mirrors 000533 1761141700.388957 read 0x01f5f1 thru 0x0210d3 (0x1ae3 bytes) 000534 1761141700.392450 mapwrite 0x03f9b4 thru 0x03ff34 (0x581 bytes) 000535 1761141700.439725 mapwrite 0x01fcd9 thru 0x02f07c (0xf3a4 bytes) 000536 1761141700.472530 MIRROR_EXTEND to 7 mirrors 000537 1761141700.650813 mapwrite 0x034292 thru 0x03d6e6 (0x9455 bytes) 000539 1761141700.705761 trunc from 0x03ff35 to 0x02a40f (0x15b25 bytes) 000541 1761141700.716550 trunc from 0x02a410 to 0x020447 (0x9fc8 bytes) 000546 1761141700.724909 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000548 1761141700.793058 MIRROR_RESYNC 6 mirrors 000551 1761141701.050282 write 0x0315e8 thru 0x033bd1 (0x25ea bytes) 000554 1761141701.068521 trunc from 0x033bd2 to 0x01857c (0x1b655 bytes) 000555 1761141701.079522 mapread 0x00fdc7 thru 0x01857c (0x87b6 bytes) 000557 1761141701.086673 mapwrite 0x037282 thru 0x037d43 (0xac2 bytes) 000560 1761141701.109601 mapwrite 0x01a1e7 thru 0x01fa1d (0x5837 bytes) 000563 1761141701.126396 mapread 0x032890 thru 0x037d43 (0x54b4 bytes) 000565 1761141701.129646 MIRROR_EXTEND to 7 mirrors 000567 1761141701.303567 trunc from 0x037d44 to 0x0087a2 (0x2f5a1 bytes) 000568 1761141701.330453 trunc from 0x0087a3 to 0x026279 (0x1dad7 bytes) 000569 1761141701.336851 mapread 0x010e22 thru 0x0183b4 (0x7593 bytes) 000571 1761141701.344517 write 0x0365d1 thru 0x03ffff (0x9a2f bytes) 000579 1761141701.348222 write 0x01c731 thru 0x02526f (0x8b3f bytes) 000582 1761141701.355218 trunc from 0x040000 to 0x01b0f5 (0x24f0a bytes) 000583 1761141701.361733 mapread 0x0153f3 thru 0x01579c (0x3aa bytes) 000584 1761141701.364698 MIRROR_EXTEND to 8 mirrors 000585 1761141701.538839 read 0x008a8b thru 0x00d1ca (0x4740 bytes) 000586 1761141701.542535 trunc from 0x01b0f6 to 0x0343a6 (0x192b1 bytes) 000587 1761141701.570012 mapwrite 0x03ae62 thru 0x03ffff (0x519e bytes) 000588 1761141701.605222 mapwrite 0x0340f7 thru 0x0375fe (0x3508 bytes) 000590 1761141701.621549 mapread 0x03dfdb thru 0x03ffff (0x2025 bytes) 000595 1761141701.623467 mapread 0x02a600 thru 0x02cd7c (0x277d bytes) 000596 1761141701.627166 trunc from 0x040000 to 0x02572e (0x1a8d1 bytes) 000597 1761141701.632991 write 0x018a5b thru 0x025945 (0xceeb bytes) 000598 1761141701.636573 trunc from 0x025946 to 0x0242b4 (0x1691 bytes) 000599 1761141701.642145 write 0x025e56 thru 0x0276ea (0x1895 bytes) 000602 1761141701.643685 read 0x00a24c thru 0x0187bf (0xe574 bytes) 000606 1761141701.649085 mapwrite 0x0398ab thru 0x03ffff (0x6755 bytes) 000607 1761141701.674965 trunc from 0x040000 to 0x01cbed (0x23412 bytes) 000608 1761141701.680546 trunc from 0x01cbee to 0x038662 (0x1ba75 bytes) 000609 1761141701.685594 mapwrite 0x015634 thru 0x01c6ad (0x707a bytes) 000610 1761141701.700770 mapwrite 0x0262e8 thru 0x0317d4 (0xb4ed bytes) 000618 1761141701.736814 trunc from 0x038663 to 0x024bbe (0x13aa4 bytes) 000619 1761141701.742883 read 0x017222 thru 0x02226d (0xb04c bytes) 000620 1761141701.746241 trunc from 0x024bbf to 0x034b6a (0xffac bytes) 000621 1761141701.751353 mapwrite 0x012a8d thru 0x014dda (0x234e bytes) 000622 1761141701.768650 MIRROR_RESYNC 8 mirrors 000623 1761141702.117623 read 0x015401 thru 0x024325 (0xef25 bytes) 000625 1761141702.122762 MIRROR_RESYNC 8 mirrors 000626 1761141702.225299 read 0x0323b3 thru 0x034b6a (0x27b8 bytes) 000627 1761141702.227406 read 0x00ef19 thru 0x0163ec (0x74d4 bytes) 000632 1761141702.231562 write 0x013fd3 thru 0x015809 (0x1837 bytes) 000634 1761141702.252439 read 0x00d432 thru 0x01c8a6 (0xf475 bytes) 000636 1761141702.257142 mapwrite 0x025b4e thru 0x0345f6 (0xeaa9 bytes) 000637 1761141702.278681 read 0x019622 thru 0x01a324 (0xd03 bytes) 000639 1761141702.279711 read 0x0073b0 thru 0x016653 (0xf2a4 bytes) 000640 1761141702.280265 mapwrite 0x03ebf6 thru 0x03ffff (0x140a bytes) 000642 1761141702.308280 trunc from 0x040000 to 0x02abf5 (0x1540a bytes) 000644 1761141702.313686 read 0x01dc44 thru 0x02abf5 (0xcfb2 bytes) 000647 1761141702.316330 MIRROR_RESYNC 8 mirrors 000648 1761141702.672682 read 0x02ab4b thru 0x02abf5 (0xab bytes) 000652 1761141702.676914 read 0x0069f8 thru 0x00f845 (0x8e4e bytes) 000653 1761141702.682028 mapread 0x00dfdf thru 0x016ca9 (0x8ccb bytes) 000657 1761141702.683343 mapread 0x01472a thru 0x01a9d0 (0x62a7 bytes) 000658 1761141702.683865 trunc from 0x02abf6 to 0x01fc98 (0xaf5d bytes) 000659 1761141702.719169 trunc from 0x01fc99 to 0x018fb0 (0x6ce8 bytes) 000663 1761141702.728524 MIRROR_RESYNC 8 mirrors 000664 1761141703.075517 write 0x00d900 thru 0x00dfc7 (0x6c8 bytes) 000665 1761141703.094606 read 0x016db7 thru 0x018fb0 (0x21fa bytes) 000666 1761141703.096932 read 0x014075 thru 0x01774e (0x36da bytes) 000669 1761141703.101132 write 0x03b2e4 thru 0x03ffad (0x4cca bytes) 000671 1761141703.102656 mapwrite 0x03a17a thru 0x03ed92 (0x4c19 bytes) 000675 1761141703.124392 read 0x03ae81 thru 0x03ffad (0x512d bytes) 000681 1761141703.126640 write 0x009966 thru 0x011ebd (0x8558 bytes) 000682 1761141703.127634 mapread 0x0290b8 thru 0x02dd43 (0x4c8c bytes) 000683 1761141703.128460 mapread 0x034664 thru 0x03bb0b (0x74a8 bytes) 000686 1761141703.130177 read 0x0220d7 thru 0x027c34 (0x5b5e bytes) 000688 1761141703.131007 mapwrite 0x0171a5 thru 0x026df4 (0xfc50 bytes) 000690 1761141703.152124 trunc from 0x03ffae to 0x039a34 (0x6579 bytes) 000691 1761141703.167630 trunc from 0x039a35 to 0x02c364 (0xd6d0 bytes) 000692 1761141703.173818 mapread 0x01ed34 thru 0x01f423 (0x6f0 bytes) 000693 1761141703.174580 read 0x013800 thru 0x01aeae (0x76af bytes) 000695 1761141703.175575 mapwrite 0x035fb1 thru 0x03e28c (0x82dc bytes) 000696 1761141703.203430 write 0x03e1bb thru 0x03f1b6 (0xffc bytes) 000697 1761141703.204312 write 0x034f20 thru 0x03c7dd (0x78be bytes) 000699 1761141703.207716 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000700 1761141703.299943 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000701 1761141703.384171 write 0x013960 thru 0x01ca82 (0x9123 bytes) 000702 1761141703.389832 mapread 0x01dbc9 thru 0x023a3e (0x5e76 bytes) 000703 1761141703.398084 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000704 1761141703.511562 write 0x005eaf thru 0x008cd6 (0x2e28 bytes) 000706 1761141703.516738 read 0x0174ec thru 0x02304b (0xbb60 bytes) 000708 1761141703.521737 trunc from 0x03f1b7 to 0x0253d9 (0x19ddd bytes) 000709 1761141703.535222 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000712 1761141703.623006 write 0x0325e2 thru 0x036a46 (0x4465 bytes) 000716 1761141703.625624 mapwrite 0x03409c thru 0x03af81 (0x6ee6 bytes) 000717 1761141703.659379 trunc from 0x03af82 to 0x02f0c9 (0xbeb8 bytes) 000718 1761141703.665907 mapwrite 0x025da2 thru 0x030549 (0xa7a8 bytes) 000719 1761141703.707272 MIRROR_RESYNC 4 mirrors 000720 1761141703.977105 mapread 0x02bc26 thru 0x030549 (0x4924 bytes) 000721 1761141703.984506 write 0x0054a3 thru 0x006da9 (0x1907 bytes) 000724 1761141704.011154 trunc from 0x03054a to 0x00d61a (0x22f2f bytes) 000725 1761141704.024723 mapwrite 0x013733 thru 0x01ff4c (0xc81a bytes) 000729 1761141704.055667 read 0x01c54d thru 0x01ff4c (0x3a00 bytes) 000730 1761141704.056235 write 0x0079f6 thru 0x00e9d5 (0x6fe0 bytes) 000735 1761141704.058711 trunc from 0x01ff4d to 0x030f26 (0x10fda bytes) 000736 1761141704.066306 mapread 0x01d27f thru 0x020fad (0x3d2f bytes) 000737 1761141704.070928 mapread 0x0122eb thru 0x01888a (0x65a0 bytes) 000738 1761141704.071777 mapwrite 0x0325b1 thru 0x03b2ba (0x8d0a bytes) 000740 1761141704.111645 trunc from 0x03b2bb to 0x01bca5 (0x1f615 bytes) 000743 1761141704.119894 read 0x0129ee thru 0x0177e3 (0x4df6 bytes) 000745 1761141704.120966 mapread 0x00cd4b thru 0x017904 (0xabba bytes) 000746 1761141704.121756 trunc from 0x01bca6 to 0x023695 (0x79f0 bytes) 000747 1761141704.130033 mapwrite 0x02ce85 thru 0x03656d (0x96e9 bytes) 000748 1761141704.169164 MIRROR_RESYNC 4 mirrors 000749 1761141704.397389 trunc from 0x03656e to 0x01c8b4 (0x19cb9 bytes) 000752 1761141704.429304 trunc from 0x01c8b5 to 0x0358ad (0x18ff9 bytes) 000755 1761141704.440315 trunc from 0x0358ae to 0x01a879 (0x1b034 bytes) 000757 1761141704.450344 MIRROR_EXTEND to 5 mirrors 000758 1761141704.654434 write 0x0030cc thru 0x00bf63 (0x8e98 bytes) 000761 1761141704.700628 mapread 0x00a333 thru 0x014dfe (0xaacc bytes) 000762 1761141704.709168 mapread 0x00c5ca thru 0x01545f (0x8e96 bytes) 000766 1761141704.721132 MIRROR_RESYNC 5 mirrors 000767 1761141705.052548 read 0x011edd thru 0x01581b (0x393f bytes) 000768 1761141705.056435 write 0x0361db thru 0x03f2d6 (0x90fc bytes) 000770 1761141705.081695 trunc from 0x03f2d7 to 0x031b1b (0xd7bb bytes) 000771 1761141705.097466 mapwrite 0x020972 thru 0x02f195 (0xe824 bytes) 000772 1761141705.134676 write 0x00f01d thru 0x018176 (0x915a bytes) 000775 1761141705.143771 mapwrite 0x006da4 thru 0x015058 (0xe2b5 bytes) 000776 1761141705.171205 mapread 0x01024e thru 0x01d1fd (0xcfb0 bytes) 000777 1761141705.174935 mapwrite 0x00a2f9 thru 0x01414a (0x9e52 bytes) 000778 1761141705.194486 mapwrite 0x03c253 thru 0x03ffff (0x3dad bytes) 000779 1761141705.225308 write 0x001dd0 thru 0x00e255 (0xc486 bytes) 000781 1761141705.226538 mapread 0x031817 thru 0x03a5c9 (0x8db3 bytes) 000783 1761141705.231522 write 0x01b762 thru 0x0219d8 (0x6277 bytes) 000784 1761141705.235207 MIRROR_EXTEND to 6 mirrors 000785 1761141705.506678 MIRROR_RESYNC 6 mirrors 000786 1761141705.838009 mapread 0x033750 thru 0x03ee8d (0xb73e bytes) 000787 1761141705.849569 MIRROR_RESYNC 6 mirrors 000788 1761141705.947157 write 0x0261df thru 0x0310ac (0xaece bytes) 000790 1761141705.969831 mapread 0x02a26c thru 0x032182 (0x7f17 bytes) 000791 1761141705.973227 trunc from 0x040000 to 0x03c14a (0x3eb5 bytes) 000793 1761141705.986258 mapwrite 0x00e377 thru 0x0161a6 (0x7e30 bytes) 000794 1761141706.009038 trunc from 0x03c14b to 0x00175d (0x3a9ed bytes) 000796 1761141706.015555 trunc from 0x00175e to 0x01d5f3 (0x1be96 bytes) 000799 1761141706.022975 mapwrite 0x03c767 thru 0x03ffff (0x3899 bytes) 000800 1761141706.055583 mapread 0x0193f3 thru 0x028213 (0xee21 bytes) 000801 1761141706.063599 mapwrite 0x02dadd thru 0x03a4c4 (0xc9e8 bytes) 000802 1761141706.083221 mapread 0x00d507 thru 0x01935c (0xbe56 bytes) 000804 1761141706.087620 mapwrite 0x00066d thru 0x00fde2 (0xf776 bytes) 000806 1761141706.113090 MIRROR_RESYNC 6 mirrors 000809 1761141706.401014 mapread 0x0102ca thru 0x01599a (0x56d1 bytes) 000813 1761141706.408475 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000815 1761141706.536078 mapread 0x02061c thru 0x02d717 (0xd0fc bytes) 000816 1761141706.549025 mapread 0x0329d5 thru 0x03ffff (0xd62b bytes) 000821 1761141706.559221 MIRROR_EXTEND to 6 mirrors 000822 1761141706.847261 read 0x00f161 thru 0x019cdb (0xab7b bytes) 000823 1761141706.851448 read 0x01d045 thru 0x0224a6 (0x5462 bytes) 000826 1761141706.856155 trunc from 0x040000 to 0x00f7fd (0x30802 bytes) 000827 1761141706.900741 mapread 0x00b554 thru 0x00f7fd (0x42aa bytes) 000829 1761141706.912474 mapread 0x00e0e8 thru 0x00e9d1 (0x8ea bytes) 000830 1761141706.913320 read 0x004779 thru 0x00f7fd (0xb085 bytes) 000831 1761141706.920301 MIRROR_RESYNC 6 mirrors 000832 1761141707.332459 mapread 0x000c2c thru 0x00cb3f (0xbf14 bytes) 000833 1761141707.347640 mapread 0x005d19 thru 0x00f7fd (0x9ae5 bytes) 000834 1761141707.348231 mapwrite 0x022a40 thru 0x026139 (0x36fa bytes) 000835 1761141707.405134 read 0x011656 thru 0x015cf9 (0x46a4 bytes) 000836 1761141707.409472 read 0x000eb9 thru 0x00a52b (0x9673 bytes) 000837 1761141707.413911 trunc from 0x02613a to 0x02a2da (0x41a1 bytes) 000838 1761141707.422459 mapwrite 0x01392b thru 0x013d60 (0x436 bytes) 000839 1761141707.451662 MIRROR_EXTEND to 7 mirrors 000840 1761141707.659775 read 0x018679 thru 0x01ea5c (0x63e4 bytes) 000841 1761141707.665224 read 0x00099a thru 0x00c650 (0xbcb7 bytes) 000845 1761141707.677255 mapread 0x00c55c thru 0x00df25 (0x19ca bytes) 000847 1761141707.682956 read 0x0052b0 thru 0x00d41a (0x816b bytes) 000848 1761141707.685147 mapwrite 0x02faa2 thru 0x03bf33 (0xc492 bytes) 000850 1761141707.749003 mapwrite 0x00167e thru 0x00285d (0x11e0 bytes) 000851 1761141707.766392 mapwrite 0x01dbf8 thru 0x024c02 (0x700b bytes) 000852 1761141707.787672 trunc from 0x03bf34 to 0x030586 (0xb9ad bytes) 000853 1761141707.799344 mapwrite 0x02a30e thru 0x02d68c (0x337f bytes) 000854 1761141707.822564 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000856 1761141707.905312 mapwrite 0x0103f3 thru 0x0134dd (0x30eb bytes) 000857 1761141707.926445 trunc from 0x030587 to 0x005185 (0x2b401 bytes) 000859 1761141707.935836 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000862 1761141708.013328 mapwrite 0x009db2 thru 0x0105e3 (0x6832 bytes) 000863 1761141708.047325 trunc from 0x0105e4 to 0x027316 (0x16d33 bytes) 000869 1761141708.055738 trunc from 0x027317 to 0x011960 (0x159b6 bytes) 000870 1761141708.061909 write 0x0196b3 thru 0x020dae (0x76fc bytes) 000871 1761141708.062701 read 0x019227 thru 0x020dae (0x7b88 bytes) 000872 1761141708.063091 mapwrite 0x033242 thru 0x03ffff (0xcdbe bytes) 000874 1761141708.091647 read 0x00aa0d thru 0x016772 (0xbd66 bytes) 000875 1761141708.095610 mapwrite 0x006f90 thru 0x00b321 (0x4392 bytes) 000876 1761141708.113703 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000877 1761141708.183710 mapread 0x000df3 thru 0x008141 (0x734f bytes) 000878 1761141708.192994 mapread 0x009c84 thru 0x017218 (0xd595 bytes) 000879 1761141708.197601 write 0x00a984 thru 0x011596 (0x6c13 bytes) 000880 1761141708.198663 mapread 0x037ee3 thru 0x038b1e (0xc3c bytes) 000881 1761141708.206276 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000882 1761141708.279426 mapwrite 0x012288 thru 0x018a4e (0x67c7 bytes) 000883 1761141708.306053 MIRROR_RESYNC 3 mirrors 000885 1761141708.520103 mapwrite 0x020bcc thru 0x023038 (0x246d bytes) 000886 1761141708.581189 trunc from 0x040000 to 0x01fc6b (0x20394 bytes) 000887 1761141708.597266 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000888 1761141708.668760 trunc from 0x01fc6c to 0x027436 (0x77cb bytes) 000889 1761141708.676076 write 0x014bfe thru 0x015df3 (0x11f6 bytes) 000890 1761141708.680943 mapread 0x0151c2 thru 0x01d433 (0x8272 bytes) 000893 1761141708.691867 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c3:0x0]: Invalid argument (22) 000894 1761141708.760747 mapwrite 0x004740 thru 0x00d223 (0x8ae4 bytes) 000896 1761141708.803250 mapwrite 0x016eb5 thru 0x026e71 (0xffbd bytes) 000897 1761141708.842857 MIRROR_EXTEND to 2 mirrors 000898 1761141709.049379 mapread 0x0120e8 thru 0x01a188 (0x80a1 bytes) 000900 1761141709.059667 MIRROR_RESYNC 2 mirrors 000902 1761141709.137072 MIRROR_EXTEND to 3 mirrors 000905 1761141709.342008 mapread 0x007ca0 thru 0x00c8f7 (0x4c58 bytes) 000906 1761141709.350573 MIRROR_RESYNC 3 mirrors 000907 1761141709.438559 trunc from 0x027437 to 0x009d76 (0x1d6c0 bytes) 000909 1761141709.468293 mapwrite 0x0280e1 thru 0x02a4b2 (0x23d2 bytes) 000910 1761141709.495182 write 0x00c2e7 thru 0x0114cc (0x51e6 bytes) 000913 1761141709.498677 write 0x01aec6 thru 0x028c6e (0xdda9 bytes) 000917 1761141709.503901 mapread 0x01636f thru 0x02626d (0xfeff bytes) 000920 1761141709.507339 MIRROR_EXTEND to 4 mirrors 000924 1761141709.679174 mapread 0x014d6b thru 0x019c60 (0x4ef6 bytes) 000926 1761141709.689082 mapread 0x01ab50 thru 0x022840 (0x7cf1 bytes) 000927 1761141709.695213 read 0x0069c7 thru 0x01133d (0xa977 bytes) 000929 1761141709.700826 mapwrite 0x00948f thru 0x00be22 (0x2994 bytes) 000931 1761141709.751432 trunc from 0x02a4b3 to 0x00e8f6 (0x1bbbc bytes) 000932 1761141709.765970 read 0x002264 thru 0x00e8f6 (0xc693 bytes) 000933 1761141709.766593 write 0x01760c thru 0x0259df (0xe3d4 bytes) 000934 1761141709.768008 trunc from 0x0259e0 to 0x00152d (0x244b2 bytes) 000936 1761141709.775126 mapwrite 0x01230a thru 0x01d9b9 (0xb6b0 bytes) 000939 1761141709.802453 trunc from 0x01d9ba to 0x03c31b (0x1e962 bytes) 000942 1761141709.808937 mapread 0x030026 thru 0x03c31b (0xc2f6 bytes) 000943 1761141709.813643 write 0x004366 thru 0x00eb5a (0xa7f5 bytes) 000945 1761141709.815282 mapread 0x021877 thru 0x029cb6 (0x8440 bytes) 000946 1761141709.817934 MIRROR_EXTEND to 5 mirrors 000949 1761141710.009490 trunc from 0x03c31c to 0x02daf9 (0xe822 bytes) 000951 1761141710.038963 MIRROR_EXTEND to 6 mirrors 000952 1761141710.216986 MIRROR_RESYNC 6 mirrors 000953 1761141710.472315 mapread 0x0142b9 thru 0x023fd2 (0xfd1a bytes) 000954 1761141710.481160 trunc from 0x02dafa to 0x039c41 (0xc148 bytes) 000955 1761141710.503243 mapwrite 0x00103b thru 0x00c681 (0xb647 bytes) 000956 1761141710.522962 write 0x0153d8 thru 0x0168e1 (0x150a bytes) 000958 1761141710.526524 read 0x00c79f thru 0x00e988 (0x21ea bytes) 000960 1761141710.527303 trunc from 0x039c42 to 0x035968 (0x42d9 bytes) 000962 1761141710.534003 write 0x026607 thru 0x02c283 (0x5c7d bytes) 000963 1761141710.537421 read 0x00b452 thru 0x01529c (0x9e4b bytes) 000965 1761141710.539884 MIRROR_RESYNC 6 mirrors 000969 1761141710.825954 mapread 0x023db0 thru 0x02ab01 (0x6d52 bytes) 000970 1761141710.832149 mapread 0x019573 thru 0x01ee45 (0x58d3 bytes) 000971 1761141710.835158 read 0x00c0f9 thru 0x014831 (0x8739 bytes) 000973 1761141710.838167 trunc from 0x035969 to 0x015528 (0x20440 bytes) 000974 1761141710.866042 mapwrite 0x0229ca thru 0x0297e8 (0x6e1f bytes) 000975 1761141710.892039 MIRROR_RESYNC 6 mirrors 000976 1761141711.190329 trunc from 0x0297e9 to 0x012abf (0x16d29 bytes) 000980 1761141711.217030 trunc from 0x012ac0 to 0x02f3fd (0x1c93e bytes) 000982 1761141711.224720 trunc from 0x02f3fe to 0x00fc1e (0x1f7df bytes) 000984 1761141711.232542 trunc from 0x00fc1f to 0x03a5fa (0x2a9dc bytes) 000986 1761141711.238236 write 0x026974 thru 0x032232 (0xb8bf bytes) 000987 1761141711.243287 mapread 0x00421e thru 0x00be10 (0x7bf3 bytes) 000989 1761141711.247725 write 0x01360a thru 0x01da7a (0xa471 bytes) 000991 1761141711.258157 read 0x02b085 thru 0x031e6d (0x6de9 bytes) 000992 1761141711.258596 mapread 0x006ada thru 0x013883 (0xcdaa bytes) 000994 1761141711.259776 read 0x01ce3f thru 0x029536 (0xc6f8 bytes) 000998 1761141711.265094 mapread 0x0231d1 thru 0x028e28 (0x5c58 bytes) 000999 1761141711.265920 write 0x01b0b6 thru 0x0204b2 (0x53fd bytes) All operations completed A-OK! PASS 70a (24s) == sanity-flr test 71: check mirror extend parent fid ==== 10:01:53 (1761141713) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 10:01:54 (1761141714) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4695008 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7237184 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8580544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6474912 .. done Extending file size to 8357696 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8964832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9027968 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8798176 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8998048 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9091904 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57454 57455 57457 57459 57461 done 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:03:01 (1761141781) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6587328 .. Extending file size to 3666880 .. done Extending file size to 4044160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8567488 .. Extending file size to 8658784 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8593696 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9108896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8699936 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8704160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9219360 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60966 60967 60969 60971 60973 done 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) 192.168.203.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:04:07 (1761141847) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 4864 2203264 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 4864 2203264 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0416486 s, 50.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111054 s, 18.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:04:15 (1761141855) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0340221 s, 61.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 10:04:17 (1761141857) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x602:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x831:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x831:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x832:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x82c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:04:20 (1761141860) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.085029 s, 24.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x833:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x82d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x82e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:04:23 (1761141863) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0878449 s, 23.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x834:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x835:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x82f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x834:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x835:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x82f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x830:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:04:27 (1761141867) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0614776 s, 34.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x836:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x831:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x836:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x831:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x832:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:04:52 (1761141892) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314648 s, 33.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x837:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x834:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x833:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x837:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x834:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x833:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x838:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:05:38 (1761141938) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x835:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x836:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0430883 s, 48.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x835:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x836:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x839:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x835:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x836:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x839:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:06:23 (1761141983) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg305-server mds-ost sync done. Creating new pool test_204f oleg305-server: Pool lustre.test_204f created Adding targets to pool oleg305-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg305-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0425743 s, 49.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x837:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x83a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x837:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x83a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg305-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg305-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg305-server: Pool lustre.test_204f destroyed PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:07:22 (1761142042) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:07:24 (1761142044) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:07:27 (1761142047) oleg305-server: Pool lustre.test_206 created Adding targets to pool oleg305-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg305-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg305-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg305-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg305-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:07:42 (1761142062) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283327 s, 37.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00937181 s, 112 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd06f3deb Mirror 3: 0xd06f3deb PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:07:45 (1761142065) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:07:46 (1761142066) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:07:47 (1761142067) Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.027939 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2400000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24c0000 Read this (4 bytes): BBBB 192.168.203.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg305-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:08:03 (1761142083) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00325594 s, 2.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:08:05 (1761142085) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142059 s, 73.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x627 lmm_fid: [0x200000404:0x627:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x627 lmm_fid: [0x200000404:0x627:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x844:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835495 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 10:08:06.000000000 -0400 Modify: 2025-10-22 10:08:06.000000000 -0400 Change: 2025-10-22 10:08:06.000000000 -0400 Birth: 2025-10-22 10:08:05.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:08:08 (1761142088) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:08:09 (1761142089) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.124822 s, 328 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128088 s, 320 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 22 10:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x629:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 22 10:08 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 862 sec ============ 10:08:12 (1761142092) === sanity-flr: start cleanup 10:08:12 (1761142092) === === sanity-flr: finish cleanup 10:08:14 (1761142094) ===