-----============= acceptance-small: sanity-flr ============----- Thu Oct 23 12:41:29 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg629-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:41:46 (1761237706) === oleg629-client.virtnet: executing check_config_client /mnt/lustre oleg629-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg629-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ed8e0860800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ed8e0860800.idle_timeout=debug disable quota as required oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:42:13 (1761237733) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7581] == sanity-flr test 0a: lfs mirror create with -N option == 12:42:15 (1761237735) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:42:28 (1761237748) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:42:31 (1761237751) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:42:34 (1761237754) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:42:55 (1761237775) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:42:57 (1761237777) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:43:00 (1761237780) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ed8e0860800.stats=clear osc.lustre-OST0001-osc-ffff8ed8e0860800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:43:06 (1761237786) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:43:16 (1761237796) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:43:23 (1761237803) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:43:26 (1761237806) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 12:43:34 (1761237814) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:43:40 (1761237820) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:43:42 (1761237822) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:43:48 (1761237828) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:43:57 (1761237837) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:44:03 (1761237843) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.01045 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144582 s, 7.3 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:44:11 (1761237851) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.361755 s, 55.1 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:44:18 (1761237858) Stopping /mnt/lustre-ost1 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg629-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg629-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (52s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:45:10 (1761237910) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101735 s, 20.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg629-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (32s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:45:42 (1761237942) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg629-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg629-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (59s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:46:41 (1761238001) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.259 s, 12.3 MB/s -rw-r--r-- 1 root root 52428800 Oct 23 12:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.899s user 0m0.000s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.912s user 0m0.000s sys 0m0.164s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg629-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg629-server oleg629-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.992s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg629-server: oleg629-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg629-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg629-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ed8e0860800.ost_server_uuid in FULL state after 0 sec PASS 33b (65s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:47:46 (1761238066) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:47:49 (1761238069) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:47:52 (1761238072) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 12:47:54 (1761238074) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 12:48:00 (1761238080) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.611113 s, 20.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:48:11 (1761238091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020596 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356135 s, 29.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:48:17 (1761238097) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:48:24 (1761238104) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000741361 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000688817 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0051601 s, 198 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00078686 s, 1.3 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 12:48:31 (1761238111) create test files with size 15734324 .. eaa319a24439df8614e38cf4b42ff764 - 2a577c47e930013979f86ec59fc53eba - 7bd16115424e27a3eaeaf1f4de0433b3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg629-client.virtnet /mnt/lustre (opts:) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 12:48:52 (1761238132) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:49:05 (1761238145) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:49:11 (1761238151) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135217 s, 7.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274331 s, 38.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 12:49:18 (1761238158) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0963135 s, 21.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.209034 s, 20.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227448 s, 9.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 12:49:27 (1761238167) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:49:30 (1761238170) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:49:33 (1761238173) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202806 s, 51.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 12:49:40 (1761238180) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:49:43 (1761238183) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:49:52 (1761238192) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.206174 s, 50.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg629-client.virtnet /mnt/lustre (opts:) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:50:00 (1761238200) llite.lustre-ffff8ed8d25d7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250847 s, 41.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg629-client.virtnet /mnt/lustre (opts:) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ed8c3595000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:50:10 (1761238210) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:50:44 (1761238244) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:51:15 (1761238275) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:51:18 (1761238278) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.165205 s, 19.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.120458 s, 26.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9e7e6a75ffbd27949b9e5697caaa6850 - mirror 1 checksum: 3e648e2f17d7b40b6b68232333960328 - mirror 2 checksum: 9e7e6a75ffbd27949b9e5697caaa6850 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:51:20 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:51:20 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9e7e6a75ffbd27949b9e5697caaa6850 - mirror 1 checksum: 3e648e2f17d7b40b6b68232333960328 - mirror 2 checksum: 3e648e2f17d7b40b6b68232333960328 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:51:20 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:51:20 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:51:29 (1761238289) Failing mds1 on oleg629-server Stopping /mnt/lustre-mds1 (opts:) on oleg629-server 12:51:34 (1761238294) shut down facet: mds1 facet_host: oleg629-server facet_failover_host: oleg629-server Failover mds1 to oleg629-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg629-server: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-MDT0000 12:51:54 (1761238314) targets are mounted 12:51:54 (1761238314) facet_failover done oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290746 s, 36.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:52:06 (1761238326) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:52:10 (1761238330) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:52:12 (1761238332) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00628782 s, 2.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:52:19 (1761238339) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:52:22 (1761238342) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:52:25 (1761238345) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:52:45 (1761238365) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:53:12 (1761238392) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:53:34 (1761238414) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:53:56 (1761238436) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 12:54:02 (1761238442) mirror create pid 52276 mirror split pid 52277 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52276 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52277 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 12:55:08 (1761238508) Seed set to 8509 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1761238510.036367 mapwrite 0x016528 thru 0x01e9f8 (0x84d1 bytes) 000005 1761238510.130757 mapwrite 0x02cf60 thru 0x02de4d (0xeee bytes) 000007 1761238510.247711 mapwrite 0x02cbf3 thru 0x02d1aa (0x5b8 bytes) 000009 1761238510.300953 read 0x017299 thru 0x020213 (0x8f7b bytes) 000011 1761238510.307255 read 0x01f749 thru 0x02de4d (0xe705 bytes) truncating to largest ever: 0x2afbe 000015 1761238510.320246 trunc from 0x02de4e to 0x02afbd (0x2e90 bytes) 000017 1761238510.337643 mapwrite 0x032c31 thru 0x03ce1e (0xa1ee bytes) 000020 1761238510.417245 trunc from 0x03ce1f to 0x00f104 (0x2dd1a bytes) 000024 1761238510.440160 read 0x0060f0 thru 0x00b1cc (0x50dd bytes) 000025 1761238510.441424 trunc from 0x00f105 to 0x0000b8 (0xf04c bytes) 000026 1761238510.460719 read 0x000014 thru 0x0000b8 (0xa5 bytes) 000027 1761238510.461381 mapwrite 0x017a14 thru 0x017d30 (0x31d bytes) 000028 1761238510.525951 write 0x02f246 thru 0x031da7 (0x2b62 bytes) truncating to largest ever: 0x30b26 000031 1761238510.527917 trunc from 0x031da8 to 0x030b25 (0x1282 bytes) 000032 1761238510.551505 write 0x01e834 thru 0x0292fc (0xaac9 bytes) 000036 1761238510.564404 mapwrite 0x02954e thru 0x02f99b (0x644e bytes) 000037 1761238510.619631 read 0x01187d thru 0x01404b (0x27cf bytes) 000039 1761238510.620777 mapread 0x028212 thru 0x028e33 (0xc22 bytes) 000040 1761238510.621635 mapread 0x008e16 thru 0x00d0db (0x42c6 bytes) 000041 1761238510.629577 trunc from 0x030b26 to 0x014bbc (0x1bf69 bytes) 000044 1761238510.665436 mapwrite 0x00b827 thru 0x01042d (0x4c07 bytes) 000046 1761238510.710277 mapwrite 0x02939a thru 0x03158c (0x81f3 bytes) 000047 1761238510.781900 write 0x0195d4 thru 0x025168 (0xbb95 bytes) 000051 1761238510.790841 MIRROR_EXTEND to 2 mirrors 000052 1761238511.401146 mapwrite 0x01db12 thru 0x01fe47 (0x2336 bytes) 000054 1761238511.579096 mapread 0x0029da thru 0x00499d (0x1fc4 bytes) 000055 1761238511.589213 mapread 0x01d709 thru 0x01d902 (0x1fa bytes) truncating to largest ever: 0x3630e 000059 1761238511.592162 trunc from 0x03158d to 0x03630d (0x4d81 bytes) 000061 1761238511.621342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000064 1761238511.759827 read 0x02b626 thru 0x02d842 (0x221d bytes) 000067 1761238511.777677 MIRROR_EXTEND to 2 mirrors 000068 1761238512.422946 mapread 0x032117 thru 0x03310c (0xff6 bytes) truncating to largest ever: 0x3907f 000069 1761238512.432096 trunc from 0x03630e to 0x03907e (0x2d71 bytes) 000070 1761238512.507924 trunc from 0x03907f to 0x014cf8 (0x24386 bytes) 000071 1761238512.527367 read 0x0013ad thru 0x00b288 (0x9edc bytes) 000072 1761238512.534778 trunc from 0x014cf9 to 0x01b7e8 (0x6af0 bytes) 000073 1761238512.555124 mapwrite 0x028667 thru 0x031f96 (0x9930 bytes) 000075 1761238512.648877 read 0x011df6 thru 0x015629 (0x3834 bytes) 000076 1761238512.673701 mapread 0x0249c8 thru 0x03091d (0xbf56 bytes) 000077 1761238512.674945 trunc from 0x031f97 to 0x002f9f (0x2eff7 bytes) 000078 1761238512.692448 write 0x01cd76 thru 0x01ddeb (0x1076 bytes) 000079 1761238512.698595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000082 1761238512.872968 trunc from 0x01ddec to 0x026ec3 (0x90d8 bytes) 000083 1761238512.908286 read 0x003ae8 thru 0x00cadd (0x8ff6 bytes) 000086 1761238512.921989 mapread 0x003a41 thru 0x004457 (0xa17 bytes) 000094 1761238512.931007 mapwrite 0x00aca8 thru 0x0113d1 (0x672a bytes) 000095 1761238513.005772 trunc from 0x026ec4 to 0x01315b (0x13d68 bytes) 000097 1761238513.038278 mapread 0x00fcec thru 0x01315b (0x3470 bytes) 000100 1761238513.039840 read 0x00d221 thru 0x01315b (0x5f3b bytes) 000102 1761238513.041164 write 0x0081ef thru 0x016747 (0xe559 bytes) 000103 1761238513.054696 MIRROR_EXTEND to 2 mirrors 000104 1761238513.528508 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000105 1761238513.668458 mapread 0x013798 thru 0x016747 (0x2fb0 bytes) 000106 1761238513.683099 write 0x036ac2 thru 0x03ffff (0x953e bytes) 000108 1761238513.691960 write 0x030365 thru 0x033101 (0x2d9d bytes) 000110 1761238513.703516 mapread 0x01a768 thru 0x02241c (0x7cb5 bytes) 000115 1761238513.713510 mapread 0x02040e thru 0x02b87e (0xb471 bytes) 000119 1761238513.721122 read 0x01632a thru 0x0234d9 (0xd1b0 bytes) 000120 1761238513.721668 write 0x027815 thru 0x027b98 (0x384 bytes) 000123 1761238513.723740 read 0x02ad34 thru 0x039702 (0xe9cf bytes) 000124 1761238513.725078 write 0x01d18f thru 0x01f3af (0x2221 bytes) 000125 1761238513.726646 read 0x0160e8 thru 0x01bbb8 (0x5ad1 bytes) 000126 1761238513.728075 mapwrite 0x0035b0 thru 0x00a41e (0x6e6f bytes) 000128 1761238513.823510 mapwrite 0x019de6 thru 0x02689f (0xcaba bytes) 000130 1761238513.867065 mapwrite 0x02c560 thru 0x031dcd (0x586e bytes) 000131 1761238513.904037 mapread 0x02693e thru 0x02a12a (0x37ed bytes) 000133 1761238513.906607 read 0x01217b thru 0x019338 (0x71be bytes) 000134 1761238513.907761 write 0x0118a5 thru 0x016997 (0x50f3 bytes) 000137 1761238513.912861 write 0x02b330 thru 0x036cf3 (0xb9c4 bytes) 000140 1761238513.918445 read 0x00e256 thru 0x0157a4 (0x754f bytes) 000142 1761238513.921029 write 0x0303db thru 0x032ebd (0x2ae3 bytes) 000143 1761238513.922689 mapwrite 0x00e4eb thru 0x016ff8 (0x8b0e bytes) 000144 1761238514.010081 trunc from 0x040000 to 0x02ecf4 (0x1130b bytes) 000145 1761238514.077738 write 0x023d4b thru 0x02abbf (0x6e75 bytes) 000146 1761238514.082677 read 0x00f235 thru 0x01a662 (0xb42e bytes) 000147 1761238514.083295 read 0x0193de thru 0x01bd4b (0x296e bytes) 000149 1761238514.088895 mapread 0x0086c4 thru 0x00bda5 (0x36e2 bytes) 000153 1761238514.091056 write 0x00c38f thru 0x0106a2 (0x4314 bytes) 000155 1761238514.101998 mapread 0x0097ae thru 0x00b01e (0x1871 bytes) 000157 1761238514.105754 mapread 0x006f99 thru 0x00855b (0x15c3 bytes) 000158 1761238514.107778 trunc from 0x02ecf5 to 0x00be02 (0x22ef2 bytes) 000159 1761238514.151037 mapread 0x00a99b thru 0x00be02 (0x1468 bytes) 000161 1761238514.152432 read 0x00a36f thru 0x00be02 (0x1a94 bytes) 000165 1761238514.157059 write 0x021bba thru 0x02fa0b (0xde52 bytes) 000166 1761238514.164399 mapwrite 0x038be5 thru 0x03ecf2 (0x610e bytes) 000168 1761238514.251358 trunc from 0x03ecf3 to 0x038491 (0x6861 bytes) 000169 1761238514.268051 mapread 0x0325b0 thru 0x037428 (0x4e79 bytes) 000170 1761238514.268974 trunc from 0x038492 to 0x00b046 (0x2d44b bytes) 000171 1761238514.300153 mapread 0x00aab5 thru 0x00b046 (0x592 bytes) 000172 1761238514.301338 trunc from 0x00b047 to 0x00f600 (0x45ba bytes) 000175 1761238514.324994 trunc from 0x00f601 to 0x029fa5 (0x1a9a5 bytes) 000177 1761238514.347373 trunc from 0x029fa6 to 0x03071e (0x6779 bytes) 000178 1761238514.364353 trunc from 0x03071f to 0x002d8f (0x2d98f bytes) 000183 1761238514.379983 write 0x027490 thru 0x0300fe (0x8c6f bytes) 000185 1761238514.382973 read 0x019b2f thru 0x01e46f (0x4941 bytes) 000191 1761238514.394552 read 0x02e080 thru 0x0300fe (0x207f bytes) 000192 1761238514.395718 read 0x008ebc thru 0x00a386 (0x14cb bytes) 000193 1761238514.396623 read 0x00edcc thru 0x010aa0 (0x1cd5 bytes) 000194 1761238514.397473 mapwrite 0x021209 thru 0x024823 (0x361b bytes) 000195 1761238514.455265 write 0x0349d4 thru 0x03bb3b (0x7168 bytes) 000198 1761238514.458710 mapwrite 0x021d09 thru 0x02c9a9 (0xaca1 bytes) 000200 1761238514.505145 write 0x024521 thru 0x024f87 (0xa67 bytes) 000201 1761238514.506002 mapwrite 0x00eb72 thru 0x016045 (0x74d4 bytes) 000202 1761238514.557489 read 0x027a57 thru 0x031a7c (0xa026 bytes) 000203 1761238514.570877 trunc from 0x03bb3c to 0x00ce9e (0x2ec9d bytes) 000204 1761238514.599951 read 0x0082b1 thru 0x0096ca (0x141a bytes) 000205 1761238514.600495 read 0x005099 thru 0x00ce9e (0x7e06 bytes) 000209 1761238514.602313 mapread 0x00c985 thru 0x00ce9e (0x51a bytes) 000210 1761238514.606625 write 0x00a005 thru 0x016ba0 (0xcb9c bytes) 000211 1761238514.617090 mapread 0x0149f5 thru 0x016ba0 (0x21ac bytes) 000216 1761238514.640784 read 0x003174 thru 0x00b9a3 (0x8830 bytes) 000217 1761238514.644462 mapwrite 0x03bd22 thru 0x03f314 (0x35f3 bytes) 000218 1761238514.720882 write 0x016336 thru 0x022e7b (0xcb46 bytes) 000223 1761238514.724812 read 0x00bbc3 thru 0x010827 (0x4c65 bytes) 000224 1761238514.725576 mapread 0x033276 thru 0x03f314 (0xc09f bytes) 000225 1761238514.726827 mapwrite 0x02c621 thru 0x033751 (0x7131 bytes) 000226 1761238514.775746 mapwrite 0x033c47 thru 0x0366a3 (0x2a5d bytes) 000233 1761238514.821413 mapread 0x02ac4a thru 0x030983 (0x5d3a bytes) 000238 1761238514.826736 read 0x00a669 thru 0x017c99 (0xd631 bytes) 000239 1761238514.827800 mapread 0x01f810 thru 0x023393 (0x3b84 bytes) 000244 1761238514.835498 mapwrite 0x028cbe thru 0x03735a (0xe69d bytes) 000245 1761238514.895939 write 0x00c6dc thru 0x00d217 (0xb3c bytes) 000246 1761238514.898153 trunc from 0x03f315 to 0x015bf1 (0x29723 bytes) 000249 1761238514.915364 mapread 0x0091c2 thru 0x00d65c (0x449b bytes) 000251 1761238514.916967 mapwrite 0x0340e4 thru 0x03e6c5 (0xa5e2 bytes) 000252 1761238514.985715 mapwrite 0x008965 thru 0x00e943 (0x5fdf bytes) 000253 1761238515.030944 mapread 0x03965f thru 0x03e6c5 (0x5067 bytes) 000254 1761238515.031830 mapread 0x01e808 thru 0x0222f1 (0x3aea bytes) 000255 1761238515.032562 read 0x00f686 thru 0x0180ab (0x8a26 bytes) 000258 1761238515.034492 read 0x00cc81 thru 0x00ef41 (0x22c1 bytes) 000260 1761238515.037119 mapwrite 0x0070c2 thru 0x008ebc (0x1dfb bytes) 000263 1761238515.115061 mapread 0x01662b thru 0x01956d (0x2f43 bytes) 000264 1761238515.117117 mapread 0x01c719 thru 0x02af35 (0xe81d bytes) 000266 1761238515.118511 mapread 0x00aa48 thru 0x0121ef (0x77a8 bytes) 000267 1761238515.119336 mapwrite 0x03e0f5 thru 0x03ffff (0x1f0b bytes) 000268 1761238515.187138 mapwrite 0x01f5cd thru 0x02f07c (0xfab0 bytes) 000271 1761238515.246445 write 0x017bab thru 0x01f9df (0x7e35 bytes) 000272 1761238515.249064 mapread 0x00ac1f thru 0x017f4f (0xd331 bytes) 000273 1761238515.249960 trunc from 0x040000 to 0x00a212 (0x35ded bytes) 000274 1761238515.282138 MIRROR_EXTEND to 2 mirrors 000275 1761238515.898424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000277 1761238516.065087 mapread 0x006ea7 thru 0x008928 (0x1a82 bytes) 000280 1761238516.089150 write 0x006be1 thru 0x00cf2d (0x634d bytes) 000281 1761238516.113276 mapread 0x003768 thru 0x00cf2d (0x97c6 bytes) 000282 1761238516.114262 write 0x039502 thru 0x03cb37 (0x3636 bytes) 000283 1761238516.115436 mapread 0x01d074 thru 0x02b9b9 (0xe946 bytes) 000285 1761238516.137407 mapread 0x016e06 thru 0x01e401 (0x75fc bytes) 000286 1761238516.138465 mapwrite 0x004a4e thru 0x014167 (0xf71a bytes) 000288 1761238516.211860 MIRROR_EXTEND to 2 mirrors 000289 1761238516.896774 mapwrite 0x000e9b thru 0x00144c (0x5b2 bytes) 000290 1761238517.075403 write 0x038297 thru 0x03ec57 (0x69c1 bytes) 000297 1761238517.094604 mapread 0x0208e4 thru 0x0232c2 (0x29df bytes) 000299 1761238517.108791 mapread 0x004aba thru 0x00f43b (0xa982 bytes) 000301 1761238517.110214 read 0x0374c0 thru 0x03ec57 (0x7798 bytes) 000303 1761238517.111797 read 0x02707a thru 0x036d5d (0xfce4 bytes) 000304 1761238517.124254 write 0x01ca86 thru 0x022228 (0x57a3 bytes) 000305 1761238517.125494 write 0x020858 thru 0x02ab5a (0xa303 bytes) 000307 1761238517.128041 trunc from 0x03ec58 to 0x0097a6 (0x354b1 bytes) 000308 1761238517.159339 MIRROR_RESYNC 2 mirrors 000310 1761238517.686838 write 0x004178 thru 0x00f233 (0xb0bc bytes) 000312 1761238517.763029 trunc from 0x00f234 to 0x03829b (0x29068 bytes) 000313 1761238517.808681 read 0x001eef thru 0x00350a (0x161c bytes) 000314 1761238517.816714 write 0x0156a2 thru 0x017f0e (0x286d bytes) 000315 1761238517.833436 MIRROR_EXTEND to 3 mirrors 000317 1761238518.518247 MIRROR_EXTEND to 4 mirrors 000320 1761238519.240865 mapread 0x0189cc thru 0x01a254 (0x1889 bytes) 000321 1761238519.264771 mapwrite 0x01a66b thru 0x01da5a (0x33f0 bytes) 000324 1761238519.489707 mapread 0x037a2e thru 0x03829b (0x86e bytes) 000325 1761238519.498622 mapread 0x0079b1 thru 0x01268b (0xacdb bytes) 000326 1761238519.527945 mapwrite 0x01e423 thru 0x023e71 (0x5a4f bytes) truncating to largest ever: 0x3e7b0 000328 1761238519.640029 trunc from 0x03829c to 0x03e7af (0x6514 bytes) 000329 1761238519.687455 trunc from 0x03e7b0 to 0x02dfd1 (0x107de bytes) 000331 1761238519.716099 MIRROR_EXTEND to 5 mirrors 000335 1761238520.650164 write 0x039a3d thru 0x03e4cc (0x4a90 bytes) 000339 1761238520.757479 trunc from 0x03e4cd to 0x0392b2 (0x521a bytes) 000341 1761238520.783469 mapwrite 0x034eb0 thru 0x03bd7e (0x6ecf bytes) 000344 1761238520.893778 trunc from 0x03bd7f to 0x039cf7 (0x2087 bytes) 000347 1761238520.927325 mapwrite 0x00836d thru 0x012ae4 (0xa778 bytes) 000349 1761238521.023550 write 0x03cffd thru 0x03ffff (0x3003 bytes) 000351 1761238521.025140 trunc from 0x040000 to 0x027148 (0x18eb7 bytes) 000352 1761238521.055585 mapread 0x0068b0 thru 0x010900 (0xa051 bytes) 000354 1761238521.057062 read 0x001826 thru 0x005d2f (0x450a bytes) 000355 1761238521.057642 mapread 0x01d710 thru 0x023237 (0x5b28 bytes) 000356 1761238521.102303 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000357 1761238521.417279 write 0x03026f thru 0x03f033 (0xedc5 bytes) 000359 1761238521.419386 read 0x0275fb thru 0x02f12a (0x7b30 bytes) 000360 1761238521.444797 read 0x030504 thru 0x03e039 (0xdb36 bytes) 000361 1761238521.448732 read 0x00f22e thru 0x01baa4 (0xc877 bytes) 000362 1761238521.468775 trunc from 0x03f034 to 0x00b0cf (0x33f64 bytes) 000366 1761238521.495054 write 0x008235 thru 0x0128fb (0xa6c7 bytes) 000367 1761238521.497031 mapwrite 0x020c64 thru 0x0223d8 (0x1775 bytes) 000370 1761238521.616778 read 0x00c902 thru 0x010387 (0x3a86 bytes) 000371 1761238521.640707 mapread 0x00655d thru 0x00b377 (0x4e1b bytes) 000372 1761238521.641475 write 0x02a5b3 thru 0x039bc4 (0xf612 bytes) 000373 1761238521.643026 write 0x03d763 thru 0x03ffff (0x289d bytes) 000375 1761238521.650497 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000376 1761238521.922535 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000377 1761238522.143035 MIRROR_RESYNC 2 mirrors 000378 1761238522.799687 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000379 1761238522.925725 mapread 0x005ee8 thru 0x0061f3 (0x30c bytes) 000381 1761238522.943263 mapread 0x0333da thru 0x03ffff (0xcc26 bytes) 000384 1761238522.960858 trunc from 0x040000 to 0x01d206 (0x22df9 bytes) 000385 1761238523.000394 mapwrite 0x020314 thru 0x02bfc9 (0xbcb6 bytes) 000386 1761238523.099240 MIRROR_EXTEND to 2 mirrors 000388 1761238523.609955 trunc from 0x02bfca to 0x006c63 (0x25366 bytes) 000389 1761238523.710497 mapread 0x0060e9 thru 0x006c63 (0xb7b bytes) 000391 1761238523.718347 mapread 0x002eaa thru 0x006c63 (0x3dba bytes) 000392 1761238523.746961 write 0x010e20 thru 0x020706 (0xf8e7 bytes) 000393 1761238523.748628 mapwrite 0x00ba14 thru 0x0166b1 (0xac9e bytes) 000394 1761238523.802398 write 0x0378fc thru 0x03ffff (0x8704 bytes) 000397 1761238523.804411 write 0x03d2cb thru 0x03ffff (0x2d35 bytes) 000398 1761238523.810651 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000400 1761238523.957865 mapread 0x01ded8 thru 0x02cee9 (0xf012 bytes) 000401 1761238523.995374 mapread 0x03e192 thru 0x03ffff (0x1e6e bytes) 000407 1761238524.016591 write 0x00f051 thru 0x014d22 (0x5cd2 bytes) 000409 1761238524.026040 mapread 0x019809 thru 0x01f142 (0x593a bytes) 000411 1761238524.027712 mapwrite 0x029580 thru 0x037a6d (0xe4ee bytes) 000412 1761238524.081003 mapwrite 0x01dce5 thru 0x02cf47 (0xf263 bytes) 000415 1761238524.126521 write 0x002b01 thru 0x010f58 (0xe458 bytes) 000418 1761238524.136110 write 0x02907e thru 0x02b220 (0x21a3 bytes) 000420 1761238524.138614 trunc from 0x040000 to 0x015c53 (0x2a3ac bytes) 000421 1761238524.157651 mapwrite 0x03949f thru 0x039d72 (0x8d4 bytes) 000422 1761238524.253723 trunc from 0x039d73 to 0x0048d2 (0x354a0 bytes) 000427 1761238524.280563 trunc from 0x0048d3 to 0x020649 (0x1bd77 bytes) 000430 1761238524.299535 mapread 0x018a0c thru 0x020649 (0x7c3e bytes) 000431 1761238524.309496 mapwrite 0x03f60d thru 0x03ffff (0x9f3 bytes) 000434 1761238524.378271 write 0x037dc0 thru 0x03ffff (0x8240 bytes) 000435 1761238524.387841 mapread 0x0017d0 thru 0x005aec (0x431d bytes) 000436 1761238524.397138 read 0x00e2e8 thru 0x013605 (0x531e bytes) 000437 1761238524.397753 read 0x01ba32 thru 0x01e88f (0x2e5e bytes) 000438 1761238524.398271 mapwrite 0x004e63 thru 0x00808c (0x322a bytes) 000439 1761238524.444423 read 0x011be5 thru 0x01b94f (0x9d6b bytes) 000440 1761238524.445128 trunc from 0x040000 to 0x032553 (0xdaac bytes) 000441 1761238524.468071 mapwrite 0x026d00 thru 0x0329b9 (0xbcba bytes) 000442 1761238524.543857 mapread 0x020d2a thru 0x02c531 (0xb808 bytes) 000444 1761238524.545274 read 0x030d09 thru 0x0329b9 (0x1cb1 bytes) 000445 1761238524.545801 write 0x03556b thru 0x03ffff (0xaa95 bytes) 000446 1761238524.547653 write 0x00be09 thru 0x016953 (0xab4b bytes) 000448 1761238524.550734 mapwrite 0x007ff5 thru 0x009900 (0x190c bytes) 000450 1761238524.608901 mapwrite 0x012cf2 thru 0x01dbf3 (0xaf02 bytes) 000451 1761238524.657339 write 0x025962 thru 0x0326ee (0xcd8d bytes) 000453 1761238524.659544 mapwrite 0x006040 thru 0x00faeb (0x9aac bytes) 000455 1761238524.732463 mapwrite 0x01e2d0 thru 0x02b8b6 (0xd5e7 bytes) 000457 1761238524.776939 mapwrite 0x029c7a thru 0x0380f6 (0xe47d bytes) 000460 1761238524.840697 mapwrite 0x00e607 thru 0x016b5c (0x8556 bytes) 000461 1761238524.881728 mapread 0x0342f6 thru 0x039f29 (0x5c34 bytes) 000463 1761238524.890534 MIRROR_EXTEND to 2 mirrors 000466 1761238525.656055 MIRROR_EXTEND to 3 mirrors 000467 1761238526.294633 mapwrite 0x0047ef thru 0x013b5e (0xf370 bytes) 000469 1761238526.525118 mapread 0x0375d8 thru 0x03f2d9 (0x7d02 bytes) 000472 1761238526.544151 mapwrite 0x0206e8 thru 0x02bf70 (0xb889 bytes) 000475 1761238526.631138 read 0x009813 thru 0x00eb0b (0x52f9 bytes) 000476 1761238526.631943 write 0x025b60 thru 0x02b081 (0x5522 bytes) 000477 1761238526.633668 mapread 0x036da9 thru 0x03ffff (0x9257 bytes) 000479 1761238526.639866 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000481 1761238526.878132 trunc from 0x040000 to 0x028367 (0x17c98 bytes) 000482 1761238526.902123 MIRROR_EXTEND to 3 mirrors 000483 1761238527.566786 trunc from 0x028368 to 0x01dd68 (0xa5ff bytes) 000485 1761238527.663595 mapwrite 0x0340c6 thru 0x037c8d (0x3bc8 bytes) 000486 1761238527.783645 write 0x0323d2 thru 0x03687b (0x44aa bytes) 000487 1761238527.784785 mapwrite 0x030493 thru 0x037e5f (0x79cd bytes) 000489 1761238527.851851 trunc from 0x037e60 to 0x029758 (0xe707 bytes) 000490 1761238527.879571 write 0x009cab thru 0x011b1e (0x7e74 bytes) 000491 1761238527.900723 write 0x00a3ef thru 0x00f035 (0x4c47 bytes) 000492 1761238527.901676 trunc from 0x029759 to 0x024642 (0x5116 bytes) 000493 1761238527.926223 mapread 0x0037a0 thru 0x00e011 (0xa872 bytes) 000494 1761238527.927533 read 0x0181cf thru 0x019d7d (0x1baf bytes) 000495 1761238527.936556 read 0x0024c5 thru 0x00939a (0x6ed6 bytes) 000497 1761238527.943230 MIRROR_RESYNC 3 mirrors 000498 1761238528.638284 read 0x0134a6 thru 0x01b271 (0x7dcc bytes) 000499 1761238528.645905 trunc from 0x024643 to 0x00babc (0x18b86 bytes) 000501 1761238528.779154 mapwrite 0x02687c thru 0x02b3c7 (0x4b4c bytes) 000502 1761238528.913819 write 0x013a15 thru 0x01eb82 (0xb16e bytes) 000503 1761238528.947994 mapwrite 0x02a7ba thru 0x03573d (0xaf84 bytes) 000504 1761238529.050086 MIRROR_RESYNC 3 mirrors 000505 1761238529.812032 mapread 0x017ac3 thru 0x01c98f (0x4ecd bytes) 000508 1761238529.830495 mapwrite 0x0150fd thru 0x0158f0 (0x7f4 bytes) 000509 1761238529.966626 MIRROR_RESYNC 3 mirrors 000510 1761238530.636364 MIRROR_RESYNC 3 mirrors 000515 1761238530.840819 read 0x00b65f thru 0x00bcb5 (0x657 bytes) 000516 1761238530.847010 mapwrite 0x038e1e thru 0x03ac7e (0x1e61 bytes) 000517 1761238531.002640 trunc from 0x03ac7f to 0x034655 (0x6629 bytes) 000518 1761238531.025063 read 0x01ae50 thru 0x026d42 (0xbef3 bytes) 000519 1761238531.035562 trunc from 0x034656 to 0x029bd2 (0xaa83 bytes) 000522 1761238531.056655 write 0x001b41 thru 0x00c265 (0xa725 bytes) 000523 1761238531.057873 trunc from 0x029bd3 to 0x022bf3 (0x6fdf bytes) 000524 1761238531.083081 mapread 0x008463 thru 0x01286f (0xa40d bytes) 000525 1761238531.084399 mapread 0x00a7f3 thru 0x0179d3 (0xd1e1 bytes) 000526 1761238531.090222 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000527 1761238531.333332 mapwrite 0x03a19f thru 0x03ffff (0x5e61 bytes) 000528 1761238531.421563 trunc from 0x040000 to 0x028c2b (0x173d4 bytes) 000529 1761238531.437761 mapwrite 0x02aef2 thru 0x030224 (0x5333 bytes) 000530 1761238531.532693 trunc from 0x030225 to 0x02bcb6 (0x456e bytes) 000533 1761238531.557898 mapwrite 0x02273e thru 0x02c605 (0x9ec8 bytes) 000534 1761238531.642456 mapread 0x01cadd thru 0x020c12 (0x4136 bytes) 000537 1761238531.649539 MIRROR_RESYNC 2 mirrors 000539 1761238532.156711 write 0x03ff70 thru 0x03ffff (0x90 bytes) 000540 1761238532.222144 mapwrite 0x031db8 thru 0x038fd3 (0x721c bytes) 000541 1761238532.293216 MIRROR_EXTEND to 3 mirrors 000542 1761238532.789789 read 0x01570b thru 0x019f02 (0x47f8 bytes) 000543 1761238532.796866 trunc from 0x040000 to 0x02abca (0x15435 bytes) 000547 1761238532.911329 mapwrite 0x020629 thru 0x02b045 (0xaa1d bytes) 000548 1761238533.001030 trunc from 0x02b046 to 0x01d557 (0xdaee bytes) 000549 1761238533.024632 MIRROR_RESYNC 3 mirrors 000550 1761238533.566703 trunc from 0x01d558 to 0x014985 (0x8bd2 bytes) 000553 1761238533.659410 read 0x007fd1 thru 0x0107ec (0x881c bytes) 000555 1761238533.672574 mapread 0x000fa7 thru 0x009ce2 (0x8d3c bytes) 000556 1761238533.689964 write 0x021a1f thru 0x02f0f5 (0xd6d7 bytes) 000561 1761238533.693520 read 0x00bbde thru 0x00da06 (0x1e29 bytes) 000564 1761238533.695769 mapwrite 0x000359 thru 0x00db90 (0xd838 bytes) 000565 1761238533.758109 trunc from 0x02f0f6 to 0x020525 (0xebd0 bytes) 000567 1761238533.785957 mapwrite 0x0295b7 thru 0x035ea9 (0xc8f3 bytes) 000569 1761238533.855198 mapwrite 0x016898 thru 0x025079 (0xe7e2 bytes) 000570 1761238533.909722 read 0x013499 thru 0x01d2c5 (0x9e2d bytes) 000573 1761238533.915281 MIRROR_RESYNC 3 mirrors 000574 1761238534.614934 MIRROR_EXTEND to 4 mirrors 000575 1761238535.382094 write 0x033433 thru 0x03edd6 (0xb9a4 bytes) 000578 1761238535.485560 write 0x03bfce thru 0x03ffff (0x4032 bytes) 000579 1761238535.491395 trunc from 0x040000 to 0x0341a4 (0xbe5b bytes) 000581 1761238535.523576 read 0x00a168 thru 0x01972b (0xf5c4 bytes) 000583 1761238535.541604 MIRROR_RESYNC 4 mirrors 000584 1761238536.713897 mapread 0x014a6d thru 0x01e165 (0x96f9 bytes) 000585 1761238536.734515 MIRROR_EXTEND to 5 mirrors 000586 1761238537.496812 trunc from 0x0341a5 to 0x00b26b (0x28f39 bytes) 000589 1761238537.594321 read 0x00a7f3 thru 0x00b26b (0xa79 bytes) 000590 1761238537.603785 write 0x02076b thru 0x02c96b (0xc201 bytes) 000593 1761238537.607031 write 0x01fbf3 thru 0x02fb6a (0xff78 bytes) 000594 1761238537.617567 read 0x008416 thru 0x014f68 (0xcb53 bytes) 000595 1761238537.636844 mapwrite 0x015547 thru 0x01e9e6 (0x94a0 bytes) 000596 1761238537.689949 read 0x0005fe thru 0x008c31 (0x8634 bytes) 000601 1761238537.693264 trunc from 0x02fb6b to 0x0056dd (0x2a48d bytes) 000602 1761238537.726782 mapread 0x002d7a thru 0x0056dd (0x2964 bytes) 000608 1761238537.730002 mapread 0x0025fb thru 0x0056dd (0x30e3 bytes) 000613 1761238537.737762 MIRROR_EXTEND to 6 mirrors 000614 1761238538.467957 write 0x023be9 thru 0x031b86 (0xdf9e bytes) 000615 1761238538.542838 mapread 0x02f04c thru 0x031b86 (0x2b3b bytes) 000616 1761238538.543631 mapwrite 0x03c24f thru 0x03ffff (0x3db1 bytes) 000618 1761238538.629990 read 0x018411 thru 0x020e7f (0x8a6f bytes) 000620 1761238538.653630 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000622 1761238538.880050 read 0x00d0fc thru 0x00f662 (0x2567 bytes) 000623 1761238538.886127 mapread 0x016a80 thru 0x01dd90 (0x7311 bytes) 000625 1761238538.904802 mapread 0x01b4de thru 0x026119 (0xac3c bytes) 000626 1761238538.913760 mapwrite 0x034fd5 thru 0x036bfe (0x1c2a bytes) 000628 1761238538.958341 mapread 0x039491 thru 0x03ffff (0x6b6f bytes) 000629 1761238538.974432 MIRROR_RESYNC 5 mirrors 000630 1761238539.910827 MIRROR_EXTEND to 6 mirrors 000632 1761238540.723638 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000633 1761238541.200977 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000635 1761238541.602813 trunc from 0x040000 to 0x01910e (0x26ef1 bytes) 000638 1761238541.700269 mapread 0x016c71 thru 0x01910e (0x249e bytes) 000639 1761238541.714946 write 0x00e1ef thru 0x01d1af (0xefc1 bytes) 000640 1761238541.728042 MIRROR_RESYNC 4 mirrors 000643 1761238542.469006 trunc from 0x01d1b0 to 0x03dabc (0x2090d bytes) 000646 1761238542.530771 mapwrite 0x039cc2 thru 0x03ffff (0x633e bytes) 000647 1761238542.643393 read 0x001b6d thru 0x00eeee (0xd382 bytes) 000649 1761238542.659033 mapread 0x03e25d thru 0x03eb63 (0x907 bytes) 000651 1761238542.660336 mapread 0x038747 thru 0x038eb8 (0x772 bytes) 000652 1761238542.661054 mapread 0x00f58f thru 0x01cb43 (0xd5b5 bytes) 000654 1761238542.679087 read 0x00f836 thru 0x011a87 (0x2252 bytes) 000655 1761238542.679710 trunc from 0x040000 to 0x02c013 (0x13fec bytes) 000656 1761238542.701340 write 0x012103 thru 0x01662d (0x452b bytes) 000657 1761238542.702437 mapread 0x02b2af thru 0x02c013 (0xd65 bytes) 000658 1761238542.712812 read 0x00eec9 thru 0x015cd5 (0x6e0d bytes) 000660 1761238542.719128 MIRROR_RESYNC 4 mirrors 000661 1761238543.403106 read 0x01bd95 thru 0x01e720 (0x298c bytes) 000663 1761238543.409328 trunc from 0x02c014 to 0x033eab (0x7e98 bytes) 000665 1761238543.482106 MIRROR_RESYNC 4 mirrors 000667 1761238544.128758 mapwrite 0x01f3da thru 0x029c4c (0xa873 bytes) 000668 1761238544.304419 mapread 0x0285ef thru 0x033eab (0xb8bd bytes) 000670 1761238544.311604 mapwrite 0x018e07 thru 0x023001 (0xa1fb bytes) 000671 1761238544.389024 write 0x03f700 thru 0x03ffff (0x900 bytes) 000672 1761238544.390008 trunc from 0x040000 to 0x018ed4 (0x2712b bytes) 000674 1761238544.415725 read 0x00cd7e thru 0x0158f1 (0x8b74 bytes) 000675 1761238544.422318 trunc from 0x018ed5 to 0x001d49 (0x1718b bytes) 000677 1761238544.450607 mapwrite 0x022397 thru 0x028a39 (0x66a3 bytes) 000681 1761238544.540126 write 0x00cb5d thru 0x015c2d (0x90d1 bytes) 000682 1761238544.567027 MIRROR_EXTEND to 5 mirrors 000683 1761238545.236778 mapwrite 0x024489 thru 0x029ecc (0x5a44 bytes) 000684 1761238545.437874 mapread 0x01fa15 thru 0x0278ed (0x7ed9 bytes) 000685 1761238545.443655 MIRROR_RESYNC 5 mirrors 000686 1761238546.383433 MIRROR_EXTEND to 6 mirrors 000688 1761238547.031240 mapread 0x01418d thru 0x018ba0 (0x4a14 bytes) 000690 1761238547.073649 write 0x009d93 thru 0x010eed (0x715b bytes) 000691 1761238547.208612 write 0x038bdf thru 0x03ffff (0x7421 bytes) 000692 1761238547.209946 mapread 0x036b93 thru 0x03ffff (0x946d bytes) 000693 1761238547.223352 mapread 0x011f6a thru 0x016d4f (0x4de6 bytes) 000694 1761238547.232864 read 0x008179 thru 0x00dc3d (0x5ac5 bytes) 000695 1761238547.233473 read 0x005936 thru 0x009a45 (0x4110 bytes) 000696 1761238547.233980 trunc from 0x040000 to 0x00243d (0x3dbc2 bytes) 000697 1761238547.256833 write 0x00c4a6 thru 0x0158ef (0x944a bytes) 000698 1761238547.259644 mapwrite 0x0196af thru 0x028bcc (0xf51e bytes) 000699 1761238547.361915 mapread 0x0052ce thru 0x00cefd (0x7c30 bytes) 000701 1761238547.363272 trunc from 0x028bcd to 0x02d2f6 (0x472a bytes) 000703 1761238547.385585 trunc from 0x02d2f7 to 0x02d286 (0x70 bytes) 000704 1761238547.399777 write 0x02cb13 thru 0x02e616 (0x1b04 bytes) 000708 1761238547.408623 mapread 0x0297e4 thru 0x02e616 (0x4e33 bytes) 000709 1761238547.409594 mapwrite 0x028581 thru 0x02b0be (0x2b3e bytes) 000712 1761238547.460485 read 0x0173ba thru 0x021465 (0xa0ac bytes) 000713 1761238547.461282 trunc from 0x02e617 to 0x00e661 (0x1ffb5 bytes) 000715 1761238547.476265 mapread 0x006eac thru 0x00cdd9 (0x5f2e bytes) 000716 1761238547.477259 read 0x00b7c5 thru 0x00e661 (0x2e9d bytes) 000717 1761238547.477827 mapwrite 0x031563 thru 0x0391a4 (0x7c42 bytes) 000718 1761238547.549890 read 0x0355b9 thru 0x0391a4 (0x3bec bytes) 000719 1761238547.550518 write 0x00818c thru 0x01706c (0xeee1 bytes) 000720 1761238547.556448 MIRROR_RESYNC 6 mirrors 000721 1761238548.456046 read 0x022640 thru 0x030721 (0xe0e2 bytes) 000722 1761238548.465284 write 0x008809 thru 0x0172bf (0xeab7 bytes) 000723 1761238548.536081 MIRROR_RESYNC 6 mirrors 000725 1761238549.738023 read 0x018794 thru 0x021bd8 (0x9445 bytes) 000726 1761238549.759106 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000728 1761238550.131635 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000730 1761238550.463074 write 0x00b289 thru 0x015ee5 (0xac5d bytes) 000731 1761238550.576922 write 0x038499 thru 0x03ffff (0x7b67 bytes) 000734 1761238550.590403 read 0x01e8cb thru 0x02c44a (0xdb80 bytes) 000736 1761238550.602113 trunc from 0x040000 to 0x03337c (0xcc83 bytes) 000739 1761238550.632826 write 0x000569 thru 0x005d9b (0x5833 bytes) 000740 1761238550.649356 MIRROR_EXTEND to 5 mirrors 000741 1761238551.375909 write 0x03e1e6 thru 0x03ffff (0x1e1a bytes) 000742 1761238551.452814 MIRROR_RESYNC 5 mirrors 000743 1761238552.306655 mapread 0x014369 thru 0x022221 (0xdeb9 bytes) 000745 1761238552.331031 mapread 0x02339d thru 0x02be56 (0x8aba bytes) 000747 1761238552.343563 MIRROR_RESYNC 5 mirrors 000748 1761238552.596987 mapwrite 0x001d7d thru 0x004f8b (0x320f bytes) 000750 1761238552.796380 MIRROR_EXTEND to 6 mirrors 000753 1761238553.514834 trunc from 0x040000 to 0x0148e0 (0x2b71f bytes) 000754 1761238553.595365 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000756 1761238553.796869 mapread 0x003b3c thru 0x0104ff (0xc9c4 bytes) 000760 1761238553.826475 MIRROR_RESYNC 5 mirrors 000763 1761238554.572831 trunc from 0x0148e1 to 0x007034 (0xd8ac bytes) 000765 1761238554.653497 mapread 0x00309c thru 0x007034 (0x3f99 bytes) 000766 1761238554.672942 mapread 0x005df6 thru 0x007034 (0x123f bytes) 000767 1761238554.678149 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000768 1761238554.932753 mapread 0x001803 thru 0x002f45 (0x1743 bytes) 000769 1761238554.941849 mapread 0x002d8a thru 0x007034 (0x42ab bytes) 000770 1761238554.950383 read 0x003331 thru 0x007034 (0x3d04 bytes) 000771 1761238554.953928 MIRROR_RESYNC 4 mirrors 000772 1761238555.795620 MIRROR_EXTEND to 5 mirrors 000773 1761238556.457497 mapwrite 0x0161ff thru 0x024de8 (0xebea bytes) 000775 1761238556.622392 read 0x0179f3 thru 0x018b80 (0x118e bytes) 000777 1761238556.624679 trunc from 0x024de9 to 0x010d8a (0x1405e bytes) 000779 1761238556.650158 write 0x02cf5e thru 0x02ea5f (0x1b02 bytes) 000782 1761238556.658503 MIRROR_RESYNC 5 mirrors 000785 1761238557.447163 mapread 0x01c60d thru 0x021a72 (0x5466 bytes) 000786 1761238557.465896 read 0x01b8b2 thru 0x021210 (0x595f bytes) 000787 1761238557.466287 mapwrite 0x0372be thru 0x03ffff (0x8d42 bytes) 000788 1761238557.621348 trunc from 0x040000 to 0x02aabe (0x15541 bytes) 000791 1761238557.650481 write 0x026d9a thru 0x0296b5 (0x291c bytes) 000793 1761238557.659831 read 0x00d04d thru 0x012249 (0x51fd bytes) 000794 1761238557.670328 trunc from 0x02aabf to 0x024e79 (0x5c45 bytes) 000798 1761238557.694705 read 0x00992e thru 0x00f6dc (0x5daf bytes) 000799 1761238557.706657 MIRROR_EXTEND to 6 mirrors 000805 1761238558.173083 mapread 0x014703 thru 0x015d2a (0x1628 bytes) 000807 1761238558.188284 mapread 0x010122 thru 0x01c7f6 (0xc6d5 bytes) 000809 1761238558.212448 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000811 1761238558.443429 MIRROR_EXTEND to 6 mirrors 000812 1761238559.003271 MIRROR_EXTEND to 7 mirrors 000813 1761238559.771564 trunc from 0x024e7a to 0x013d8c (0x110ed bytes) 000814 1761238559.894535 write 0x02c240 thru 0x0389f4 (0xc7b5 bytes) 000816 1761238559.897540 mapwrite 0x029b2b thru 0x03882a (0xed00 bytes) 000819 1761238560.001148 mapwrite 0x00c4c9 thru 0x010bfb (0x4733 bytes) 000821 1761238560.047950 write 0x009960 thru 0x00a269 (0x90a bytes) 000824 1761238560.049766 mapwrite 0x03766d thru 0x0381e8 (0xb7c bytes) 000826 1761238560.089337 read 0x036b0e thru 0x0389f4 (0x1ee7 bytes) 000828 1761238560.091148 mapread 0x00350f thru 0x005872 (0x2364 bytes) 000830 1761238560.106446 MIRROR_RESYNC 7 mirrors 000833 1761238561.312365 MIRROR_EXTEND to 8 mirrors 000837 1761238562.018418 mapwrite 0x00b2c5 thru 0x017a36 (0xc772 bytes) 000838 1761238562.200392 read 0x020f50 thru 0x027a6d (0x6b1e bytes) 000839 1761238562.222580 mapread 0x005a4d thru 0x00a267 (0x481b bytes) 000840 1761238562.224450 trunc from 0x0389f5 to 0x00f9a2 (0x29052 bytes) 000841 1761238562.268105 mapread 0x005bae thru 0x00ea22 (0x8e75 bytes) 000842 1761238562.269059 trunc from 0x00f9a3 to 0x02749d (0x17afb bytes) 000843 1761238562.294050 trunc from 0x02749e to 0x038c76 (0x117d9 bytes) 000844 1761238562.314141 read 0x025d87 thru 0x034aa2 (0xed1c bytes) 000846 1761238562.327019 read 0x004a77 thru 0x005aab (0x1035 bytes) 000849 1761238562.329797 write 0x0027a8 thru 0x00526e (0x2ac7 bytes) 000851 1761238562.331228 mapwrite 0x003ad1 thru 0x00ae12 (0x7342 bytes) 000852 1761238562.377533 mapread 0x008435 thru 0x010441 (0x800d bytes) 000853 1761238562.378551 read 0x01ac48 thru 0x01e888 (0x3c41 bytes) 000854 1761238562.379970 write 0x0074da thru 0x015b0c (0xe633 bytes) 000855 1761238562.381652 trunc from 0x038c77 to 0x013d69 (0x24f0d bytes) 000860 1761238562.402994 mapwrite 0x03e526 thru 0x03ffff (0x1ada bytes) 000861 1761238562.484000 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000862 1761238562.699732 write 0x03074d thru 0x038c97 (0x854b bytes) 000864 1761238562.713549 trunc from 0x040000 to 0x008f9f (0x37060 bytes) 000866 1761238562.736853 mapread 0x0022c5 thru 0x008f9f (0x6cdb bytes) 000868 1761238562.748054 mapwrite 0x0331e8 thru 0x03a653 (0x746c bytes) 000872 1761238562.817756 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000875 1761238563.018978 MIRROR_RESYNC 6 mirrors 000876 1761238563.995471 write 0x01b7e5 thru 0x01df89 (0x27a5 bytes) 000877 1761238564.068035 mapread 0x037bde thru 0x03a653 (0x2a76 bytes) 000878 1761238564.089999 MIRROR_EXTEND to 7 mirrors 000879 1761238564.686798 MIRROR_RESYNC 7 mirrors 000881 1761238565.666532 MIRROR_EXTEND to 8 mirrors 000883 1761238566.244699 MIRROR_RESYNC 8 mirrors 000884 1761238566.509161 read 0x01093b thru 0x01515d (0x4823 bytes) 000885 1761238566.519982 trunc from 0x03a654 to 0x029787 (0x10ecc bytes) 000886 1761238566.644531 write 0x00136c thru 0x002278 (0xf0d bytes) 000887 1761238566.659398 mapwrite 0x02f61e thru 0x03d54c (0xdf2f bytes) 000890 1761238566.772806 mapread 0x00ee19 thru 0x01a36f (0xb557 bytes) 000893 1761238566.794284 write 0x03f70c thru 0x03ffff (0x8f4 bytes) 000894 1761238566.795210 read 0x002365 thru 0x011ceb (0xf987 bytes) 000895 1761238566.804655 write 0x012e3e thru 0x01991f (0x6ae2 bytes) 000897 1761238566.806253 write 0x0109d6 thru 0x01376d (0x2d98 bytes) 000901 1761238566.808431 write 0x03a680 thru 0x03ea9c (0x441d bytes) 000903 1761238566.809872 mapread 0x01c651 thru 0x0240c4 (0x7a74 bytes) 000904 1761238566.810824 mapwrite 0x02c572 thru 0x030a7c (0x450b bytes) 000906 1761238566.861010 read 0x007c61 thru 0x012c77 (0xb017 bytes) 000907 1761238566.861679 read 0x0045d0 thru 0x007fc8 (0x39f9 bytes) 000911 1761238566.872729 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000912 1761238567.170414 trunc from 0x040000 to 0x02a8ef (0x15710 bytes) 000913 1761238567.207660 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000914 1761238567.425157 trunc from 0x02a8f0 to 0x013c39 (0x16cb6 bytes) 000915 1761238567.441821 read 0x006102 thru 0x00c90f (0x680e bytes) 000917 1761238567.452402 write 0x01cc7e thru 0x027aab (0xae2e bytes) 000919 1761238567.454136 write 0x00cf45 thru 0x00dcda (0xd96 bytes) 000921 1761238567.474397 MIRROR_RESYNC 6 mirrors 000922 1761238568.515784 write 0x02ed6a thru 0x03ac88 (0xbf1f bytes) 000923 1761238568.592107 mapread 0x02d679 thru 0x03ac88 (0xd610 bytes) 000926 1761238568.602223 mapwrite 0x03f3b0 thru 0x03ffff (0xc50 bytes) 000929 1761238568.686656 read 0x018d80 thru 0x023e12 (0xb093 bytes) 000930 1761238568.698026 read 0x01c109 thru 0x022e8e (0x6d86 bytes) 000934 1761238568.700678 mapread 0x001eda thru 0x003996 (0x1abd bytes) 000936 1761238568.712401 trunc from 0x040000 to 0x01a668 (0x25997 bytes) 000937 1761238568.727567 mapread 0x00a02a thru 0x0112ad (0x7284 bytes) 000938 1761238568.728613 mapwrite 0x03139f thru 0x03ffff (0xec61 bytes) 000939 1761238568.881240 MIRROR_EXTEND to 7 mirrors 000941 1761238569.441578 mapwrite 0x027837 thru 0x034cab (0xd475 bytes) 000942 1761238569.629696 write 0x03caf8 thru 0x03ffff (0x3508 bytes) 000943 1761238569.638022 trunc from 0x040000 to 0x00ddce (0x32231 bytes) 000944 1761238569.656563 mapread 0x004b44 thru 0x00a5b3 (0x5a70 bytes) 000945 1761238569.667348 MIRROR_EXTEND to 8 mirrors 000947 1761238570.117080 read 0x0097ea thru 0x00ddce (0x45e5 bytes) 000948 1761238570.130163 write 0x0166c7 thru 0x016c66 (0x5a0 bytes) 000950 1761238570.201315 write 0x0078d4 thru 0x00ee25 (0x7552 bytes) 000953 1761238570.211111 trunc from 0x016c67 to 0x002a79 (0x141ed bytes) 000955 1761238570.258347 mapwrite 0x017deb thru 0x020067 (0x827d bytes) 000957 1761238570.330690 read 0x019273 thru 0x01b64d (0x23db bytes) 000959 1761238570.331611 trunc from 0x020068 to 0x007d10 (0x18357 bytes) 000961 1761238570.346066 read 0x000632 thru 0x000df3 (0x7c2 bytes) 000962 1761238570.356074 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000966 1761238570.556843 trunc from 0x007d11 to 0x00ca60 (0x4d50 bytes) 000967 1761238570.589783 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000968 1761238570.793108 write 0x021217 thru 0x02c766 (0xb550 bytes) 000969 1761238570.794656 mapwrite 0x004bbc thru 0x007e07 (0x324c bytes) 000971 1761238570.858663 mapwrite 0x008c22 thru 0x00c1f9 (0x35d8 bytes) 000972 1761238570.907846 trunc from 0x02c767 to 0x03bee1 (0xf77b bytes) 000975 1761238570.924309 mapwrite 0x034823 thru 0x034c40 (0x41e bytes) 000976 1761238571.003071 MIRROR_EXTEND to 7 mirrors 000980 1761238571.609747 mapwrite 0x026cb3 thru 0x02fdf2 (0x9140 bytes) 000981 1761238571.827477 mapread 0x0055b7 thru 0x011b87 (0xc5d1 bytes) 000985 1761238571.871516 read 0x019b37 thru 0x025ed1 (0xc39b bytes) 000986 1761238571.889685 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) 000989 1761238572.114611 read 0x0375e6 thru 0x03bee1 (0x48fc bytes) 000993 1761238572.126419 write 0x02d789 thru 0x034a92 (0x730a bytes) 000994 1761238572.136716 mapread 0x008904 thru 0x0182cd (0xf9ca bytes) 000996 1761238572.154053 trunc from 0x03bee2 to 0x01da54 (0x1e48d bytes) 000997 1761238572.170670 mapwrite 0x01ec11 thru 0x02a19b (0xb58b bytes) 000998 1761238572.242431 write 0x0281a7 thru 0x035cd6 (0xdb30 bytes) 001000 1761238572.247326 MIRROR_RESYNC 6 mirrors All operations completed A-OK! PASS 70a (69s) == sanity-flr test 71: check mirror extend parent fid ==== 12:56:17 (1761238577) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 12:56:19 (1761238579) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre2 Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2564736 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6642816 .. Extending file size to 9330752 .. Extending file size to 5038368 .. Extending file size to 5319104 .. Extending file size to 5845504 .. Extending file size to 6978720 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8139456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9000800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9232928 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 54915 54917 54918 54919 54920 failed 192.168.206.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre2 (opts:) 192.168.206.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:57:37 (1761238657) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre2 Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4911328 .. Extending file size to 8175040 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7186112 .. Extending file size to 8472064 .. Extending file size to 9027040 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9239552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8908672 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56969 56970 56971 56973 56974 done 192.168.206.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre2 (opts:) 192.168.206.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:58:50 (1761238730) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141055 s, 14.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314964 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:59:04 (1761238744) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0777136 s, 27.0 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 12:59:10 (1761238750) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:59:14 (1761238754) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109711 s, 19.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:59:19 (1761238759) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184678 s, 11.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:59:24 (1761238764) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158965 s, 13.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:59:53 (1761238793) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg629-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999282 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:00:46 (1761238846) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg629-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179963 s, 11.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:01:36 (1761238896) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg629-server mds-ost sync done. Creating new pool test_204f oleg629-server: Pool lustre.test_204f created Adding targets to pool oleg629-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg629-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223044 s, 9.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg629-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg629-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg629-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:02:45 (1761238965) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:02:51 (1761238971) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:02:58 (1761238978) oleg629-server: Pool lustre.test_206 created Adding targets to pool oleg629-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg629-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg629-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg629-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg629-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:03:18 (1761238998) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480072 s, 21.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370008 s, 28.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe52b2871 Mirror 3: 0xe52b2871 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:03:26 (1761239006) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:03:29 (1761239009) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:03:31 (1761239011) Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0952996 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x21f0000 Read this (4 bytes): BBBB 192.168.206.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg629-client.virtnet /mnt/lustre2 (opts:) PASS 209a (27s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:03:58 (1761239038) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00877006 s, 934 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:04:04 (1761239044) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325843 s, 32.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x25c lmm_fid: [0x200000404:0x25c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x25c lmm_fid: [0x200000404:0x25c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834524 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-23 13:04:06.000000000 -0400 Modify: 2025-10-23 13:04:06.000000000 -0400 Change: 2025-10-23 13:04:07.000000000 -0400 Birth: 2025-10-23 13:04:05.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:04:12 (1761239052) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:04:15 (1761239055) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.679874 s, 60.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.086454 s, 47.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 23 13:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 23 13:04 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test complete, duration 1374 sec =========== 13:04:24 (1761239064) === sanity-flr: start cleanup 13:04:26 (1761239066) === === sanity-flr: finish cleanup 13:04:31 (1761239071) ===