-----============= acceptance-small: sanity-flr ============----- Thu Oct 23 14:55:43 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg108-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:56:00 (1761245760) === oleg108-client.virtnet: executing check_config_client /mnt/lustre oleg108-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg108-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d7604508000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d7604508000.idle_timeout=debug disable quota as required oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:56:29 (1761245789) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7556] == sanity-flr test 0a: lfs mirror create with -N option == 14:56:30 (1761245790) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:56:44 (1761245804) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:56:47 (1761245807) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:56:50 (1761245810) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:57:11 (1761245831) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:57:14 (1761245834) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:57:17 (1761245837) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d7604508000.stats=clear osc.lustre-OST0001-osc-ffff9d7604508000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:57:24 (1761245844) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:57:32 (1761245852) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:57:39 (1761245859) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:57:42 (1761245862) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 14:57:49 (1761245869) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:57:55 (1761245875) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:57:58 (1761245878) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:58:05 (1761245885) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:58:14 (1761245894) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:58:19 (1761245899) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.04942 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192169 s, 5.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:58:31 (1761245911) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.150841 s, 34.8 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:58:41 (1761245921) Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (51s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:59:32 (1761245972) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.481445 s, 15.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:00:06 (1761246006) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (61s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:01:07 (1761246067) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.23372 s, 10.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 23 15:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.634s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.002s user 0m0.002s sys 0m0.208s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg108-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg108-server oleg108-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.504s user 0m0.000s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg108-server: oleg108-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg108-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d7604508000.ost_server_uuid in FULL state after 0 sec PASS 33b (82s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:02:29 (1761246149) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:02:32 (1761246152) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:02:35 (1761246155) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:02:38 (1761246158) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:02:46 (1761246166) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.396469 s, 31.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:02:59 (1761246179) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230735 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362499 s, 28.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:03:06 (1761246186) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:03:14 (1761246194) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149663 s, 684 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000820857 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101296 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00837188 s, 122 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:03:25 (1761246205) create test files with size 15753433 .. f55d0ed2f39439b42e3bb28357cef017 - c1246fac113686b31fa43f4ca268d510 - c73f3850da5b0d1f51eaadf789803b77 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 15:03:48 (1761246228) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:04:01 (1761246241) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:04:08 (1761246248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123138 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443793 s, 23.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:04:17 (1761246257) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155562 s, 13.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.420415 s, 10.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181549 s, 11.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 15:04:29 (1761246269) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:04:33 (1761246273) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:04:36 (1761246276) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180556 s, 58.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 15:04:43 (1761246283) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:04:46 (1761246286) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:04:56 (1761246296) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296145 s, 35.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:05:07 (1761246307) llite.lustre-ffff9d762c69a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.217758 s, 48.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff9d760471a000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:05:19 (1761246319) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:05:55 (1761246355) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:06:25 (1761246385) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 15:06:27 (1761246387) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.152076 s, 20.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.165757 s, 19.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 72e8e3b7e2f0c0d6089991781fef29b1 - mirror 1 checksum: 8149869a87607dfeaecbcb54ccf60413 - mirror 2 checksum: 72e8e3b7e2f0c0d6089991781fef29b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 15:06:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 15:06:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 72e8e3b7e2f0c0d6089991781fef29b1 - mirror 1 checksum: 8149869a87607dfeaecbcb54ccf60413 - mirror 2 checksum: 8149869a87607dfeaecbcb54ccf60413 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 15:06:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 15:06:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:06:37 (1761246397) Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 15:06:43 (1761246403) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 15:07:06 (1761246426) targets are mounted 15:07:06 (1761246426) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248484 s, 42.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:07:19 (1761246439) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:07:22 (1761246442) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:07:25 (1761246445) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00922344 s, 1.8 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:07:32 (1761246452) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:07:36 (1761246456) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:07:39 (1761246459) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:08:02 (1761246482) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:08:30 (1761246510) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:08:53 (1761246533) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:09:15 (1761246555) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 15:09:22 (1761246562) mirror create pid 52278 mirror split pid 52279 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7639: 52278 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52279 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 15:10:30 (1761246630) Seed set to 6631 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x31fc5 000002 1761246631.999386 trunc from 00000000 to 0x031fc4 (0x31fc5 bytes) 000003 1761246632.019073 read 0x024d7c thru 0x026f58 (0x21dd bytes) 000004 1761246632.027406 read 0x0318df thru 0x031fc4 (0x6e6 bytes) 000005 1761246632.040660 mapwrite 0x014325 thru 0x017143 (0x2e1f bytes) 000006 1761246632.116552 mapwrite 0x037c8b thru 0x03b6f8 (0x3a6e bytes) 000007 1761246632.246079 read 0x034d33 thru 0x03b6f8 (0x69c6 bytes) 000008 1761246632.250569 MIRROR_EXTEND to 2 mirrors 000010 1761246632.887111 trunc from 0x03b6f9 to 0x01fccf (0x1ba29 bytes) 000011 1761246632.992036 write 0x03d650 thru 0x03ffff (0x29b0 bytes) 000013 1761246632.994623 write 0x02e24e thru 0x02fb42 (0x18f5 bytes) 000014 1761246633.005150 read 0x020f3f thru 0x02ec07 (0xdcc9 bytes) 000016 1761246633.016197 mapwrite 0x0003af thru 0x0093dd (0x902f bytes) 000017 1761246633.107898 write 0x01dfd3 thru 0x02c23c (0xe26a bytes) 000019 1761246633.126975 trunc from 0x040000 to 0x0296a7 (0x16958 bytes) 000020 1761246633.150643 read 0x014e80 thru 0x01d17a (0x82fb bytes) 000024 1761246633.160844 read 0x022274 thru 0x023361 (0x10ee bytes) 000025 1761246633.161539 mapwrite 0x011680 thru 0x015a9a (0x441b bytes) 000026 1761246633.226592 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x34645 000027 1761246633.895800 trunc from 0x0296a8 to 0x034644 (0xaf9d bytes) 000028 1761246634.002164 read 0x00f2bd thru 0x012427 (0x316b bytes) 000029 1761246634.009586 read 0x004c2f thru 0x00d300 (0x86d2 bytes) 000030 1761246634.035373 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000033 1761246634.239711 mapwrite 0x027a6f thru 0x029715 (0x1ca7 bytes) 000035 1761246634.307609 write 0x024280 thru 0x0339a1 (0xf722 bytes) 000038 1761246634.315150 mapwrite 0x000afc thru 0x00de04 (0xd309 bytes) 000041 1761246634.408066 write 0x026725 thru 0x02dcbf (0x759b bytes) 000047 1761246634.413319 mapread 0x00c9d1 thru 0x00f520 (0x2b50 bytes) 000048 1761246634.415230 read 0x0149d4 thru 0x022ed5 (0xe502 bytes) 000049 1761246634.430032 mapwrite 0x02b3b4 thru 0x02b6ba (0x307 bytes) 000052 1761246634.511068 read 0x033e1c thru 0x034644 (0x829 bytes) 000054 1761246634.526281 mapread 0x01c76c thru 0x028cd9 (0xc56e bytes) 000055 1761246634.527462 mapread 0x019e23 thru 0x021365 (0x7543 bytes) 000056 1761246634.533147 trunc from 0x034645 to 0x01a725 (0x19f1f bytes) 000058 1761246634.571321 write 0x02e99b thru 0x03d6b3 (0xed19 bytes) 000059 1761246634.573475 mapwrite 0x038f28 thru 0x03c185 (0x325e bytes) 000060 1761246634.646605 mapwrite 0x003df6 thru 0x00e7c0 (0xa9cb bytes) 000062 1761246634.708889 mapwrite 0x00b45d thru 0x00ff7d (0x4b21 bytes) 000065 1761246634.759076 mapread 0x01fef1 thru 0x0284e5 (0x85f5 bytes) 000066 1761246634.845134 read 0x02603a thru 0x028f63 (0x2f2a bytes) 000068 1761246634.846254 write 0x038001 thru 0x03ffff (0x7fff bytes) 000071 1761246634.849129 read 0x0322fa thru 0x03af1a (0x8c21 bytes) 000072 1761246634.851743 write 0x024ff7 thru 0x0319c0 (0xc9ca bytes) truncating to largest ever: 0x3d4f1 000073 1761246634.856792 trunc from 0x040000 to 0x03d4f0 (0x2b0f bytes) 000074 1761246634.892213 mapread 0x01ef30 thru 0x02b80d (0xc8de bytes) 000075 1761246634.898194 read 0x022cb9 thru 0x0247b4 (0x1afc bytes) 000078 1761246634.899921 mapread 0x02331a thru 0x02ec39 (0xb920 bytes) 000080 1761246634.901974 write 0x02a71c thru 0x033767 (0x904c bytes) 000081 1761246634.904094 write 0x0251ad thru 0x026ab7 (0x190b bytes) 000082 1761246634.909305 mapread 0x00ff09 thru 0x012140 (0x2238 bytes) 000085 1761246634.917885 MIRROR_EXTEND to 2 mirrors 000086 1761246635.703589 mapread 0x002768 thru 0x00ab2b (0x83c4 bytes) 000089 1761246635.725173 trunc from 0x03d4f1 to 0x03082a (0xccc6 bytes) 000092 1761246635.804080 trunc from 0x03082b to 0x0003b9 (0x30471 bytes) 000094 1761246635.847259 MIRROR_EXTEND to 3 mirrors 000098 1761246636.551186 read 0x00022a thru 0x0003b9 (0x190 bytes) 000099 1761246636.596793 MIRROR_RESYNC 3 mirrors 000101 1761246637.301240 mapwrite 0x03dff2 thru 0x03ffff (0x200e bytes) 000104 1761246637.485588 read 0x0220fb thru 0x02bc1b (0x9b21 bytes) 000105 1761246637.495939 mapread 0x030928 thru 0x036923 (0x5ffc bytes) 000106 1761246637.506534 trunc from 0x040000 to 0x00f1c4 (0x30e3b bytes) 000107 1761246637.533665 mapwrite 0x03e4fc thru 0x03ffff (0x1b04 bytes) 000108 1761246637.622654 mapwrite 0x03fcc5 thru 0x03ffff (0x33b bytes) 000110 1761246637.696618 write 0x01ba8f thru 0x026d1c (0xb28e bytes) 000112 1761246637.721471 mapwrite 0x03eec6 thru 0x03ffff (0x113a bytes) 000113 1761246637.797183 write 0x01eada thru 0x02c634 (0xdb5b bytes) 000114 1761246637.814413 write 0x00df35 thru 0x01791b (0x99e7 bytes) 000116 1761246637.825626 mapwrite 0x0121e8 thru 0x013c08 (0x1a21 bytes) 000117 1761246637.897638 read 0x02827c thru 0x0351e1 (0xcf66 bytes) 000119 1761246637.909826 write 0x032bc1 thru 0x03abef (0x802f bytes) 000120 1761246637.910898 trunc from 0x040000 to 0x02a565 (0x15a9a bytes) 000121 1761246637.933554 MIRROR_EXTEND to 4 mirrors 000123 1761246638.584112 read 0x01b55b thru 0x0283ab (0xce51 bytes) 000124 1761246638.592073 mapwrite 0x038000 thru 0x03b175 (0x3176 bytes) 000126 1761246638.750140 mapwrite 0x00e3d7 thru 0x0101e5 (0x1e0f bytes) 000129 1761246638.811362 mapwrite 0x00c067 thru 0x013ab2 (0x7a4c bytes) 000130 1761246638.874207 read 0x00015f thru 0x005cbe (0x5b60 bytes) 000131 1761246638.896518 write 0x01310c thru 0x01589a (0x278f bytes) 000132 1761246638.897478 write 0x00ffa8 thru 0x019ced (0x9d46 bytes) 000135 1761246638.905023 mapread 0x00f9bd thru 0x0123d6 (0x2a1a bytes) 000136 1761246638.905969 mapread 0x02dfd3 thru 0x03b175 (0xd1a3 bytes) 000137 1761246638.906805 mapread 0x01118c thru 0x01628d (0x5102 bytes) 000138 1761246638.908571 trunc from 0x03b176 to 0x00d40f (0x2dd66 bytes) 000141 1761246638.937032 trunc from 0x00d410 to 0x01a9ad (0xd59e bytes) 000142 1761246638.959582 read 0x00a02c thru 0x015056 (0xb02b bytes) 000143 1761246638.969502 write 0x0164b4 thru 0x0215cb (0xb118 bytes) 000144 1761246638.971801 write 0x03b63e thru 0x03eba9 (0x356c bytes) truncating to largest ever: 0x3efe8 000145 1761246638.973968 trunc from 0x03ebaa to 0x03efe7 (0x43e bytes) 000146 1761246639.001743 mapwrite 0x01b952 thru 0x028dbd (0xd46c bytes) 000148 1761246639.071656 read 0x013c06 thru 0x01ee0b (0xb206 bytes) 000149 1761246639.081217 MIRROR_RESYNC 4 mirrors 000151 1761246639.897832 trunc from 0x03efe8 to 0x01108a (0x2df5d bytes) 000152 1761246640.013860 MIRROR_EXTEND to 5 mirrors 000153 1761246640.702123 read 0x003ae3 thru 0x0090cb (0x55e9 bytes) 000157 1761246640.746215 write 0x01c5ce thru 0x01eef1 (0x2924 bytes) 000159 1761246640.848913 MIRROR_EXTEND to 6 mirrors 000166 1761246641.490467 mapwrite 0x026ff1 thru 0x03384d (0xc85d bytes) 000168 1761246641.711762 write 0x02219f thru 0x02c87a (0xa6dc bytes) 000169 1761246641.728762 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000171 1761246642.039573 mapwrite 0x030b4a thru 0x03fd58 (0xf20f bytes) 000174 1761246642.166612 read 0x019b90 thru 0x01efd7 (0x5448 bytes) 000175 1761246642.180041 mapread 0x022656 thru 0x02e4c3 (0xbe6e bytes) 000177 1761246642.193470 mapwrite 0x016b52 thru 0x01feca (0x9379 bytes) 000179 1761246642.256006 mapwrite 0x01f6aa thru 0x0214c0 (0x1e17 bytes) 000180 1761246642.297627 write 0x00c5de thru 0x01260c (0x602f bytes) 000181 1761246642.307225 read 0x021276 thru 0x02dd9e (0xcb29 bytes) 000183 1761246642.313971 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000184 1761246642.564503 trunc from 0x03fd59 to 0x0164e1 (0x29877 bytes) 000185 1761246642.597373 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000188 1761246642.871955 write 0x01b843 thru 0x025ac9 (0xa287 bytes) 000189 1761246642.873224 read 0x0252b0 thru 0x025ac9 (0x81a bytes) 000190 1761246642.873847 trunc from 0x025aca to 0x024541 (0x1588 bytes) 000192 1761246642.885967 mapread 0x008ce5 thru 0x01286d (0x9b89 bytes) 000193 1761246642.905233 mapread 0x01902c thru 0x01c5ca (0x359f bytes) 000194 1761246642.930569 mapread 0x013647 thru 0x01d905 (0xa2bf bytes) 000196 1761246642.931936 read 0x023a06 thru 0x024541 (0xb3c bytes) 000197 1761246642.932505 trunc from 0x024542 to 0x025807 (0x12c6 bytes) 000200 1761246642.963412 mapwrite 0x02aab1 thru 0x02b61c (0xb6c bytes) 000201 1761246643.054295 MIRROR_RESYNC 3 mirrors 000202 1761246643.821481 trunc from 0x02b61d to 0x03584d (0xa231 bytes) 000203 1761246643.914331 MIRROR_RESYNC 3 mirrors 000204 1761246644.781378 write 0x00e461 thru 0x011282 (0x2e22 bytes) 000206 1761246644.852008 mapread 0x0090fd thru 0x0136f5 (0xa5f9 bytes) 000207 1761246644.875069 trunc from 0x03584e to 0x03ac81 (0x5434 bytes) 000210 1761246644.906359 mapwrite 0x0377c4 thru 0x03a7b0 (0x2fed bytes) 000214 1761246644.990358 MIRROR_EXTEND to 4 mirrors 000216 1761246645.713818 write 0x008723 thru 0x00d700 (0x4fde bytes) 000217 1761246645.835880 mapread 0x022b44 thru 0x029abd (0x6f7a bytes) 000218 1761246645.856548 read 0x017e69 thru 0x01b0a9 (0x3241 bytes) 000219 1761246645.875095 MIRROR_RESYNC 4 mirrors 000220 1761246646.742567 mapwrite 0x019ff0 thru 0x020664 (0x6675 bytes) 000221 1761246646.927270 trunc from 0x03ac82 to 0x0245a5 (0x166dc bytes) 000223 1761246646.965728 MIRROR_RESYNC 4 mirrors 000225 1761246647.858413 write 0x017f67 thru 0x021fc9 (0xa063 bytes) 000228 1761246647.938494 trunc from 0x0245a6 to 0x001b27 (0x22a7e bytes) 000229 1761246647.965055 write 0x006690 thru 0x00e69d (0x800e bytes) 000230 1761246647.966300 read 0x002439 thru 0x00e69d (0xc265 bytes) 000233 1761246647.975050 trunc from 0x00e69e to 0x003b3f (0xab5e bytes) 000234 1761246647.995572 mapread 0x000579 thru 0x003b3f (0x35c7 bytes) 000237 1761246648.002597 trunc from 0x003b40 to 0x027cc4 (0x24185 bytes) 000239 1761246648.022683 write 0x009627 thru 0x01907c (0xfa56 bytes) 000240 1761246648.060683 read 0x008f0d thru 0x015c96 (0xcd8a bytes) 000241 1761246648.061313 trunc from 0x027cc5 to 0x0163c5 (0x118ff bytes) 000244 1761246648.087623 trunc from 0x0163c6 to 0x017fd4 (0x1c0f bytes) 000245 1761246648.112855 trunc from 0x017fd5 to 0x0143b5 (0x3c1f bytes) 000246 1761246648.133894 trunc from 0x0143b6 to 0x01b8c4 (0x750f bytes) 000247 1761246648.149931 trunc from 0x01b8c5 to 0x01fff0 (0x472c bytes) 000248 1761246648.180318 mapwrite 0x0244d6 thru 0x0267a8 (0x22d3 bytes) 000249 1761246648.306315 mapread 0x00acb1 thru 0x011ffd (0x734d bytes) 000252 1761246648.313729 MIRROR_EXTEND to 5 mirrors 000254 1761246648.834850 mapread 0x01cb1b thru 0x0267a8 (0x9c8e bytes) 000255 1761246648.862832 mapwrite 0x02d6fa thru 0x03cedb (0xf7e2 bytes) 000258 1761246649.080057 read 0x03bf5f thru 0x03cedb (0xf7d bytes) 000260 1761246649.081126 read 0x0260d1 thru 0x02b8d7 (0x5807 bytes) 000261 1761246649.081957 write 0x03cacb thru 0x03ffff (0x3535 bytes) 000262 1761246649.082962 mapwrite 0x037c8d thru 0x03ffff (0x8373 bytes) 000263 1761246649.143360 mapread 0x00a46c thru 0x00d429 (0x2fbe bytes) 000264 1761246649.155773 mapread 0x01bb79 thru 0x01daf9 (0x1f81 bytes) 000265 1761246649.156591 trunc from 0x040000 to 0x009167 (0x36e98 bytes) 000266 1761246649.189865 read 0x002a35 thru 0x009167 (0x6733 bytes) 000267 1761246649.190513 read 0x00869a thru 0x009167 (0xace bytes) 000268 1761246649.191013 write 0x035053 thru 0x03806e (0x301c bytes) 000269 1761246649.191971 trunc from 0x03806f to 0x02fb8d (0x84e1 bytes) 000270 1761246649.221798 mapwrite 0x00e9bb thru 0x01ca03 (0xe049 bytes) 000271 1761246649.301665 trunc from 0x02fb8e to 0x02d186 (0x2a07 bytes) 000272 1761246649.331058 mapwrite 0x004288 thru 0x008f93 (0x4d0c bytes) 000274 1761246649.394668 mapwrite 0x036208 thru 0x03b6f0 (0x54e9 bytes) 000276 1761246649.510019 mapwrite 0x00b51b thru 0x01ad76 (0xf85c bytes) 000277 1761246649.569197 trunc from 0x03b6f1 to 0x01998a (0x21d66 bytes) 000278 1761246649.592349 mapread 0x01642b thru 0x01998a (0x3560 bytes) 000279 1761246649.594252 mapwrite 0x015103 thru 0x0217a2 (0xc6a0 bytes) 000283 1761246649.724798 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000285 1761246650.061797 MIRROR_EXTEND to 5 mirrors 000286 1761246650.705036 read 0x01e83f thru 0x0217a2 (0x2f64 bytes) 000287 1761246650.717113 read 0x00d49e thru 0x010a36 (0x3599 bytes) 000288 1761246650.746740 mapwrite 0x03f6ee thru 0x03ffff (0x912 bytes) 000290 1761246650.937448 mapread 0x01a40c thru 0x02466e (0xa263 bytes) 000293 1761246650.998220 mapread 0x01bc1e thru 0x022a0b (0x6dee bytes) 000296 1761246650.999781 trunc from 0x040000 to 0x00a143 (0x35ebc bytes) 000298 1761246651.029850 trunc from 0x00a144 to 0x01443a (0xa2f7 bytes) 000299 1761246651.057783 MIRROR_RESYNC 5 mirrors 000301 1761246652.144793 MIRROR_EXTEND to 6 mirrors 000302 1761246652.844269 read 0x013580 thru 0x01443a (0xebb bytes) 000303 1761246652.851134 mapwrite 0x00cc87 thru 0x00ebfb (0x1f75 bytes) 000305 1761246653.050862 write 0x0395bb thru 0x03b246 (0x1c8c bytes) 000306 1761246653.056910 MIRROR_RESYNC 6 mirrors 000308 1761246654.179072 trunc from 0x03b247 to 0x0348f0 (0x6956 bytes) 000309 1761246654.308411 read 0x0139d7 thru 0x017f02 (0x452c bytes) 000310 1761246654.323818 mapwrite 0x0257df thru 0x03554f (0xfd71 bytes) 000311 1761246654.461347 mapwrite 0x00a276 thru 0x00d8e0 (0x366b bytes) 000312 1761246654.514289 read 0x007c4c thru 0x00c548 (0x48fd bytes) 000313 1761246654.515733 mapwrite 0x028162 thru 0x0351dd (0xd07c bytes) 000315 1761246654.568280 mapread 0x020e45 thru 0x02540b (0x45c7 bytes) 000317 1761246654.570287 mapread 0x026bd9 thru 0x034705 (0xdb2d bytes) 000319 1761246654.572617 write 0x01b1d2 thru 0x01bee7 (0xd16 bytes) 000320 1761246654.580180 mapwrite 0x02157e thru 0x02f182 (0xdc05 bytes) 000322 1761246654.639914 mapread 0x00f7d6 thru 0x01d625 (0xde50 bytes) 000325 1761246654.641607 mapread 0x002c66 thru 0x008b7a (0x5f15 bytes) 000326 1761246654.642301 mapread 0x01cbe9 thru 0x022b35 (0x5f4d bytes) 000327 1761246654.643531 mapwrite 0x0142f3 thru 0x014d9c (0xaaa bytes) 000329 1761246654.689412 mapread 0x00f99e thru 0x01ea31 (0xf094 bytes) 000330 1761246654.690334 trunc from 0x035550 to 0x02f340 (0x620f bytes) 000331 1761246654.714168 mapread 0x01175e thru 0x016ac2 (0x5365 bytes) 000332 1761246654.715057 trunc from 0x02f341 to 0x015dc0 (0x19580 bytes) 000333 1761246654.741535 mapwrite 0x0179f6 thru 0x024cee (0xd2f9 bytes) 000334 1761246654.891581 mapread 0x016ae7 thru 0x018dd3 (0x22ed bytes) 000336 1761246654.905309 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000339 1761246655.135812 read 0x014d68 thru 0x020461 (0xb6fa bytes) 000342 1761246655.147900 read 0x014cc6 thru 0x0211b5 (0xc4f0 bytes) 000343 1761246655.175000 read 0x01ee80 thru 0x024cee (0x5e6f bytes) 000345 1761246655.175970 read 0x01665d thru 0x019076 (0x2a1a bytes) 000348 1761246655.177340 mapwrite 0x024345 thru 0x02e929 (0xa5e5 bytes) 000349 1761246655.267921 write 0x0083b3 thru 0x014e12 (0xca60 bytes) 000351 1761246655.269657 mapread 0x027abc thru 0x02e929 (0x6e6e bytes) 000355 1761246655.273249 read 0x012333 thru 0x020de1 (0xeaaf bytes) 000356 1761246655.275044 mapwrite 0x015723 thru 0x0206ba (0xaf98 bytes) 000357 1761246655.331945 write 0x036096 thru 0x03dcc8 (0x7c33 bytes) 000358 1761246655.333322 read 0x0128ff thru 0x01a8a1 (0x7fa3 bytes) 000359 1761246655.334928 mapread 0x0238ea thru 0x0315da (0xdcf1 bytes) 000361 1761246655.346888 trunc from 0x03dcc9 to 0x030ef5 (0xcdd3 bytes) 000362 1761246655.389713 read 0x022e14 thru 0x0253ed (0x25da bytes) 000364 1761246655.392277 write 0x022310 thru 0x0267a3 (0x4494 bytes) 000366 1761246655.394187 read 0x012729 thru 0x0225ca (0xfea2 bytes) 000367 1761246655.396196 trunc from 0x030ef6 to 0x03eedf (0xdfea bytes) 000368 1761246655.424842 mapread 0x00985e thru 0x00e734 (0x4ed7 bytes) 000370 1761246655.427826 trunc from 0x03eee0 to 0x028759 (0x16786 bytes) 000371 1761246655.452933 write 0x02ec27 thru 0x03519f (0x6579 bytes) 000374 1761246655.456756 read 0x014391 thru 0x023878 (0xf4e8 bytes) 000375 1761246655.458551 mapread 0x025c8f thru 0x034e93 (0xf205 bytes) 000377 1761246655.472241 trunc from 0x0351a0 to 0x0235af (0x11bf0 bytes) 000379 1761246655.493609 mapread 0x016aa9 thru 0x01747c (0x9d4 bytes) 000380 1761246655.494459 read 0x014212 thru 0x01bded (0x7bdc bytes) 000381 1761246655.494910 mapread 0x01d545 thru 0x0235af (0x606b bytes) 000383 1761246655.496152 read 0x01354c thru 0x0176ca (0x417f bytes) 000385 1761246655.497338 mapread 0x023429 thru 0x0235af (0x187 bytes) 000386 1761246655.498340 trunc from 0x0235b0 to 0x01c0d5 (0x74da bytes) 000387 1761246655.511783 trunc from 0x01c0d6 to 0x00bc22 (0x104b3 bytes) 000388 1761246655.540514 MIRROR_RESYNC 5 mirrors 000389 1761246656.552069 mapwrite 0x022bd5 thru 0x026375 (0x37a1 bytes) 000392 1761246656.740018 mapwrite 0x021ccd thru 0x02b213 (0x9547 bytes) 000393 1761246656.818940 mapwrite 0x0006eb thru 0x007829 (0x713f bytes) 000394 1761246656.926568 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000395 1761246657.166179 mapwrite 0x03b5a0 thru 0x03ffff (0x4a60 bytes) 000396 1761246657.257560 mapread 0x001270 thru 0x01061d (0xf3ae bytes) 000397 1761246657.275707 trunc from 0x040000 to 0x035b3e (0xa4c1 bytes) 000399 1761246657.294834 trunc from 0x035b3f to 0x02ecac (0x6e92 bytes) 000401 1761246657.314775 write 0x00e04f thru 0x01bba0 (0xdb52 bytes) 000402 1761246657.321239 mapwrite 0x000f19 thru 0x00b08d (0xa175 bytes) 000403 1761246657.384884 trunc from 0x02ecad to 0x017365 (0x17947 bytes) 000405 1761246657.403094 trunc from 0x017366 to 0x024da6 (0xda41 bytes) 000407 1761246657.426090 write 0x0225ee thru 0x02c1a7 (0x9bba bytes) 000408 1761246657.437140 mapwrite 0x03bdf2 thru 0x03ffff (0x420e bytes) 000409 1761246657.528742 trunc from 0x040000 to 0x020d46 (0x1f2b9 bytes) 000410 1761246657.551831 trunc from 0x020d47 to 0x002341 (0x1ea05 bytes) 000415 1761246657.574278 mapwrite 0x03b2aa thru 0x03ffff (0x4d56 bytes) 000416 1761246657.683705 mapwrite 0x00d90b thru 0x016119 (0x880f bytes) 000417 1761246657.757029 mapwrite 0x025717 thru 0x02cdb2 (0x769c bytes) 000418 1761246657.816067 mapread 0x001605 thru 0x010b0b (0xf507 bytes) 000419 1761246657.824742 mapwrite 0x03ea61 thru 0x03ffff (0x159f bytes) 000420 1761246657.875146 read 0x0193d0 thru 0x025bd7 (0xc808 bytes) 000421 1761246657.883304 MIRROR_EXTEND to 5 mirrors 000422 1761246658.579754 MIRROR_EXTEND to 6 mirrors 000423 1761246659.330125 mapwrite 0x01e567 thru 0x0243eb (0x5e85 bytes) 000424 1761246659.593035 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000425 1761246659.864038 mapwrite 0x033e8f thru 0x037f53 (0x40c5 bytes) 000426 1761246659.977187 mapwrite 0x01c11b thru 0x0229b5 (0x689b bytes) 000428 1761246660.058314 read 0x0124a5 thru 0x01ad52 (0x88ae bytes) 000429 1761246660.065156 write 0x03f331 thru 0x03ffff (0xccf bytes) 000430 1761246660.072248 mapread 0x03f668 thru 0x03f72b (0xc4 bytes) 000431 1761246660.073172 trunc from 0x040000 to 0x0138c6 (0x2c739 bytes) 000432 1761246660.099600 write 0x021094 thru 0x02f3df (0xe34c bytes) 000434 1761246660.101374 read 0x0287be thru 0x02f3df (0x6c22 bytes) 000437 1761246660.102962 mapwrite 0x00e947 thru 0x01e36a (0xfa24 bytes) 000439 1761246660.177318 write 0x012b59 thru 0x01952c (0x69d4 bytes) 000440 1761246660.179639 read 0x015535 thru 0x018053 (0x2b1f bytes) 000441 1761246660.187638 MIRROR_EXTEND to 6 mirrors 000443 1761246660.883797 mapwrite 0x03b1a3 thru 0x03be9f (0xcfd bytes) 000445 1761246661.064644 trunc from 0x03bea0 to 0x02da3e (0xe461 bytes) 000449 1761246661.094915 trunc from 0x02da3f to 0x027e47 (0x5bf7 bytes) 000450 1761246661.109643 read 0x01c0ff thru 0x027e47 (0xbd49 bytes) 000452 1761246661.120415 mapwrite 0x03b597 thru 0x03ffff (0x4a69 bytes) 000454 1761246661.224795 write 0x004331 thru 0x005606 (0x12d6 bytes) 000455 1761246661.235489 read 0x000321 thru 0x0070db (0x6dbb bytes) 000458 1761246661.248833 read 0x0127aa thru 0x014d97 (0x25ee bytes) 000462 1761246661.251270 mapwrite 0x037ae4 thru 0x03ffff (0x851c bytes) 000463 1761246661.309211 write 0x003090 thru 0x00c8f4 (0x9865 bytes) 000464 1761246661.310374 write 0x022741 thru 0x02e62d (0xbeed bytes) 000465 1761246661.311757 mapwrite 0x035407 thru 0x03a376 (0x4f70 bytes) 000467 1761246661.374303 mapread 0x0397db thru 0x03ffff (0x6825 bytes) 000469 1761246661.375571 mapwrite 0x027502 thru 0x036aed (0xf5ec bytes) 000470 1761246661.437482 write 0x02e9d5 thru 0x02f590 (0xbbc bytes) 000471 1761246661.444993 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000475 1761246661.746305 MIRROR_EXTEND to 6 mirrors 000477 1761246662.366314 trunc from 0x040000 to 0x01cad3 (0x2352c bytes) 000478 1761246662.452026 mapread 0x005bcc thru 0x007370 (0x17a5 bytes) 000479 1761246662.475606 trunc from 0x01cad4 to 0x0103cb (0xc708 bytes) 000480 1761246662.514041 read 0x00a571 thru 0x0103cb (0x5e5b bytes) 000481 1761246662.514688 write 0x0002b0 thru 0x0057a8 (0x54f9 bytes) 000482 1761246662.528001 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000483 1761246662.783745 MIRROR_EXTEND to 6 mirrors 000484 1761246663.520720 mapwrite 0x03531a thru 0x035e93 (0xb7a bytes) 000485 1761246663.703802 write 0x020813 thru 0x026a74 (0x6262 bytes) 000486 1761246663.719595 mapread 0x02c4a3 thru 0x02e7ed (0x234b bytes) 000489 1761246663.730216 write 0x02fe36 thru 0x03efc9 (0xf194 bytes) 000490 1761246663.736896 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000493 1761246664.020756 mapread 0x03abb5 thru 0x03efc9 (0x4415 bytes) 000495 1761246664.048005 MIRROR_EXTEND to 6 mirrors 000497 1761246664.763032 read 0x0350bd thru 0x03bbac (0x6af0 bytes) 000499 1761246664.782930 trunc from 0x03efca to 0x00eede (0x300eb bytes) 000500 1761246664.869917 read 0x0083c1 thru 0x00eede (0x6b1e bytes) 000501 1761246664.883358 read 0x00c212 thru 0x00eede (0x2ccd bytes) 000502 1761246664.884091 trunc from 0x00eedf to 0x0392c5 (0x2a3e7 bytes) 000503 1761246664.911204 trunc from 0x0392c6 to 0x012c42 (0x26683 bytes) 000505 1761246664.940861 mapwrite 0x027428 thru 0x0328f0 (0xb4c9 bytes) 000507 1761246665.106585 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000508 1761246665.372589 trunc from 0x0328f1 to 0x013055 (0x1f89b bytes) 000509 1761246665.406300 mapwrite 0x023dad thru 0x0316e0 (0xd934 bytes) 000513 1761246665.574982 read 0x01bc26 thru 0x01dcfe (0x20d9 bytes) 000515 1761246665.586258 mapwrite 0x00bc85 thru 0x00f819 (0x3b95 bytes) 000516 1761246665.647376 read 0x006e1c thru 0x00ace0 (0x3ec5 bytes) 000517 1761246665.652153 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000519 1761246665.960368 read 0x0283ac thru 0x0295bc (0x1211 bytes) 000520 1761246665.969116 read 0x011cf8 thru 0x01bb24 (0x9e2d bytes) 000521 1761246665.996887 MIRROR_EXTEND to 5 mirrors 000522 1761246666.792144 mapread 0x01f5dc thru 0x0298f4 (0xa319 bytes) 000523 1761246666.855314 write 0x000a39 thru 0x00ef37 (0xe4ff bytes) 000524 1761246666.954587 mapread 0x011010 thru 0x01712b (0x611c bytes) 000527 1761246666.967191 write 0x036681 thru 0x03ffff (0x997f bytes) 000531 1761246666.970341 read 0x00eb8c thru 0x012b6e (0x3fe3 bytes) 000534 1761246666.984339 MIRROR_EXTEND to 6 mirrors 000535 1761246667.671163 write 0x00c471 thru 0x00d292 (0xe22 bytes) 000536 1761246667.779403 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000538 1761246668.110382 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000539 1761246668.447817 read 0x023939 thru 0x030aa4 (0xd16c bytes) 000540 1761246668.467891 mapread 0x008b95 thru 0x016636 (0xdaa2 bytes) 000541 1761246668.522879 mapwrite 0x015d27 thru 0x020d43 (0xb01d bytes) 000543 1761246668.620014 trunc from 0x040000 to 0x002190 (0x3de6f bytes) 000544 1761246668.640820 trunc from 0x002191 to 0x02695f (0x247cf bytes) 000546 1761246668.660772 mapread 0x0036ba thru 0x00bf3a (0x8881 bytes) 000547 1761246668.674297 mapread 0x017af2 thru 0x0264fd (0xea0c bytes) 000549 1761246668.694617 mapread 0x01ac2e thru 0x020d3e (0x6111 bytes) 000550 1761246668.695461 read 0x010138 thru 0x018270 (0x8139 bytes) 000551 1761246668.696023 write 0x02ecba thru 0x03aa13 (0xbd5a bytes) 000552 1761246668.703332 MIRROR_EXTEND to 5 mirrors 000553 1761246669.605770 trunc from 0x03aa14 to 0x00ba19 (0x2effa bytes) 000554 1761246669.707693 mapwrite 0x02b458 thru 0x032908 (0x74b1 bytes) 000555 1761246669.825014 mapwrite 0x0022aa thru 0x00f4d4 (0xd22b bytes) 000556 1761246669.954255 mapwrite 0x00c706 thru 0x0112cd (0x4bc8 bytes) 000558 1761246670.044893 mapwrite 0x0156b3 thru 0x023a3b (0xe389 bytes) 000560 1761246670.173341 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000561 1761246670.527669 read 0x008569 thru 0x00bd01 (0x3799 bytes) 000562 1761246670.538409 mapwrite 0x0082c7 thru 0x017980 (0xf6ba bytes) 000563 1761246670.668231 MIRROR_EXTEND to 5 mirrors 000564 1761246671.556120 mapwrite 0x032d2e thru 0x03ffff (0xd2d2 bytes) 000565 1761246671.745276 mapread 0x02b33a thru 0x0385cb (0xd292 bytes) 000568 1761246671.766036 MIRROR_RESYNC 5 mirrors 000570 1761246672.805967 trunc from 0x040000 to 0x0008b2 (0x3f74d bytes) 000573 1761246672.894752 read 0x000702 thru 0x0008b2 (0x1b1 bytes) 000574 1761246672.907259 mapread 0x00064c thru 0x0008b2 (0x267 bytes) 000575 1761246672.909543 mapread 0x000480 thru 0x0008b2 (0x433 bytes) 000576 1761246672.911089 trunc from 0x0008b3 to 0x021921 (0x2106f bytes) 000577 1761246672.945180 write 0x039a16 thru 0x03f038 (0x5623 bytes) 000578 1761246672.946337 write 0x03ddec thru 0x03ef80 (0x1195 bytes) 000579 1761246672.947084 trunc from 0x03f039 to 0x00b335 (0x33d03 bytes) 000580 1761246672.984079 write 0x00e9d7 thru 0x0131e1 (0x480b bytes) 000581 1761246672.988197 mapwrite 0x011264 thru 0x017932 (0x66cf bytes) 000582 1761246673.087704 trunc from 0x017933 to 0x034f42 (0x1d610 bytes) 000584 1761246673.109981 mapread 0x001c9f thru 0x00df73 (0xc2d5 bytes) 000585 1761246673.121117 mapwrite 0x033467 thru 0x036e41 (0x39db bytes) 000586 1761246673.229254 mapwrite 0x00b04a thru 0x0154e5 (0xa49c bytes) 000588 1761246673.320518 mapwrite 0x033016 thru 0x03ece8 (0xbcd3 bytes) 000589 1761246673.411727 trunc from 0x03ece9 to 0x03bbdd (0x310b bytes) 000590 1761246673.435219 MIRROR_EXTEND to 6 mirrors 000591 1761246674.274888 mapwrite 0x02b660 thru 0x0346f2 (0x9093 bytes) 000593 1761246674.498666 read 0x018fc9 thru 0x01e0c9 (0x5101 bytes) 000599 1761246674.512712 mapread 0x00d4ca thru 0x01114f (0x3c86 bytes) 000601 1761246674.521853 read 0x02a884 thru 0x0389f2 (0xe16f bytes) 000604 1761246674.533857 mapwrite 0x03ec6d thru 0x03ffff (0x1393 bytes) 000606 1761246674.610995 MIRROR_RESYNC 6 mirrors 000610 1761246675.916966 mapread 0x024ce7 thru 0x033d2c (0xf046 bytes) 000613 1761246675.966131 mapwrite 0x0044a5 thru 0x009720 (0x527c bytes) 000615 1761246676.163054 read 0x00939e thru 0x00e1f2 (0x4e55 bytes) 000617 1761246676.164122 mapread 0x01584b thru 0x016d06 (0x14bc bytes) 000618 1761246676.184896 write 0x00e5f6 thru 0x016ade (0x84e9 bytes) 000619 1761246676.186298 write 0x032c02 thru 0x03d04c (0xa44b bytes) 000620 1761246676.213725 write 0x02e5a0 thru 0x03e125 (0xfb86 bytes) 000621 1761246676.228587 write 0x00d32c thru 0x017464 (0xa139 bytes) 000622 1761246676.240249 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000623 1761246676.627984 mapread 0x03daab thru 0x03ffff (0x2555 bytes) 000627 1761246676.656286 write 0x02a87f thru 0x02fe45 (0x55c7 bytes) 000628 1761246676.669757 mapwrite 0x007869 thru 0x0087d0 (0xf68 bytes) 000629 1761246676.741160 mapread 0x02a635 thru 0x034aba (0xa486 bytes) 000631 1761246676.750869 read 0x039361 thru 0x03ffff (0x6c9f bytes) 000633 1761246676.757012 MIRROR_EXTEND to 6 mirrors 000637 1761246677.668005 read 0x02b3a7 thru 0x03767b (0xc2d5 bytes) 000638 1761246677.695398 mapwrite 0x0171c5 thru 0x019120 (0x1f5c bytes) 000639 1761246677.948863 mapwrite 0x0354c7 thru 0x03e4dd (0x9017 bytes) 000640 1761246678.063968 read 0x03655e thru 0x03b1d5 (0x4c78 bytes) 000641 1761246678.067574 mapwrite 0x026808 thru 0x02aac5 (0x42be bytes) 000642 1761246678.122836 mapwrite 0x012af3 thru 0x01c30e (0x981c bytes) 000643 1761246678.187955 MIRROR_EXTEND to 7 mirrors 000646 1761246678.922525 mapwrite 0x0360c9 thru 0x03ffff (0x9f37 bytes) 000647 1761246679.093503 write 0x0000e2 thru 0x004af8 (0x4a17 bytes) 000650 1761246679.102873 write 0x0004b6 thru 0x007cc8 (0x7813 bytes) 000651 1761246679.106937 mapwrite 0x0155c4 thru 0x01d6ec (0x8129 bytes) 000654 1761246679.175140 mapread 0x02632c thru 0x031f59 (0xbc2e bytes) 000655 1761246679.200375 write 0x01d2ac thru 0x01ea08 (0x175d bytes) 000656 1761246679.208280 trunc from 0x040000 to 0x038251 (0x7dae bytes) 000658 1761246679.272174 mapwrite 0x024758 thru 0x02c6b9 (0x7f62 bytes) 000660 1761246679.333207 mapwrite 0x013813 thru 0x016e2f (0x361d bytes) 000661 1761246679.386906 MIRROR_RESYNC 7 mirrors 000662 1761246680.636747 mapwrite 0x03e86a thru 0x03ffff (0x1796 bytes) 000664 1761246680.797932 read 0x012914 thru 0x018952 (0x603f bytes) 000666 1761246680.806557 trunc from 0x040000 to 0x0188bc (0x27743 bytes) 000667 1761246680.830875 read 0x018042 thru 0x0188bc (0x87b bytes) 000669 1761246680.832197 mapwrite 0x00c545 thru 0x01488a (0x8346 bytes) 000671 1761246680.900752 read 0x0004c2 thru 0x000af4 (0x633 bytes) 000672 1761246680.901569 read 0x002b17 thru 0x009e2b (0x7315 bytes) 000674 1761246680.902772 read 0x00a7a9 thru 0x0188bc (0xe114 bytes) 000676 1761246680.904204 write 0x03fbb4 thru 0x03ffff (0x44c bytes) 000680 1761246680.913035 write 0x003ca6 thru 0x00aea6 (0x7201 bytes) 000681 1761246680.916336 write 0x0378bd thru 0x039ffc (0x2740 bytes) 000683 1761246680.929866 mapwrite 0x003d0c thru 0x00fe29 (0xc11e bytes) 000684 1761246681.014993 mapwrite 0x021a84 thru 0x031676 (0xfbf3 bytes) 000685 1761246681.101990 trunc from 0x040000 to 0x0314d9 (0xeb26 bytes) 000688 1761246681.140481 mapwrite 0x03a0ae thru 0x03f3c2 (0x5315 bytes) 000690 1761246681.273994 MIRROR_EXTEND to 8 mirrors 000693 1761246681.812993 trunc from 0x03f3c3 to 0x0356d8 (0x9cea bytes) 000694 1761246681.945750 trunc from 0x0356d9 to 0x008be8 (0x2caf0 bytes) 000695 1761246681.970414 read 0x002754 thru 0x008be8 (0x6495 bytes) 000697 1761246681.982367 MIRROR_RESYNC 8 mirrors 000698 1761246683.240404 write 0x032c6b thru 0x03c5d0 (0x9966 bytes) 000700 1761246683.321770 write 0x00ca8b thru 0x00e69e (0x1c14 bytes) 000702 1761246683.345394 MIRROR_EXTEND to 9 mirrors 000703 1761246684.114726 read 0x0041df thru 0x006a8c (0x28ae bytes) 000704 1761246684.128474 trunc from 0x03c5d1 to 0x01ef50 (0x1d680 bytes) 000705 1761246684.239694 MIRROR_RESYNC 9 mirrors 000706 1761246685.556319 trunc from 0x01ef51 to 0x0353ad (0x1645d bytes) 000707 1761246685.656806 trunc from 0x0353ae to 0x011944 (0x23a69 bytes) 000709 1761246685.677640 read 0x0007ab thru 0x0102ee (0xfb44 bytes) 000710 1761246685.694346 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000711 1761246685.963863 trunc from 0x011945 to 0x01d0cc (0xb788 bytes) 000712 1761246685.981345 mapwrite 0x01d5f1 thru 0x023724 (0x6134 bytes) 000714 1761246686.094797 mapread 0x00425e thru 0x00bc8a (0x7a2d bytes) 000715 1761246686.114355 mapread 0x012a54 thru 0x01c94b (0x9ef8 bytes) 000717 1761246686.144339 mapread 0x01707f thru 0x023724 (0xc6a6 bytes) 000718 1761246686.146044 mapwrite 0x035f2a thru 0x036baa (0xc81 bytes) 000719 1761246686.227712 mapwrite 0x03246f thru 0x0398fd (0x748f bytes) 000726 1761246686.308556 mapread 0x02cde7 thru 0x033b5e (0x6d78 bytes) 000727 1761246686.326132 mapread 0x026cc3 thru 0x030281 (0x95bf bytes) 000730 1761246686.328799 read 0x004e29 thru 0x0079f1 (0x2bc9 bytes) 000732 1761246686.329841 mapwrite 0x034828 thru 0x035703 (0xedc bytes) 000733 1761246686.366520 mapwrite 0x02151a thru 0x0236f1 (0x21d8 bytes) 000734 1761246686.412735 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000737 1761246686.692637 MIRROR_RESYNC 7 mirrors 000739 1761246687.879060 mapread 0x0030f0 thru 0x00da3e (0xa94f bytes) 000740 1761246687.903739 trunc from 0x0398fe to 0x01452a (0x253d3 bytes) 000743 1761246687.992269 mapread 0x00b8ea thru 0x01287c (0x6f93 bytes) 000745 1761246688.002020 read 0x00f955 thru 0x01452a (0x4bd6 bytes) 000747 1761246688.016106 read 0x001360 thru 0x00f423 (0xe0c4 bytes) 000749 1761246688.024684 write 0x02bb06 thru 0x034cfc (0x91f7 bytes) 000750 1761246688.042129 MIRROR_RESYNC 7 mirrors 000753 1761246689.108531 read 0x002efd thru 0x00e605 (0xb709 bytes) 000755 1761246689.119379 read 0x026b30 thru 0x02c4aa (0x597b bytes) 000757 1761246689.130369 trunc from 0x034cfd to 0x029552 (0xb7aa bytes) 000759 1761246689.296881 write 0x03577d thru 0x03ffff (0xa883 bytes) 000760 1761246689.304183 trunc from 0x040000 to 0x02b6e9 (0x14916 bytes) 000761 1761246689.341834 mapwrite 0x02b77b thru 0x039449 (0xdccf bytes) 000762 1761246689.495702 write 0x0078a3 thru 0x010ae6 (0x9244 bytes) 000763 1761246689.517563 MIRROR_RESYNC 7 mirrors 000764 1761246690.647594 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000765 1761246691.081673 mapread 0x001d7f thru 0x00c97f (0xac01 bytes) 000768 1761246691.112073 mapread 0x00c086 thru 0x016047 (0x9fc2 bytes) 000769 1761246691.140737 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000770 1761246691.557687 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000771 1761246691.953319 MIRROR_EXTEND to 5 mirrors 000772 1761246692.618023 mapwrite 0x03e6e3 thru 0x03ffff (0x191d bytes) 000774 1761246692.814706 mapread 0x01ecab thru 0x028e8a (0xa1e0 bytes) 000775 1761246692.842281 mapread 0x005bd7 thru 0x010d16 (0xb140 bytes) 000777 1761246692.856092 write 0x0356fa thru 0x03bf6d (0x6874 bytes) 000779 1761246692.870673 trunc from 0x040000 to 0x000bb7 (0x3f448 bytes) 000781 1761246692.903799 trunc from 0x000bb8 to 0x034200 (0x33649 bytes) 000783 1761246692.955290 MIRROR_RESYNC 5 mirrors 000785 1761246694.009250 mapread 0x032a1a thru 0x034200 (0x17e7 bytes) 000786 1761246694.025371 trunc from 0x034201 to 0x011ad5 (0x2272b bytes) 000787 1761246694.109601 mapwrite 0x018c11 thru 0x026d9b (0xe18b bytes) 000788 1761246694.262760 write 0x02a9bf thru 0x031928 (0x6f6a bytes) 000790 1761246694.264466 read 0x00f9e8 thru 0x019c41 (0xa25a bytes) 000791 1761246694.273269 read 0x014e30 thru 0x023af7 (0xecc8 bytes) 000792 1761246694.274385 read 0x00e67f thru 0x01de4a (0xf7cc bytes) 000794 1761246694.298165 write 0x0226b3 thru 0x03074f (0xe09d bytes) 000795 1761246694.300906 write 0x0188a0 thru 0x02759c (0xecfd bytes) 000796 1761246694.302263 mapread 0x025671 thru 0x031928 (0xc2b8 bytes) 000798 1761246694.303733 mapread 0x01af00 thru 0x029ad7 (0xebd8 bytes) 000800 1761246694.307270 read 0x003e17 thru 0x008f39 (0x5123 bytes) 000801 1761246694.315123 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000804 1761246694.586737 trunc from 0x031929 to 0x00f02c (0x228fc bytes) 000807 1761246694.604707 mapread 0x00b20e thru 0x00e980 (0x3773 bytes) 000808 1761246694.625158 write 0x004265 thru 0x0058f2 (0x168e bytes) 000809 1761246694.635152 mapread 0x00aaba thru 0x00f02c (0x4573 bytes) 000810 1761246694.636030 trunc from 0x00f02d to 0x032b34 (0x23b08 bytes) 000811 1761246694.653486 mapread 0x029074 thru 0x02dea4 (0x4e31 bytes) 000813 1761246694.663221 mapread 0x002f9c thru 0x00e883 (0xb8e8 bytes) 000818 1761246694.666052 write 0x003db6 thru 0x00e47c (0xa6c7 bytes) 000820 1761246694.675171 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000824 1761246694.961466 write 0x008730 thru 0x00c18b (0x3a5c bytes) 000827 1761246694.978923 MIRROR_EXTEND to 4 mirrors 000828 1761246695.698786 write 0x020cef thru 0x021b76 (0xe88 bytes) 000829 1761246695.797335 mapread 0x00a697 thru 0x014814 (0xa17e bytes) 000832 1761246695.819791 mapread 0x007e56 thru 0x013667 (0xb812 bytes) 000836 1761246695.822038 trunc from 0x032b35 to 0x017b1a (0x1b01a bytes) 000840 1761246695.846267 write 0x03ee1c thru 0x03ffff (0x11e4 bytes) 000841 1761246695.847848 mapread 0x028df1 thru 0x033049 (0xa259 bytes) 000842 1761246695.873855 mapwrite 0x0313e1 thru 0x035381 (0x3fa1 bytes) 000843 1761246695.943302 MIRROR_RESYNC 4 mirrors 000845 1761246696.860494 MIRROR_EXTEND to 5 mirrors 000846 1761246697.472988 write 0x0379e3 thru 0x03d1e4 (0x5802 bytes) 000847 1761246697.568612 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000850 1761246697.854595 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000851 1761246698.102913 mapwrite 0x00af97 thru 0x012c7f (0x7ce9 bytes) 000854 1761246698.201653 mapread 0x020aef thru 0x0269de (0x5ef0 bytes) 000857 1761246698.216864 mapread 0x019e57 thru 0x01d512 (0x36bc bytes) 000858 1761246698.217826 mapread 0x0120c8 thru 0x0156f9 (0x3632 bytes) 000859 1761246698.226160 mapwrite 0x03b3e7 thru 0x03f9bc (0x45d6 bytes) 000862 1761246698.366464 read 0x02a5e4 thru 0x02eea9 (0x48c6 bytes) 000863 1761246698.388075 write 0x022685 thru 0x0240ce (0x1a4a bytes) 000865 1761246698.390616 write 0x02876a thru 0x03097b (0x8212 bytes) 000868 1761246698.398297 write 0x00da3f thru 0x019e4c (0xc40e bytes) 000869 1761246698.399851 write 0x03e092 thru 0x03ffff (0x1f6e bytes) 000873 1761246698.402545 read 0x039932 thru 0x03ffff (0x66ce bytes) 000877 1761246698.406408 trunc from 0x040000 to 0x03e2af (0x1d50 bytes) 000879 1761246698.442872 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000880 1761246698.715842 mapwrite 0x03c5d9 thru 0x03ffff (0x3a27 bytes) 000881 1761246698.796979 MIRROR_RESYNC 2 mirrors 000882 1761246699.524592 write 0x023e1d thru 0x02cbcb (0x8daf bytes) 000883 1761246699.611973 write 0x0138dc thru 0x014fc3 (0x16e8 bytes) 000884 1761246699.630709 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000885 1761246699.825248 read 0x03995b thru 0x03d99e (0x4044 bytes) 000888 1761246699.836966 trunc from 0x040000 to 0x01080a (0x2f7f5 bytes) 000890 1761246699.866560 mapread 0x006d3b thru 0x01080a (0x9ad0 bytes) 000892 1761246699.890531 read 0x0084cf thru 0x0103f0 (0x7f22 bytes) 000893 1761246699.891146 mapwrite 0x00c4a2 thru 0x00cf10 (0xa6f bytes) 000894 1761246699.947616 trunc from 0x01080b to 0x038ee9 (0x286df bytes) 000900 1761246699.970615 trunc from 0x038eea to 0x02a25e (0xec8b bytes) 000902 1761246699.993011 mapwrite 0x007d14 thru 0x010dff (0x90ec bytes) 000903 1761246700.050642 read 0x013a04 thru 0x021419 (0xda16 bytes) 000904 1761246700.061585 read 0x022fca thru 0x024a1a (0x1a51 bytes) 000905 1761246700.062903 mapwrite 0x0189c3 thru 0x01bb0b (0x3149 bytes) 000906 1761246700.106184 read 0x009b32 thru 0x00c07b (0x254a bytes) 000907 1761246700.107412 mapread 0x029615 thru 0x02a25e (0xc4a bytes) 000908 1761246700.108264 mapwrite 0x03913e thru 0x03ffff (0x6ec2 bytes) 000910 1761246700.207281 mapwrite 0x007785 thru 0x00da89 (0x6305 bytes) 000913 1761246700.267743 write 0x029536 thru 0x037e6c (0xe937 bytes) 000915 1761246700.269497 mapwrite 0x038ead thru 0x03ffff (0x7153 bytes) 000918 1761246700.359845 MIRROR_EXTEND to 2 mirrors 000921 1761246701.008690 MIRROR_RESYNC 2 mirrors 000923 1761246701.233861 read 0x02cd08 thru 0x0399ca (0xccc3 bytes) 000924 1761246701.246187 MIRROR_RESYNC 2 mirrors 000925 1761246701.493946 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000929 1761246701.668018 mapwrite 0x02dd2c thru 0x03d1ee (0xf4c3 bytes) 000932 1761246701.775151 MIRROR_EXTEND to 2 mirrors 000935 1761246702.440963 mapread 0x01c682 thru 0x01fffd (0x397c bytes) 000937 1761246702.484047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000938 1761246702.645653 mapwrite 0x032793 thru 0x0373bd (0x4c2b bytes) 000939 1761246702.756861 mapwrite 0x01192c thru 0x01e75d (0xce32 bytes) 000943 1761246702.850370 write 0x00028d thru 0x003914 (0x3688 bytes) 000944 1761246702.861193 mapwrite 0x0133b8 thru 0x01cd73 (0x99bc bytes) 000947 1761246702.911304 trunc from 0x040000 to 0x01f1a0 (0x20e5f bytes) 000949 1761246702.937202 write 0x03f897 thru 0x03ffff (0x769 bytes) 000950 1761246702.938389 trunc from 0x040000 to 0x0099f3 (0x3660c bytes) 000951 1761246702.968032 MIRROR_EXTEND to 2 mirrors 000952 1761246703.645542 mapwrite 0x0260b7 thru 0x027de8 (0x1d32 bytes) 000954 1761246703.820797 write 0x00e0ad thru 0x00fb18 (0x1a6c bytes) 000955 1761246703.842711 MIRROR_RESYNC 2 mirrors 000957 1761246704.510681 mapwrite 0x02715b thru 0x02f00f (0x7eb5 bytes) 000959 1761246704.772211 mapwrite 0x00d651 thru 0x00e02d (0x9dd bytes) 000961 1761246704.866050 mapread 0x02c6e7 thru 0x02f00f (0x2929 bytes) 000962 1761246704.868070 mapwrite 0x01c6d7 thru 0x01d4d9 (0xe03 bytes) 000963 1761246704.943241 trunc from 0x02f010 to 0x01be4a (0x131c5 bytes) 000964 1761246704.986493 MIRROR_RESYNC 2 mirrors 000965 1761246705.765380 write 0x0384f2 thru 0x03ffff (0x7b0e bytes) 000966 1761246705.865226 mapread 0x02c3ca thru 0x030d32 (0x4969 bytes) 000968 1761246705.897458 read 0x0101dd thru 0x01e4fd (0xe321 bytes) 000969 1761246705.905983 trunc from 0x040000 to 0x02df40 (0x120bf bytes) 000970 1761246705.936261 write 0x02f505 thru 0x03a8cf (0xb3cb bytes) 000971 1761246705.944378 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000972 1761246706.138791 MIRROR_EXTEND to 2 mirrors 000973 1761246706.828287 MIRROR_EXTEND to 3 mirrors 000975 1761246707.696281 trunc from 0x03a8d0 to 0x037e8f (0x2a40 bytes) 000976 1761246707.802777 mapread 0x0157b2 thru 0x022397 (0xcbe6 bytes) 000980 1761246707.831343 trunc from 0x037e90 to 0x032308 (0x5b87 bytes) 000982 1761246707.868253 mapread 0x030e47 thru 0x032308 (0x14c2 bytes) 000985 1761246707.892959 trunc from 0x032309 to 0x01fe22 (0x124e6 bytes) 000987 1761246707.927658 MIRROR_RESYNC 3 mirrors 000988 1761246708.777331 MIRROR_RESYNC 3 mirrors 000991 1761246709.047026 trunc from 0x01fe23 to 0x02b9c4 (0xbba2 bytes) 000994 1761246709.117301 write 0x035b0c thru 0x036f84 (0x1479 bytes) 000995 1761246709.118357 mapwrite 0x00fe4f thru 0x0131c1 (0x3373 bytes) 000996 1761246709.207568 read 0x000e64 thru 0x00723a (0x63d7 bytes) 000998 1761246709.216368 write 0x03ca73 thru 0x03ffff (0x358d bytes) 000999 1761246709.223475 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (85s) == sanity-flr test 71: check mirror extend parent fid ==== 15:11:54 (1761246714) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 15:11:57 (1761246717) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4337504 .. Extending file size to 6763200 .. done Extending file size to 8579104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8754144 .. Extending file size to 7515616 .. done Extending file size to 8344000 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8817024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9051712 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9283872 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 54782 54783 54786 54787 54788 done 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:13:14 (1761246794) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8834336 .. Extending file size to 2978336 .. Extending file size to 3764128 .. Extending file size to 6315968 .. Extending file size to 7971424 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8589728 .. Extending file size to 8671680 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8996480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56941 56942 56945 56948 56949 done 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) 192.168.201.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:14:30 (1761246870) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119408 s, 17.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.358006 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:14:50 (1761246890) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0904525 s, 23.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 15:14:56 (1761246896) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x25c:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:15:01 (1761246901) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185122 s, 11.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:15:07 (1761246907) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234745 s, 8.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x260:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:15:13 (1761246913) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.279058 s, 7.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:15:42 (1761246942) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594304 s, 17.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:16:39 (1761246999) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155922 s, 13.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:17:33 (1761247053) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg108-server mds-ost sync done. Creating new pool test_204f oleg108-server: Pool lustre.test_204f created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118966 s, 17.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg108-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:18:46 (1761247126) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:18:53 (1761247133) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:19:01 (1761247141) oleg108-server: Pool lustre.test_206 created Adding targets to pool oleg108-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg108-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg108-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg108-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:19:24 (1761247164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409078 s, 25.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442592 s, 23.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb6e460c9 Mirror 3: 0xb6e460c9 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:19:33 (1761247173) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:19:36 (1761247176) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:19:41 (1761247181) Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.151724 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2510000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2510000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2510000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x7d0000 Read this (4 bytes): BBBB 192.168.201.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:20:19 (1761247219) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.027176 s, 301 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:20:24 (1761247224) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670116 s, 15.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x270:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834428 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-23 15:20:26.000000000 -0400 Modify: 2025-10-23 15:20:26.000000000 -0400 Change: 2025-10-23 15:20:27.000000000 -0400 Birth: 2025-10-23 15:20:27.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:20:32 (1761247232) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:20:35 (1761247235) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.40637 s, 101 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0528497 s, 77.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 23 15:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 23 15:20 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test complete, duration 1496 sec =========== 15:20:41 (1761247241) === sanity-flr: start cleanup 15:20:42 (1761247242) === === sanity-flr: finish cleanup 15:20:47 (1761247247) ===