-----============= acceptance-small: sanity-flr ============----- Thu Nov 13 19:48:40 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg251-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:48:55 (1763081335) === oleg251-client.virtnet: executing check_config_client /mnt/lustre oleg251-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg251-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff929d60018000.idle_timeout=debug osc.lustre-OST0001-osc-ffff929d60018000.idle_timeout=debug disable quota as required oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:49:18 (1763081358) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7545] == sanity-flr test 0a: lfs mirror create with -N option == 19:49:20 (1763081360) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:49:32 (1763081372) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:49:35 (1763081375) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:49:38 (1763081378) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:49:58 (1763081398) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:50:01 (1763081401) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 19:50:03 (1763081403) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff929d60018000.stats=clear osc.lustre-OST0001-osc-ffff929d60018000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:50:10 (1763081410) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:50:20 (1763081420) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:50:28 (1763081428) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:50:31 (1763081431) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 19:50:39 (1763081439) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:50:44 (1763081444) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:50:47 (1763081447) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:50:54 (1763081454) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:51:02 (1763081462) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:51:07 (1763081467) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.667736 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143092 s, 7.3 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:51:14 (1763081474) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.234603 s, 67.0 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:51:19 (1763081479) Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (49s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:52:08 (1763081528) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.305551 s, 27.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (26s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:52:34 (1763081554) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (56s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:53:31 (1763081611) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.75325 s, 14.0 MB/s -rw-r--r-- 1 root root 52428800 Nov 13 19:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.784s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.063s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.963s user 0m0.000s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff929d60018000.ost_server_uuid in FULL state after 0 sec PASS 33b (71s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:54:41 (1763081681) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:54:44 (1763081684) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:54:47 (1763081687) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 19:54:50 (1763081690) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 19:54:55 (1763081695) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.4782 s, 26.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:55:06 (1763081706) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288176 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237076 s, 44.2 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:55:12 (1763081712) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:55:20 (1763081720) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000796606 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000813856 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000950275 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105727 s, 969 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 19:55:29 (1763081729) create test files with size 15755724 .. b6523612fd64d0be477e05e5252a2719 - 177a932ea85f33eb0f8cfdc79c2615e0 - 52308e2eb609fce7bd67e8eebd156960 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre PASS 37 (15s) == sanity-flr test 38: resync ============================ 19:55:44 (1763081744) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:55:53 (1763081753) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:55:58 (1763081758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122945 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195889 s, 53.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 19:56:05 (1763081765) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119429 s, 17.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.297619 s, 14.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18463 s, 11.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (10s) == sanity-flr test 42: lfs mirror verify ================= 19:56:15 (1763081775) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:56:18 (1763081778) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:56:21 (1763081781) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468076 s, 22.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 19:56:28 (1763081788) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:56:31 (1763081791) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:56:40 (1763081800) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.2319 s, 45.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:56:49 (1763081809) llite.lustre-ffff929d42a0c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.208206 s, 50.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff929d49120800.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:56:59 (1763081819) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:57:39 (1763081859) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:58:10 (1763081890) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:58:13 (1763081893) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190136 s, 16.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138976 s, 22.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 483248303777d3ceefd10c6b139ac6d8 - mirror 1 checksum: a745d025bcdef2fc85a5cf2d5b515ef5 - mirror 2 checksum: 483248303777d3ceefd10c6b139ac6d8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Nov 13 19:58:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Nov 13 19:58:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 483248303777d3ceefd10c6b139ac6d8 - mirror 1 checksum: a745d025bcdef2fc85a5cf2d5b515ef5 - mirror 2 checksum: a745d025bcdef2fc85a5cf2d5b515ef5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Nov 13 19:58:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Nov 13 19:58:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:58:23 (1763081903) Failing mds1 on oleg251-server Stopping /mnt/lustre-mds1 (opts:) on oleg251-server 19:58:29 (1763081909) shut down facet: mds1 facet_host: oleg251-server facet_failover_host: oleg251-server Failover mds1 to oleg251-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 19:58:49 (1763081929) targets are mounted 19:58:49 (1763081929) facet_failover done oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315361 s, 33.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:59:00 (1763081940) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:59:03 (1763081943) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:59:06 (1763081946) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00577576 s, 2.8 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:59:11 (1763081951) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:59:14 (1763081954) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:59:17 (1763081957) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:59:37 (1763081977) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:00:04 (1763082004) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:00:26 (1763082026) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:00:49 (1763082049) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:00:55 (1763082055) mirror create pid 52259 mirror split pid 52260 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52259 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52260 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:02:02 (1763082122) Seed set to 2123 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000012 1763082124.186758 write 0x039478 thru 0x03990c (0x495 bytes) 000013 1763082124.194828 MIRROR_EXTEND to 2 mirrors 000014 1763082124.975618 mapwrite 0x01ed26 thru 0x01f626 (0x901 bytes) truncating to largest ever: 0x2f993 000017 1763082125.148656 trunc from 0x03990d to 0x02f992 (0x9f7a bytes) 000018 1763082125.182274 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000019 1763082125.394130 mapread 0x022ffe thru 0x02f992 (0xc995 bytes) truncating to largest ever: 0x3a50c 000020 1763082125.440629 trunc from 0x02f993 to 0x03a50b (0xab79 bytes) 000021 1763082125.467650 read 0x0396b7 thru 0x03a50b (0xe55 bytes) 000022 1763082125.479623 mapread 0x01e17a thru 0x024cfc (0x6b83 bytes) 000023 1763082125.480612 trunc from 0x03a50c to 0x028fc6 (0x11545 bytes) 000026 1763082125.506061 mapread 0x00457a thru 0x004e22 (0x8a9 bytes) truncating to largest ever: 0x3c41e 000029 1763082125.526563 trunc from 0x028fc7 to 0x03c41d (0x13457 bytes) 000033 1763082125.566940 mapread 0x02e442 thru 0x034f1b (0x6ada bytes) 000035 1763082125.584041 read 0x027867 thru 0x03257a (0xad14 bytes) truncating to largest ever: 0x3e319 000036 1763082125.584642 trunc from 0x03c41e to 0x03e318 (0x1efb bytes) 000037 1763082125.627713 write 0x038495 thru 0x03ffff (0x7b6b bytes) 000038 1763082125.637769 trunc from 0x040000 to 0x0323db (0xdc24 bytes) 000040 1763082125.660003 mapwrite 0x00f4fd thru 0x01eb6c (0xf670 bytes) 000044 1763082125.742116 trunc from 0x0323dc to 0x01c221 (0x161ba bytes) 000046 1763082125.766166 mapwrite 0x03deb3 thru 0x03ffff (0x214d bytes) 000048 1763082125.840607 write 0x000e47 thru 0x0030f5 (0x22af bytes) 000050 1763082125.842601 read 0x01b59d thru 0x02b551 (0xffb5 bytes) 000051 1763082125.854264 mapread 0x01fd36 thru 0x021a54 (0x1d1f bytes) 000054 1763082125.856084 read 0x019399 thru 0x020ab3 (0x771b bytes) 000055 1763082125.865129 MIRROR_EXTEND to 2 mirrors 000057 1763082126.464374 trunc from 0x040000 to 0x0251d7 (0x1ae28 bytes) 000059 1763082126.547251 mapwrite 0x03d9a3 thru 0x03ffff (0x265d bytes) 000060 1763082126.679233 mapwrite 0x00ea47 thru 0x00fc97 (0x1251 bytes) 000061 1763082126.746813 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000064 1763082126.916809 mapread 0x006553 thru 0x007265 (0xd13 bytes) 000068 1763082126.934678 write 0x01d3bc thru 0x02bb65 (0xe7aa bytes) 000069 1763082126.955169 mapread 0x00871a thru 0x016c27 (0xe50e bytes) 000070 1763082126.978663 write 0x00820d thru 0x00f9f5 (0x77e9 bytes) 000072 1763082126.980998 mapwrite 0x0208a6 thru 0x02b113 (0xa86e bytes) 000075 1763082127.050620 mapwrite 0x03108e thru 0x033577 (0x24ea bytes) 000076 1763082127.127523 trunc from 0x040000 to 0x00f228 (0x30dd7 bytes) 000078 1763082127.160057 trunc from 0x00f229 to 0x0399f2 (0x2a7ca bytes) 000079 1763082127.177633 mapread 0x034eb2 thru 0x0384da (0x3629 bytes) 000081 1763082127.188414 write 0x01d405 thru 0x02973f (0xc33b bytes) 000083 1763082127.203850 trunc from 0x0399f3 to 0x01afd4 (0x1ea1e bytes) 000084 1763082127.225424 write 0x012712 thru 0x014585 (0x1e74 bytes) 000085 1763082127.240846 write 0x02c89b thru 0x02d515 (0xc7b bytes) 000086 1763082127.241809 read 0x021aee thru 0x026d71 (0x5284 bytes) 000087 1763082127.253415 MIRROR_EXTEND to 2 mirrors 000088 1763082127.995233 read 0x02b20e thru 0x02d515 (0x2308 bytes) 000090 1763082128.002121 write 0x004420 thru 0x01098b (0xc56c bytes) 000091 1763082128.082085 write 0x006749 thru 0x00c691 (0x5f49 bytes) 000092 1763082128.083381 trunc from 0x02d516 to 0x020479 (0xd09c bytes) 000093 1763082128.125361 trunc from 0x02047a to 0x00981a (0x16c5f bytes) 000094 1763082128.148539 mapread 0x008582 thru 0x00981a (0x1299 bytes) 000095 1763082128.149612 write 0x003582 thru 0x0112db (0xdd5a bytes) 000096 1763082128.165020 write 0x03730b thru 0x03ab4a (0x3840 bytes) 000097 1763082128.167133 mapread 0x01324d thru 0x015ae8 (0x289c bytes) 000098 1763082128.182383 mapread 0x036f6d thru 0x03ab4a (0x3bde bytes) 000099 1763082128.188985 write 0x02e672 thru 0x0302aa (0x1c39 bytes) 000102 1763082128.197605 mapwrite 0x03d229 thru 0x03ffff (0x2dd7 bytes) 000104 1763082128.275374 MIRROR_EXTEND to 3 mirrors 000105 1763082128.811678 write 0x034279 thru 0x038121 (0x3ea9 bytes) 000106 1763082128.897469 write 0x00b3f6 thru 0x0178a1 (0xc4ac bytes) 000108 1763082128.918399 read 0x02f1d3 thru 0x03c4d9 (0xd307 bytes) 000109 1763082128.926470 mapwrite 0x011be0 thru 0x021666 (0xfa87 bytes) 000110 1763082129.007270 mapwrite 0x0067ad thru 0x00907c (0x28d0 bytes) 000112 1763082129.044592 read 0x02436d thru 0x030373 (0xc007 bytes) 000113 1763082129.054430 trunc from 0x040000 to 0x02f89a (0x10765 bytes) 000114 1763082129.091870 trunc from 0x02f89b to 0x008b80 (0x26d1a bytes) 000115 1763082129.115544 read 0x002b00 thru 0x008b80 (0x6081 bytes) 000117 1763082129.116645 write 0x037fd4 thru 0x03ffff (0x802c bytes) 000120 1763082129.119387 write 0x00a5f6 thru 0x01415b (0x9b66 bytes) 000122 1763082129.131028 mapread 0x0222f9 thru 0x026369 (0x4071 bytes) 000125 1763082129.133180 mapread 0x01041f thru 0x013d46 (0x3928 bytes) 000126 1763082129.138081 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000128 1763082129.373882 mapwrite 0x01ef19 thru 0x020eca (0x1fb2 bytes) 000129 1763082129.430609 write 0x02ccae thru 0x034d12 (0x8065 bytes) 000131 1763082129.447783 mapread 0x026864 thru 0x034ef3 (0xe690 bytes) 000132 1763082129.449583 trunc from 0x040000 to 0x02d9c9 (0x12636 bytes) truncating to largest ever: 0x3f7f5 000133 1763082129.468245 trunc from 0x02d9ca to 0x03f7f4 (0x11e2b bytes) 000134 1763082129.488893 write 0x036218 thru 0x03ffff (0x9de8 bytes) 000135 1763082129.498368 write 0x03ea52 thru 0x03ffff (0x15ae bytes) 000137 1763082129.500535 write 0x02ed2d thru 0x033744 (0x4a18 bytes) 000140 1763082129.513243 MIRROR_RESYNC 2 mirrors 000144 1763082130.094174 read 0x015d8e thru 0x018070 (0x22e3 bytes) 000145 1763082130.101861 write 0x019fa8 thru 0x01c880 (0x28d9 bytes) 000146 1763082130.197930 read 0x0103fb thru 0x01d0dc (0xcce2 bytes) 000148 1763082130.222837 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000149 1763082130.364989 mapwrite 0x023270 thru 0x02c0eb (0x8e7c bytes) 000150 1763082130.470055 write 0x000d72 thru 0x006285 (0x5514 bytes) 000151 1763082130.479283 write 0x014a29 thru 0x01e8d4 (0x9eac bytes) 000153 1763082130.487292 trunc from 0x040000 to 0x0365df (0x9a20 bytes) 000158 1763082130.523972 mapread 0x0033b5 thru 0x008fce (0x5c1a bytes) 000160 1763082130.526097 mapwrite 0x027724 thru 0x02c37a (0x4c57 bytes) 000161 1763082130.598702 read 0x035184 thru 0x0365df (0x145c bytes) 000163 1763082130.606214 write 0x03a1d9 thru 0x03ffff (0x5e27 bytes) 000165 1763082130.607740 write 0x00baf9 thru 0x00d6d5 (0x1bdd bytes) 000166 1763082130.608574 write 0x038967 thru 0x03ffff (0x7699 bytes) 000168 1763082130.632930 mapwrite 0x021c7e thru 0x028a55 (0x6dd8 bytes) 000170 1763082130.707085 trunc from 0x040000 to 0x01cd73 (0x2328c bytes) 000171 1763082130.728783 mapwrite 0x030194 thru 0x033701 (0x356e bytes) 000174 1763082130.815676 MIRROR_EXTEND to 2 mirrors 000175 1763082131.439738 mapwrite 0x010a33 thru 0x011d2f (0x12fd bytes) 000177 1763082131.630376 mapwrite 0x03c4b7 thru 0x03ffff (0x3b49 bytes) 000178 1763082131.697270 read 0x0092fc thru 0x012a25 (0x972a bytes) 000181 1763082131.703192 mapwrite 0x018c39 thru 0x02446e (0xb836 bytes) 000182 1763082131.771722 read 0x00fbb0 thru 0x015fea (0x643b bytes) 000183 1763082131.776819 MIRROR_RESYNC 2 mirrors 000184 1763082132.359983 write 0x02135c thru 0x027eb6 (0x6b5b bytes) 000186 1763082132.464966 MIRROR_EXTEND to 3 mirrors 000188 1763082133.208671 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000189 1763082133.406069 trunc from 0x040000 to 0x0283c7 (0x17c38 bytes) 000191 1763082133.426863 write 0x03bace thru 0x03ffff (0x4532 bytes) 000192 1763082133.428151 write 0x00b4bf thru 0x019d02 (0xe844 bytes) 000193 1763082133.438182 read 0x002d89 thru 0x005734 (0x29ac bytes) 000194 1763082133.443633 trunc from 0x040000 to 0x00a6ab (0x35954 bytes) 000195 1763082133.471551 mapwrite 0x018808 thru 0x025f59 (0xd752 bytes) 000196 1763082133.556516 trunc from 0x025f5a to 0x028be2 (0x2c89 bytes) 000197 1763082133.573340 mapwrite 0x032ed1 thru 0x03d066 (0xa196 bytes) 000198 1763082133.677597 mapwrite 0x00280c thru 0x00e0db (0xb8d0 bytes) 000199 1763082133.721753 trunc from 0x03d067 to 0x0138b0 (0x297b6 bytes) 000200 1763082133.743497 mapread 0x010324 thru 0x0138b0 (0x358d bytes) 000202 1763082133.749277 MIRROR_RESYNC 2 mirrors 000204 1763082134.551162 MIRROR_EXTEND to 3 mirrors 000205 1763082135.293113 mapwrite 0x0395f0 thru 0x03ffff (0x6a10 bytes) 000207 1763082135.432886 mapread 0x0167d8 thru 0x01d2d2 (0x6afb bytes) 000208 1763082135.440809 trunc from 0x040000 to 0x01297b (0x2d684 bytes) 000209 1763082135.462030 read 0x00a440 thru 0x01297b (0x853c bytes) 000210 1763082135.476774 write 0x01f118 thru 0x028fca (0x9eb3 bytes) 000211 1763082135.479289 trunc from 0x028fcb to 0x021cda (0x72f0 bytes) 000212 1763082135.507471 write 0x00dbd7 thru 0x014f49 (0x7373 bytes) 000214 1763082135.517745 write 0x01aaa9 thru 0x02085d (0x5db5 bytes) 000217 1763082135.531240 mapwrite 0x02defe thru 0x0372b1 (0x93b4 bytes) 000218 1763082135.627085 read 0x00f860 thru 0x012e96 (0x3637 bytes) 000219 1763082135.636402 MIRROR_RESYNC 3 mirrors 000220 1763082136.402908 read 0x00e20a thru 0x0120bd (0x3eb4 bytes) 000221 1763082136.414377 trunc from 0x0372b2 to 0x01b950 (0x1b961 bytes) 000222 1763082136.524703 mapwrite 0x026827 thru 0x02c458 (0x5c32 bytes) 000223 1763082136.644922 mapwrite 0x018056 thru 0x026090 (0xe03b bytes) 000226 1763082136.731366 trunc from 0x02c459 to 0x02b164 (0x12f4 bytes) 000228 1763082136.761131 write 0x0128f8 thru 0x01d1bf (0xa8c8 bytes) 000229 1763082136.762883 mapread 0x011258 thru 0x01730e (0x60b7 bytes) 000232 1763082136.765782 mapwrite 0x021683 thru 0x028602 (0x6f80 bytes) 000233 1763082136.814736 mapwrite 0x006095 thru 0x00c1df (0x614b bytes) 000234 1763082136.873030 trunc from 0x02b165 to 0x0003fd (0x2ad67 bytes) 000236 1763082136.889771 write 0x03eb50 thru 0x03f08c (0x53d bytes) 000237 1763082136.891046 read 0x006887 thru 0x008e2d (0x25a7 bytes) 000238 1763082136.903984 mapread 0x01ed1a thru 0x024edb (0x61c2 bytes) 000240 1763082136.909869 MIRROR_RESYNC 3 mirrors 000241 1763082137.681425 mapread 0x034657 thru 0x03a69e (0x6048 bytes) 000243 1763082137.705293 mapwrite 0x0002ab thru 0x00acac (0xaa02 bytes) 000244 1763082137.921547 read 0x0348fb thru 0x03f08c (0xa792 bytes) 000245 1763082137.932255 write 0x01d5e4 thru 0x020ed5 (0x38f2 bytes) 000247 1763082137.944853 read 0x008297 thru 0x00fdea (0x7b54 bytes) 000248 1763082137.945471 mapread 0x034a63 thru 0x03f08c (0xa62a bytes) 000249 1763082137.950568 MIRROR_EXTEND to 4 mirrors 000252 1763082138.650001 mapread 0x038d4f thru 0x03f08c (0x633e bytes) 000253 1763082138.672036 MIRROR_RESYNC 4 mirrors 000256 1763082139.481401 mapwrite 0x034485 thru 0x03bc1b (0x7797 bytes) 000260 1763082139.690359 mapread 0x034809 thru 0x03f08c (0xa884 bytes) 000261 1763082139.709030 trunc from 0x03f08d to 0x00523b (0x39e51 bytes) 000262 1763082139.759765 trunc from 0x00523c to 0x02f1b9 (0x29f7e bytes) 000264 1763082139.802815 mapwrite 0x02e216 thru 0x03a2a6 (0xc091 bytes) 000265 1763082139.938076 trunc from 0x03a2a7 to 0x01081e (0x29a88 bytes) 000266 1763082139.957433 trunc from 0x01081f to 0x01343a (0x2c1c bytes) 000268 1763082139.979232 trunc from 0x01343b to 0x00e729 (0x4d11 bytes) 000269 1763082140.000690 read 0x006cd7 thru 0x00cada (0x5e04 bytes) 000271 1763082140.015749 read 0x0054fa thru 0x00e729 (0x9230 bytes) 000272 1763082140.032198 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000273 1763082140.344973 trunc from 0x00e72a to 0x038a3a (0x2a311 bytes) 000274 1763082140.381554 read 0x00f6de thru 0x0181a2 (0x8ac5 bytes) 000275 1763082140.390233 write 0x0291b1 thru 0x02e73f (0x558f bytes) 000278 1763082140.411864 mapread 0x00a052 thru 0x00f8de (0x588d bytes) 000280 1763082140.432724 mapread 0x0111ba thru 0x01a8ee (0x9735 bytes) 000281 1763082140.465989 write 0x004efc thru 0x008aef (0x3bf4 bytes) 000282 1763082140.472925 MIRROR_RESYNC 3 mirrors 000283 1763082141.160589 mapread 0x02af11 thru 0x02d90d (0x29fd bytes) 000285 1763082141.179713 mapwrite 0x00b559 thru 0x01103f (0x5ae7 bytes) 000286 1763082141.364268 read 0x00e94d thru 0x01273c (0x3df0 bytes) 000288 1763082141.365247 write 0x02dc89 thru 0x037d20 (0xa098 bytes) 000289 1763082141.391621 mapwrite 0x039194 thru 0x03f5a9 (0x6416 bytes) 000290 1763082141.514878 mapread 0x01a240 thru 0x01e528 (0x42e9 bytes) 000291 1763082141.531220 trunc from 0x03f5aa to 0x02999f (0x15c0a bytes) 000293 1763082141.559785 trunc from 0x0299a0 to 0x03a0cd (0x1072e bytes) 000299 1763082141.591725 mapread 0x0309a8 thru 0x03a0cd (0x9726 bytes) 000300 1763082141.631665 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000302 1763082141.871929 mapread 0x02e2a6 thru 0x039dd5 (0xbb30 bytes) 000303 1763082141.923833 trunc from 0x03a0ce to 0x01e47d (0x1bc50 bytes) 000304 1763082141.965023 write 0x031681 thru 0x0378d3 (0x6253 bytes) 000305 1763082141.966150 mapwrite 0x03d38b thru 0x03f407 (0x207d bytes) 000306 1763082142.068044 write 0x008e0c thru 0x01509c (0xc291 bytes) 000308 1763082142.094094 trunc from 0x03f408 to 0x0324d4 (0xcf33 bytes) 000309 1763082142.126776 mapwrite 0x02d3af thru 0x039d51 (0xc9a3 bytes) 000310 1763082142.275960 mapread 0x031793 thru 0x039d51 (0x85bf bytes) 000312 1763082142.277416 trunc from 0x039d52 to 0x006f6d (0x32de4 bytes) 000314 1763082142.304995 mapwrite 0x03f335 thru 0x03ffff (0xccb bytes) 000316 1763082142.406545 mapread 0x039d03 thru 0x03af83 (0x1281 bytes) 000317 1763082142.407373 mapread 0x019104 thru 0x022ae9 (0x99e6 bytes) 000318 1763082142.444111 read 0x027f40 thru 0x028e5e (0xf1f bytes) 000321 1763082142.445658 read 0x0359e3 thru 0x039559 (0x3b77 bytes) 000322 1763082142.462579 read 0x01408a thru 0x02125a (0xd1d1 bytes) 000323 1763082142.467455 read 0x010170 thru 0x017c56 (0x7ae7 bytes) 000324 1763082142.468073 read 0x03b709 thru 0x03ffff (0x48f7 bytes) 000326 1763082142.474703 MIRROR_RESYNC 2 mirrors 000327 1763082143.007430 mapwrite 0x03ca78 thru 0x03ffff (0x3588 bytes) 000329 1763082143.219971 write 0x01d242 thru 0x026273 (0x9032 bytes) 000331 1763082143.256558 mapread 0x011074 thru 0x020eb4 (0xfe41 bytes) 000332 1763082143.284706 MIRROR_EXTEND to 3 mirrors 000334 1763082143.894119 write 0x005686 thru 0x0066d4 (0x104f bytes) 000335 1763082143.978054 mapwrite 0x03d960 thru 0x03ffff (0x26a0 bytes) 000338 1763082144.047837 mapread 0x039de2 thru 0x03ffff (0x621e bytes) 000340 1763082144.050661 trunc from 0x040000 to 0x01f656 (0x209a9 bytes) 000341 1763082144.116144 MIRROR_EXTEND to 4 mirrors 000343 1763082144.887025 write 0x035ae5 thru 0x03ffff (0xa51b bytes) 000345 1763082144.993547 trunc from 0x040000 to 0x03f744 (0x8bb bytes) 000346 1763082145.016830 mapread 0x021140 thru 0x02cc7a (0xbb3b bytes) 000348 1763082145.055945 mapwrite 0x00c018 thru 0x019014 (0xcffd bytes) 000349 1763082145.138831 read 0x02cdc1 thru 0x03913b (0xc37b bytes) 000351 1763082145.140188 read 0x028497 thru 0x037224 (0xed8e bytes) 000352 1763082145.141229 mapwrite 0x000d79 thru 0x005f07 (0x518f bytes) 000353 1763082145.207115 mapwrite 0x032b35 thru 0x03d353 (0xa81f bytes) 000354 1763082145.273483 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000355 1763082145.469580 read 0x016a83 thru 0x0193cf (0x294d bytes) 000356 1763082145.475976 trunc from 0x03f745 to 0x015829 (0x29f1b bytes) 000357 1763082145.497151 mapwrite 0x0336c1 thru 0x03ffff (0xc93f bytes) 000360 1763082145.645095 read 0x03f642 thru 0x03ffff (0x9be bytes) 000361 1763082145.646271 trunc from 0x040000 to 0x03c74b (0x38b4 bytes) 000363 1763082145.680657 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000364 1763082145.923366 read 0x03529f thru 0x03c74b (0x74ad bytes) 000365 1763082145.936431 MIRROR_RESYNC 2 mirrors 000367 1763082146.556904 mapread 0x00dc2f thru 0x01d720 (0xfaf2 bytes) 000368 1763082146.596372 mapwrite 0x03d8b5 thru 0x03ffff (0x274b bytes) 000369 1763082146.735083 mapread 0x02e54c thru 0x039eff (0xb9b4 bytes) 000370 1763082146.747665 read 0x02028b thru 0x029596 (0x930c bytes) 000371 1763082146.758656 trunc from 0x040000 to 0x00f5b0 (0x30a4f bytes) 000372 1763082146.790237 mapwrite 0x010360 thru 0x01f81a (0xf4bb bytes) 000374 1763082146.918389 read 0x01c885 thru 0x01f81a (0x2f96 bytes) 000375 1763082146.921146 write 0x03925f thru 0x03e242 (0x4fe4 bytes) 000383 1763082146.927072 read 0x023043 thru 0x02a49d (0x745b bytes) 000385 1763082146.962023 MIRROR_EXTEND to 3 mirrors 000386 1763082147.517720 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000387 1763082147.766915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000392 1763082147.901289 mapwrite 0x02a484 thru 0x038f2b (0xeaa8 bytes) 000393 1763082148.003879 trunc from 0x03e243 to 0x035367 (0x8edb bytes) 000394 1763082148.031027 write 0x02a098 thru 0x039d5d (0xfcc6 bytes) 000395 1763082148.032393 write 0x010ef7 thru 0x012f17 (0x2021 bytes) 000397 1763082148.045934 trunc from 0x039d5e to 0x02f08e (0xaccf bytes) 000398 1763082148.064481 read 0x02a0b5 thru 0x02c314 (0x2260 bytes) 000403 1763082148.066946 mapread 0x01da7c thru 0x02314f (0x56d4 bytes) 000404 1763082148.075562 mapread 0x0201da thru 0x0206c5 (0x4ec bytes) 000405 1763082148.076343 read 0x0249ec thru 0x02f08e (0xa6a3 bytes) 000406 1763082148.076865 write 0x00c5c8 thru 0x00e317 (0x1d50 bytes) 000409 1763082148.078842 mapwrite 0x03a7c2 thru 0x03ffff (0x583e bytes) 000410 1763082148.178060 write 0x03a961 thru 0x03b3d1 (0xa71 bytes) 000412 1763082148.179613 mapwrite 0x032fc9 thru 0x037956 (0x498e bytes) 000416 1763082148.246077 trunc from 0x040000 to 0x0111fd (0x2ee02 bytes) 000420 1763082148.279634 read 0x00ee27 thru 0x0111fd (0x23d7 bytes) 000421 1763082148.282164 read 0x0073ce thru 0x00ac2e (0x3861 bytes) 000422 1763082148.283488 mapread 0x003803 thru 0x0111fd (0xd9fb bytes) 000423 1763082148.287369 mapwrite 0x036968 thru 0x03ffff (0x9698 bytes) 000424 1763082148.410759 trunc from 0x040000 to 0x01900c (0x26ff3 bytes) 000427 1763082148.435555 read 0x0094e3 thru 0x010f54 (0x7a72 bytes) 000428 1763082148.436085 trunc from 0x01900d to 0x004e5d (0x141af bytes) 000429 1763082148.451898 mapread 0x0025e4 thru 0x004e5d (0x287a bytes) 000430 1763082148.453916 trunc from 0x004e5e to 0x00ab76 (0x5d19 bytes) 000433 1763082148.471911 mapwrite 0x0083f3 thru 0x00f184 (0x6d92 bytes) 000435 1763082148.540024 mapwrite 0x0038d9 thru 0x012872 (0xef9a bytes) 000436 1763082148.614541 mapwrite 0x01ddfd thru 0x029b2e (0xbd32 bytes) 000437 1763082148.694490 mapwrite 0x03dbc6 thru 0x03ffff (0x243a bytes) 000438 1763082148.775514 trunc from 0x040000 to 0x007171 (0x38e8e bytes) 000439 1763082148.798369 read 0x00133d thru 0x007171 (0x5e35 bytes) 000441 1763082148.799584 mapwrite 0x03e703 thru 0x03ffff (0x18fd bytes) 000442 1763082148.868704 read 0x0219fc thru 0x0268ca (0x4ecf bytes) 000443 1763082148.869477 mapread 0x0194cc thru 0x01d7a8 (0x42dd bytes) 000447 1763082148.872611 read 0x024667 thru 0x031e63 (0xd7fd bytes) 000448 1763082148.873636 mapwrite 0x01165f thru 0x01bded (0xa78f bytes) 000449 1763082148.913194 mapread 0x02bf72 thru 0x0364a4 (0xa533 bytes) 000450 1763082148.914278 trunc from 0x040000 to 0x037643 (0x89bc bytes) 000452 1763082148.941599 read 0x010dd2 thru 0x0112ff (0x52e bytes) 000454 1763082148.942767 read 0x0372a8 thru 0x037643 (0x39c bytes) 000455 1763082148.944903 mapwrite 0x022a07 thru 0x024348 (0x1942 bytes) 000468 1763082149.010002 mapwrite 0x03d4a7 thru 0x03ffff (0x2b59 bytes) 000469 1763082149.086794 read 0x00a53d thru 0x017353 (0xce17 bytes) 000470 1763082149.087942 trunc from 0x040000 to 0x02b916 (0x146e9 bytes) 000471 1763082149.121991 trunc from 0x02b917 to 0x01e7a9 (0xd16d bytes) 000473 1763082149.153663 trunc from 0x01e7aa to 0x03cb86 (0x1e3dd bytes) 000474 1763082149.174352 write 0x0381f2 thru 0x0397a8 (0x15b7 bytes) 000475 1763082149.184040 trunc from 0x03cb87 to 0x009e12 (0x32d74 bytes) 000478 1763082149.210878 trunc from 0x009e13 to 0x00979e (0x674 bytes) 000480 1763082149.248174 read 0x0039bf thru 0x0070cc (0x370e bytes) 000483 1763082149.249788 mapread 0x004778 thru 0x007b61 (0x33ea bytes) 000485 1763082149.251688 mapread 0x003fcc thru 0x00979e (0x57d3 bytes) 000486 1763082149.253856 read 0x00606e thru 0x00979e (0x3731 bytes) 000488 1763082149.255853 mapwrite 0x0216e7 thru 0x027fd0 (0x68ea bytes) 000489 1763082149.350237 mapread 0x00cdeb thru 0x010ed9 (0x40ef bytes) 000490 1763082149.351425 mapwrite 0x030542 thru 0x033529 (0x2fe8 bytes) 000492 1763082149.445856 mapread 0x014d33 thru 0x024166 (0xf434 bytes) 000493 1763082149.446976 mapread 0x009cc2 thru 0x012b94 (0x8ed3 bytes) 000494 1763082149.448380 write 0x035661 thru 0x03ffff (0xa99f bytes) 000497 1763082149.451970 mapread 0x01038d thru 0x019653 (0x92c7 bytes) 000498 1763082149.454061 read 0x01eefb thru 0x02863c (0x9742 bytes) 000499 1763082149.454833 trunc from 0x040000 to 0x019a0d (0x265f2 bytes) 000501 1763082149.476918 trunc from 0x019a0e to 0x0255d0 (0xbbc3 bytes) 000502 1763082149.494032 mapread 0x00403a thru 0x006f57 (0x2f1e bytes) 000503 1763082149.495081 trunc from 0x0255d1 to 0x035f69 (0x10999 bytes) 000510 1763082149.537231 mapread 0x006c87 thru 0x00c63a (0x59b4 bytes) 000511 1763082149.544718 mapread 0x02e9cd thru 0x034f8c (0x65c0 bytes) 000512 1763082149.561343 trunc from 0x035f6a to 0x007498 (0x2ead1 bytes) 000515 1763082149.584536 read 0x000400 thru 0x003e43 (0x3a44 bytes) 000519 1763082149.587139 read 0x0055ce thru 0x007498 (0x1ecb bytes) 000521 1763082149.589023 write 0x031efa thru 0x03ef24 (0xd02b bytes) 000523 1763082149.591799 mapwrite 0x01f6dd thru 0x02adb6 (0xb6da bytes) 000527 1763082149.658550 mapwrite 0x00e871 thru 0x01814f (0x98df bytes) 000530 1763082149.703091 mapread 0x02412b thru 0x0255e5 (0x14bb bytes) 000531 1763082149.704040 write 0x020efc thru 0x02c6a9 (0xb7ae bytes) 000532 1763082149.708804 mapread 0x0018f1 thru 0x00b148 (0x9858 bytes) 000535 1763082149.711560 mapwrite 0x0017ed thru 0x00aa7e (0x9292 bytes) 000536 1763082149.796255 trunc from 0x03ef25 to 0x00a82c (0x346f8 bytes) 000538 1763082149.829771 read 0x009c57 thru 0x00a82c (0xbd6 bytes) 000539 1763082149.830435 trunc from 0x00a82d to 0x00e948 (0x411c bytes) 000541 1763082149.861410 trunc from 0x00e949 to 0x0212c3 (0x1297b bytes) 000544 1763082149.882261 read 0x014781 thru 0x01ee7a (0xa6fa bytes) 000546 1763082149.896109 mapwrite 0x007f39 thru 0x00a239 (0x2301 bytes) 000551 1763082149.938957 trunc from 0x0212c4 to 0x035c14 (0x14951 bytes) 000553 1763082149.963855 read 0x0063d6 thru 0x009a44 (0x366f bytes) 000555 1763082149.965153 write 0x03d057 thru 0x03e521 (0x14cb bytes) 000557 1763082149.968652 write 0x032c51 thru 0x0399ca (0x6d7a bytes) 000558 1763082149.984660 read 0x016c04 thru 0x026904 (0xfd01 bytes) 000562 1763082149.986804 mapwrite 0x02c140 thru 0x02db93 (0x1a54 bytes) 000566 1763082150.045643 mapwrite 0x0264b4 thru 0x02fe02 (0x994f bytes) 000567 1763082150.101578 mapwrite 0x014650 thru 0x01f6ea (0xb09b bytes) 000568 1763082150.147826 mapread 0x00d638 thru 0x010b9e (0x3567 bytes) 000569 1763082150.148770 trunc from 0x03e522 to 0x01f570 (0x1efb1 bytes) 000573 1763082150.175006 MIRROR_EXTEND to 2 mirrors 000574 1763082150.779597 mapwrite 0x00af3d thru 0x00ffa4 (0x5068 bytes) 000576 1763082150.943863 trunc from 0x01f571 to 0x0267ec (0x727c bytes) 000577 1763082150.973787 write 0x02de15 thru 0x03a447 (0xc633 bytes) 000578 1763082150.983469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000580 1763082151.121363 mapread 0x01e23b thru 0x02a4cd (0xc293 bytes) 000582 1763082151.151503 mapwrite 0x009b16 thru 0x014ae2 (0xafcd bytes) 000585 1763082151.223288 mapread 0x038dbe thru 0x03a447 (0x168a bytes) 000587 1763082151.239685 trunc from 0x03a448 to 0x03c30d (0x1ec6 bytes) 000588 1763082151.256210 read 0x037424 thru 0x03c30d (0x4eea bytes) 000589 1763082151.266581 mapwrite 0x01f821 thru 0x027ca4 (0x8484 bytes) 000591 1763082151.318505 read 0x038ba9 thru 0x03c30d (0x3765 bytes) 000592 1763082151.319161 mapwrite 0x037721 thru 0x03a986 (0x3266 bytes) 000595 1763082151.361405 mapread 0x01fe7e thru 0x0226b4 (0x2837 bytes) 000597 1763082151.365715 mapwrite 0x03fa72 thru 0x03ffff (0x58e bytes) 000602 1763082151.440439 mapwrite 0x01492d thru 0x02051a (0xbbee bytes) 000603 1763082151.480602 write 0x03f00c thru 0x03ffff (0xff4 bytes) 000604 1763082151.481650 trunc from 0x040000 to 0x03864a (0x79b5 bytes) 000607 1763082151.507586 mapread 0x035bfb thru 0x03864a (0x2a50 bytes) 000609 1763082151.510465 read 0x01d393 thru 0x0224f7 (0x5165 bytes) 000610 1763082151.511084 read 0x02e80e thru 0x03864a (0x9e3d bytes) 000611 1763082151.512391 write 0x02b5fe thru 0x039898 (0xe29b bytes) 000612 1763082151.514793 mapwrite 0x01bd2f thru 0x01e065 (0x2337 bytes) 000615 1763082151.551435 write 0x02244d thru 0x02dcb6 (0xb86a bytes) 000616 1763082151.554141 trunc from 0x039899 to 0x011524 (0x28374 bytes) 000618 1763082151.578110 mapread 0x00849b thru 0x00d5b7 (0x511d bytes) 000619 1763082151.579215 trunc from 0x011525 to 0x02f9c6 (0x1e4a2 bytes) 000624 1763082151.611089 write 0x007c96 thru 0x014883 (0xcbee bytes) 000625 1763082151.619557 trunc from 0x02f9c7 to 0x010688 (0x1f33e bytes) 000626 1763082151.640183 mapwrite 0x03ff0f thru 0x03ffff (0xf1 bytes) 000628 1763082151.742780 MIRROR_EXTEND to 2 mirrors 000630 1763082152.383467 trunc from 0x040000 to 0x03f666 (0x999 bytes) 000631 1763082152.472256 mapread 0x03b1ef thru 0x03f666 (0x4478 bytes) 000633 1763082152.493525 MIRROR_EXTEND to 3 mirrors 000637 1763082153.083478 mapwrite 0x02faeb thru 0x03d761 (0xdc77 bytes) 000638 1763082153.238689 trunc from 0x03f667 to 0x01995d (0x25d09 bytes) 000641 1763082153.285880 MIRROR_RESYNC 3 mirrors 000642 1763082154.202009 read 0x004fa0 thru 0x0099ad (0x4a0e bytes) 000643 1763082154.214821 write 0x02299f thru 0x02eb75 (0xc1d7 bytes) 000645 1763082154.327999 write 0x03e332 thru 0x03ffff (0x1cce bytes) 000646 1763082154.338253 MIRROR_RESYNC 3 mirrors 000647 1763082154.998748 write 0x0333e8 thru 0x0360dc (0x2cf5 bytes) 000651 1763082155.090749 mapread 0x01455a thru 0x01bbb0 (0x7657 bytes) 000652 1763082155.110380 mapread 0x03f69d thru 0x03ffff (0x963 bytes) 000653 1763082155.126587 write 0x0322d3 thru 0x03ca23 (0xa751 bytes) 000659 1763082155.152876 mapwrite 0x01f20a thru 0x02b194 (0xbf8b bytes) 000660 1763082155.267645 mapwrite 0x0180dd thru 0x01c262 (0x4186 bytes) 000662 1763082155.312344 trunc from 0x040000 to 0x01e2f2 (0x21d0d bytes) 000663 1763082155.334590 read 0x00be4c thru 0x01b9bd (0xfb72 bytes) 000665 1763082155.340041 mapread 0x00dc77 thru 0x017f58 (0xa2e2 bytes) 000666 1763082155.341506 trunc from 0x01e2f3 to 0x002c76 (0x1b67c bytes) 000668 1763082155.359235 trunc from 0x002c77 to 0x038241 (0x355cb bytes) 000671 1763082155.377399 write 0x03b637 thru 0x03ffff (0x49c9 bytes) 000673 1763082155.379573 mapwrite 0x03ee2a thru 0x03ffff (0x11d6 bytes) 000674 1763082155.415469 read 0x0013de thru 0x0058d4 (0x44f7 bytes) 000675 1763082155.425759 read 0x000b37 thru 0x00c658 (0xbb22 bytes) 000677 1763082155.436101 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000679 1763082155.674028 mapread 0x02422b thru 0x02b4e7 (0x72bd bytes) 000683 1763082155.698737 write 0x038956 thru 0x03ffff (0x76aa bytes) 000684 1763082155.710695 read 0x00e256 thru 0x01ad31 (0xcadc bytes) 000686 1763082155.721601 trunc from 0x040000 to 0x02d30b (0x12cf4 bytes) 000688 1763082155.747057 mapwrite 0x015ef9 thru 0x01f443 (0x954b bytes) 000690 1763082155.815151 mapread 0x0037ea thru 0x00783f (0x4056 bytes) 000691 1763082155.827030 trunc from 0x02d30c to 0x01019a (0x1d171 bytes) 000692 1763082155.848832 mapread 0x0043b8 thru 0x009c1e (0x5867 bytes) 000696 1763082155.863167 MIRROR_RESYNC 2 mirrors 000697 1763082156.453265 mapwrite 0x006d0f thru 0x010d4a (0xa03c bytes) 000698 1763082156.632981 trunc from 0x010d4b to 0x0093f8 (0x7952 bytes) 000699 1763082156.674069 read 0x0008c4 thru 0x0093f8 (0x8b35 bytes) 000700 1763082156.694811 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000701 1763082156.885065 write 0x01c6b3 thru 0x02aa52 (0xe3a0 bytes) 000702 1763082156.892145 write 0x024f91 thru 0x0319da (0xca4a bytes) 000704 1763082156.899826 mapwrite 0x037638 thru 0x03982a (0x21f3 bytes) 000705 1763082157.012011 write 0x0168a4 thru 0x023d9d (0xd4fa bytes) 000707 1763082157.024454 write 0x0212ee thru 0x02d3a9 (0xc0bc bytes) 000708 1763082157.027108 mapread 0x02bd6f thru 0x03552e (0x97c0 bytes) 000711 1763082157.029832 read 0x02db29 thru 0x03982a (0xbd02 bytes) 000712 1763082157.030589 mapread 0x00d262 thru 0x0121f2 (0x4f91 bytes) 000713 1763082157.043459 trunc from 0x03982b to 0x00b8ad (0x2df7d bytes) 000714 1763082157.070576 read 0x0008e9 thru 0x004639 (0x3d51 bytes) 000716 1763082157.084098 mapwrite 0x01dc66 thru 0x029b4d (0xbee8 bytes) 000718 1763082157.176356 mapread 0x020afc thru 0x029b4d (0x9052 bytes) 000719 1763082157.178225 write 0x00e241 thru 0x0143e6 (0x61a6 bytes) 000720 1763082157.180065 read 0x0237e9 thru 0x029b4d (0x6365 bytes) 000722 1763082157.181116 mapwrite 0x016cba thru 0x0260e9 (0xf430 bytes) 000725 1763082157.232223 trunc from 0x029b4e to 0x032cf0 (0x91a3 bytes) 000730 1763082157.252004 mapread 0x004c01 thru 0x00ea05 (0x9e05 bytes) 000732 1763082157.253905 mapwrite 0x00542e thru 0x00a280 (0x4e53 bytes) 000733 1763082157.301576 write 0x000733 thru 0x006a7d (0x634b bytes) 000736 1763082157.303712 read 0x030ec5 thru 0x032cf0 (0x1e2c bytes) 000739 1763082157.326133 trunc from 0x032cf1 to 0x029c93 (0x905d bytes) 000740 1763082157.357287 read 0x01cf4f thru 0x029c93 (0xcd45 bytes) 000741 1763082157.360583 mapread 0x01ca8d thru 0x029c93 (0xd207 bytes) 000743 1763082157.363068 mapwrite 0x037785 thru 0x03ffff (0x887b bytes) 000744 1763082157.458080 trunc from 0x040000 to 0x018c5b (0x273a4 bytes) 000746 1763082157.489437 mapread 0x0153c4 thru 0x0178ee (0x252b bytes) 000748 1763082157.493925 write 0x02d7f6 thru 0x03ce70 (0xf67b bytes) 000751 1763082157.501362 mapwrite 0x00f990 thru 0x0175bc (0x7c2d bytes) 000753 1763082157.554273 trunc from 0x03ce71 to 0x01982b (0x23645 bytes) 000755 1763082157.578687 mapwrite 0x017a6b thru 0x01bea6 (0x443c bytes) 000756 1763082157.693713 trunc from 0x01bea7 to 0x002c0b (0x1929b bytes) 000759 1763082157.717623 mapwrite 0x03130b thru 0x0342ca (0x2fc0 bytes) 000760 1763082157.803056 write 0x003845 thru 0x01362c (0xfde8 bytes) 000761 1763082157.815216 mapwrite 0x00964c thru 0x009a0d (0x3c2 bytes) 000763 1763082157.853405 mapread 0x02edf4 thru 0x02f857 (0xa64 bytes) 000767 1763082157.865066 read 0x00a36a thru 0x0196a8 (0xf33f bytes) 000768 1763082157.874087 read 0x00072f thru 0x004cbb (0x458d bytes) 000769 1763082157.874717 mapwrite 0x00fd9e thru 0x0180df (0x8342 bytes) 000771 1763082157.929096 mapread 0x0279f1 thru 0x030fea (0x95fa bytes) 000773 1763082157.932586 mapwrite 0x03bb89 thru 0x03e928 (0x2da0 bytes) 000774 1763082158.005512 mapwrite 0x00a395 thru 0x00d4d4 (0x3140 bytes) 000775 1763082158.037084 trunc from 0x03e929 to 0x00d19d (0x3178b bytes) 000777 1763082158.057907 mapread 0x0039eb thru 0x004288 (0x89e bytes) 000780 1763082158.060010 read 0x001250 thru 0x0020fd (0xeae bytes) 000781 1763082158.060534 write 0x0017dc thru 0x0109d2 (0xf1f7 bytes) 000783 1763082158.062514 trunc from 0x0109d3 to 0x02bf93 (0x1b5c1 bytes) 000784 1763082158.080007 write 0x01c2e5 thru 0x01da4c (0x1768 bytes) 000786 1763082158.094590 mapread 0x001627 thru 0x0023f4 (0xdce bytes) 000789 1763082158.096509 mapwrite 0x01f80e thru 0x027160 (0x7953 bytes) 000791 1763082158.148064 MIRROR_EXTEND to 2 mirrors 000792 1763082158.647293 read 0x015e74 thru 0x01a4ed (0x467a bytes) 000793 1763082158.657407 mapwrite 0x02e262 thru 0x03217d (0x3f1c bytes) 000794 1763082158.857115 mapread 0x01c078 thru 0x01f72e (0x36b7 bytes) 000798 1763082158.868711 read 0x001c41 thru 0x002fb9 (0x1379 bytes) 000799 1763082158.880801 write 0x02b1b5 thru 0x02eb38 (0x3984 bytes) 000801 1763082158.882397 trunc from 0x03217e to 0x0303b0 (0x1dcd bytes) 000802 1763082158.908410 mapread 0x01be23 thru 0x01e218 (0x23f6 bytes) 000803 1763082158.909386 mapread 0x01d65d thru 0x02183f (0x41e3 bytes) 000806 1763082158.911096 mapread 0x010f59 thru 0x01a812 (0x98ba bytes) 000808 1763082158.913566 write 0x00ca92 thru 0x01b119 (0xe688 bytes) 000809 1763082158.918557 read 0x0271f3 thru 0x0303b0 (0x91be bytes) 000812 1763082158.926609 MIRROR_EXTEND to 3 mirrors 000813 1763082159.604498 trunc from 0x0303b1 to 0x0113cf (0x1efe1 bytes) 000814 1763082159.711208 read 0x002f94 thru 0x010fdc (0xe049 bytes) 000815 1763082159.721260 mapread 0x011059 thru 0x0113cf (0x377 bytes) 000817 1763082159.728059 trunc from 0x0113d0 to 0x032bcb (0x217fc bytes) 000818 1763082159.766004 write 0x006334 thru 0x00eff1 (0x8cbe bytes) 000819 1763082159.767885 trunc from 0x032bcc to 0x03ed58 (0xc18d bytes) 000822 1763082159.790192 trunc from 0x03ed59 to 0x000194 (0x3ebc4 bytes) 000823 1763082159.828163 MIRROR_RESYNC 3 mirrors 000826 1763082160.621283 write 0x032391 thru 0x036a02 (0x4672 bytes) 000828 1763082160.685651 trunc from 0x036a03 to 0x012acc (0x23f36 bytes) 000831 1763082160.717692 mapread 0x011ac1 thru 0x012acc (0x100c bytes) 000832 1763082160.738829 MIRROR_EXTEND to 4 mirrors 000833 1763082161.533915 mapwrite 0x028d47 thru 0x02fedf (0x7199 bytes) 000834 1763082161.725356 mapread 0x011711 thru 0x01fd09 (0xe5f9 bytes) 000837 1763082161.755125 trunc from 0x02fee0 to 0x00a6cc (0x25813 bytes) 000838 1763082161.786719 mapwrite 0x01a7d5 thru 0x0217b5 (0x6fe1 bytes) 000840 1763082161.889166 write 0x009780 thru 0x012e96 (0x9717 bytes) 000843 1763082161.919829 write 0x03db7f thru 0x03f1da (0x165c bytes) 000844 1763082161.921057 read 0x0318d9 thru 0x031fee (0x716 bytes) 000849 1763082161.932768 mapwrite 0x0323c7 thru 0x03b60f (0x9249 bytes) 000855 1763082162.037543 trunc from 0x03f1db to 0x02e929 (0x108b1 bytes) 000856 1763082162.068626 MIRROR_EXTEND to 5 mirrors 000858 1763082162.769997 write 0x032598 thru 0x03b6b4 (0x911d bytes) 000859 1763082162.841183 read 0x0241f8 thru 0x0266a1 (0x24aa bytes) 000861 1763082162.848493 read 0x024d97 thru 0x0311db (0xc445 bytes) 000863 1763082162.868726 trunc from 0x03b6b5 to 0x011f9c (0x29718 bytes) 000864 1763082162.899472 mapwrite 0x00aa38 thru 0x015ba9 (0xb172 bytes) 000866 1763082162.980827 mapread 0x01465a thru 0x015ba9 (0x1550 bytes) 000868 1763082162.982339 write 0x027d15 thru 0x03765c (0xf948 bytes) 000870 1763082162.986027 write 0x01708c thru 0x0197c5 (0x273a bytes) 000872 1763082163.000476 MIRROR_EXTEND to 6 mirrors 000874 1763082163.723201 read 0x024588 thru 0x02a804 (0x627d bytes) 000875 1763082163.734250 write 0x03ddf9 thru 0x03ffff (0x2207 bytes) 000876 1763082163.806189 trunc from 0x040000 to 0x01fc7a (0x20385 bytes) 000882 1763082163.830393 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000886 1763082164.092738 mapread 0x00ba05 thru 0x00f268 (0x3864 bytes) 000887 1763082164.113383 read 0x0183d1 thru 0x01fc7a (0x78aa bytes) 000890 1763082164.125770 MIRROR_RESYNC 5 mirrors 000891 1763082165.187832 mapread 0x00cec4 thru 0x01093b (0x3a78 bytes) 000892 1763082165.244646 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000897 1763082165.554971 read 0x00f3cb thru 0x0148ef (0x5525 bytes) 000904 1763082165.571358 MIRROR_EXTEND to 5 mirrors 000905 1763082166.308248 mapread 0x0151ee thru 0x01cb99 (0x79ac bytes) 000906 1763082166.371983 mapwrite 0x000386 thru 0x006327 (0x5fa2 bytes) 000907 1763082166.554654 MIRROR_EXTEND to 6 mirrors 000908 1763082167.113233 write 0x02a963 thru 0x037db9 (0xd457 bytes) 000910 1763082167.195795 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000911 1763082167.479704 write 0x0385e2 thru 0x03e291 (0x5cb0 bytes) 000912 1763082167.484526 mapread 0x0063d6 thru 0x00cd19 (0x6944 bytes) 000913 1763082167.515473 mapread 0x02d683 thru 0x03313f (0x5abd bytes) 000914 1763082167.525814 mapread 0x03bcb2 thru 0x03e291 (0x25e0 bytes) 000915 1763082167.526728 trunc from 0x03e292 to 0x015d94 (0x284fd bytes) 000917 1763082167.547357 trunc from 0x015d95 to 0x01a58a (0x47f6 bytes) 000918 1763082167.571507 mapread 0x008c3a thru 0x010975 (0x7d3c bytes) 000919 1763082167.572538 mapread 0x016c68 thru 0x01a58a (0x3923 bytes) 000921 1763082167.587628 mapread 0x011869 thru 0x01a58a (0x8d22 bytes) 000922 1763082167.588585 mapread 0x012f00 thru 0x01a58a (0x768b bytes) 000923 1763082167.589459 trunc from 0x01a58b to 0x02b3b9 (0x10e2f bytes) 000924 1763082167.615057 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000925 1763082167.832569 read 0x00fb56 thru 0x01693f (0x6dea bytes) 000926 1763082167.840793 write 0x004e60 thru 0x009c9b (0x4e3c bytes) 000927 1763082167.868898 MIRROR_EXTEND to 5 mirrors 000931 1763082168.566835 read 0x0202b8 thru 0x021932 (0x167b bytes) 000932 1763082168.581200 read 0x019f1f thru 0x0295fa (0xf6dc bytes) 000934 1763082168.610269 read 0x00be0d thru 0x010f90 (0x5184 bytes) 000936 1763082168.649188 write 0x036b8a thru 0x03ffff (0x9476 bytes) 000937 1763082168.720540 mapwrite 0x01d6eb thru 0x022e07 (0x571d bytes) 000938 1763082168.817728 read 0x0050a8 thru 0x00b916 (0x686f bytes) 000939 1763082168.818481 trunc from 0x040000 to 0x01d3ce (0x22c31 bytes) 000940 1763082168.847798 read 0x01610f thru 0x018768 (0x265a bytes) 000941 1763082168.848539 mapread 0x00c1de thru 0x00c5c1 (0x3e4 bytes) 000943 1763082168.849786 mapwrite 0x00dfd4 thru 0x0106f5 (0x2722 bytes) 000944 1763082168.891044 mapread 0x009653 thru 0x016fc4 (0xd972 bytes) 000945 1763082168.893458 trunc from 0x01d3cf to 0x03496e (0x175a0 bytes) 000946 1763082168.911527 read 0x019bd2 thru 0x01c454 (0x2883 bytes) 000947 1763082168.915270 read 0x02ec4c thru 0x03496e (0x5d23 bytes) 000951 1763082168.949519 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000952 1763082169.210760 read 0x0249d5 thru 0x026078 (0x16a4 bytes) 000953 1763082169.218165 trunc from 0x03496f to 0x0099f7 (0x2af77 bytes) 000954 1763082169.235218 trunc from 0x0099f8 to 0x006939 (0x30be bytes) 000957 1763082169.259116 write 0x00fdbf thru 0x01eaa8 (0xecea bytes) 000960 1763082169.266962 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000962 1763082169.492867 read 0x00d909 thru 0x01b648 (0xdd40 bytes) 000963 1763082169.506035 trunc from 0x01eaa9 to 0x023461 (0x49b9 bytes) 000964 1763082169.527204 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000966 1763082169.755856 trunc from 0x023462 to 0x00bd78 (0x176e9 bytes) 000967 1763082169.787763 MIRROR_EXTEND to 3 mirrors 000969 1763082170.464903 write 0x01ff4d thru 0x02bc14 (0xbcc8 bytes) 000970 1763082170.536191 mapwrite 0x03892b thru 0x03e387 (0x5a5d bytes) 000972 1763082170.616029 trunc from 0x03e388 to 0x037ea7 (0x64e0 bytes) 000974 1763082170.629965 trunc from 0x037ea8 to 0x012782 (0x25725 bytes) 000975 1763082170.649357 mapread 0x0007f5 thru 0x002847 (0x2053 bytes) 000976 1763082170.657748 mapwrite 0x004dac thru 0x0135ea (0xe83f bytes) 000979 1763082170.731605 read 0x006813 thru 0x007219 (0xa07 bytes) 000980 1763082170.732186 trunc from 0x0135eb to 0x00673b (0xceaf bytes) 000982 1763082170.749093 read 0x002367 thru 0x00673b (0x43d5 bytes) 000983 1763082170.750512 trunc from 0x00673c to 0x028981 (0x22246 bytes) 000984 1763082170.783289 write 0x023b43 thru 0x0245da (0xa98 bytes) 000985 1763082170.793392 mapwrite 0x01e8a4 thru 0x02a2d5 (0xba32 bytes) 000986 1763082170.877976 trunc from 0x02a2d6 to 0x0393b7 (0xf0e2 bytes) 000987 1763082170.897821 read 0x0210db thru 0x02114b (0x71 bytes) 000988 1763082170.898520 mapread 0x038f0e thru 0x0393b7 (0x4aa bytes) 000989 1763082170.920889 MIRROR_RESYNC 3 mirrors 000992 1763082171.649446 MIRROR_EXTEND to 4 mirrors 000993 1763082172.292808 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) 000995 1763082172.685403 read 0x00729f thru 0x01023f (0x8fa1 bytes) 000999 1763082172.700954 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (55s) == sanity-flr test 71: check mirror extend parent fid ==== 20:02:57 (1763082177) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 20:03:01 (1763082181) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7548672 .. Extending file size to 1311904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7965088 .. Extending file size to 8564096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8656384 .. Extending file size to 7993920 .. Extending file size to 8078016 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8817088 .. Extending file size to 8331872 .. Extending file size to 8346912 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9073280 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8882656 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9208224 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 54784 54785 54786 54787 54790 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:04:17 (1763082257) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6506240 .. Extending file size to 6907072 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6989536 .. Extending file size to 7643392 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8498816 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6941248 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8733888 .. Extending file size to 8956160 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9005568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173088 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56732 56733 56734 56736 56739 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:05:31 (1763082331) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0907399 s, 23.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255667 s, 8.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:05:47 (1763082347) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123582 s, 17.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 20:05:54 (1763082354) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x213:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:06:00 (1763082360) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170857 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:06:05 (1763082365) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164214 s, 12.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:06:11 (1763082371) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182018 s, 11.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:06:42 (1763082402) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116895 s, 9.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:07:37 (1763082457) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23895 s, 8.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:08:34 (1763082514) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. Creating new pool test_204f oleg251-server: Pool lustre.test_204f created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169047 s, 12.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg251-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:09:47 (1763082587) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:09:54 (1763082594) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:10:03 (1763082603) oleg251-server: Pool lustre.test_206 created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg251-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:10:27 (1763082627) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788094 s, 13.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607876 s, 17.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2654f07a Mirror 3: 0x2654f07a PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:10:38 (1763082638) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:10:41 (1763082641) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:10:43 (1763082643) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0871187 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1af0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x25d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x25d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x25d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xbd0000 Read this (4 bytes): BBBB 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:11:11 (1763082671) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0145791 s, 562 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:11:15 (1763082675) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407587 s, 25.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834488 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-13 20:11:16.000000000 -0500 Modify: 2025-11-13 20:11:16.000000000 -0500 Change: 2025-11-13 20:11:17.000000000 -0500 Birth: 2025-11-13 20:11:16.000000000 -0500 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:11:21 (1763082681) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:11:23 (1763082683) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.403444 s, 102 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.056118 s, 73.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 13 20:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 13 20:11 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test complete, duration 1368 sec =========== 20:11:29 (1763082689) === sanity-flr: start cleanup 20:11:30 (1763082690) === === sanity-flr: finish cleanup 20:11:36 (1763082696) ===