-----============= acceptance-small: sanity-flr ============----- Fri Oct 10 15:45:57 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:46:09 (1760125569) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff986890522800.idle_timeout=debug osc.lustre-OST0001-osc-ffff986890522800.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:46:27 (1760125587) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7554] == sanity-flr test 0a: lfs mirror create with -N option == 15:46:28 (1760125588) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:46:37 (1760125597) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:46:39 (1760125599) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:46:41 (1760125601) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:46:55 (1760125615) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:46:57 (1760125617) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 15:46:59 (1760125619) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff986890522800.stats=clear osc.lustre-OST0001-osc-ffff986890522800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:47:04 (1760125624) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:47:11 (1760125631) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:47:16 (1760125636) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:47:18 (1760125638) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 15:47:24 (1760125644) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:47:27 (1760125647) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:47:29 (1760125649) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:47:34 (1760125654) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:47:40 (1760125660) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:47:43 (1760125663) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.646681 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198178 s, 5.3 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:47:50 (1760125670) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.19107 s, 93.3 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:47:54 (1760125674) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:48:32 (1760125712) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.155789 s, 33.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:48:52 (1760125732) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (38s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:49:30 (1760125770) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.55001 s, 20.6 MB/s -rw-r--r-- 1 root root 52428800 Oct 10 15:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.533s user 0m0.000s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff986890522800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.603s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.725s user 0m0.000s sys 0m0.190s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff986890522800.ost_server_uuid in FULL state after 0 sec PASS 33b (46s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:50:16 (1760125816) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:50:18 (1760125818) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:50:20 (1760125820) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 15:50:21 (1760125821) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 15:50:25 (1760125825) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.372563 s, 33.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:50:33 (1760125833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179521 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161428 s, 65.0 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:50:36 (1760125836) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:50:42 (1760125842) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115948 s, 883 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00077713 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000740726 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103694 s, 988 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 15:50:47 (1760125847) create test files with size 15735560 .. 35d9871dd3fbb94210be3150661643be - 37685eb4c54ea961cd78da49d359ec50 - 7d439f42841835177185b28da7efcfea - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 15:50:59 (1760125859) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:51:08 (1760125868) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:51:11 (1760125871) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536434 s, 19.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155919 s, 67.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 15:51:15 (1760125875) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0648853 s, 32.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.154326 s, 27.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12004 s, 17.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 15:51:21 (1760125881) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 15:51:23 (1760125883) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:51:25 (1760125885) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238284 s, 44.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 15:51:29 (1760125889) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:51:31 (1760125891) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:51:35 (1760125895) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.138204 s, 75.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:51:51 (1760125911) llite.lustre-ffff9868880bb800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112095 s, 93.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff986898743800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:51:56 (1760125916) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:52:12 (1760125932) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:52:28 (1760125948) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 15:52:29 (1760125949) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0791847 s, 39.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0634496 s, 49.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 3a9e3f79e0436f7d5b00548cda6ad619 - mirror 1 checksum: ad6864d466b64d03c277d8b94faeb7b8 - mirror 2 checksum: 3a9e3f79e0436f7d5b00548cda6ad619 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 10 15:52:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Oct 10 15:52:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3a9e3f79e0436f7d5b00548cda6ad619 - mirror 1 checksum: ad6864d466b64d03c277d8b94faeb7b8 - mirror 2 checksum: ad6864d466b64d03c277d8b94faeb7b8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 10 15:52:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Oct 10 15:52:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:52:34 (1760125954) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 15:52:37 (1760125957) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 15:52:52 (1760125972) targets are mounted 15:52:52 (1760125972) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153479 s, 68.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:52:59 (1760125979) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:53:00 (1760125980) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:53:02 (1760125982) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00265541 s, 6.2 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:53:04 (1760125984) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:53:06 (1760125986) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:53:07 (1760125987) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:53:14 (1760125994) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:53:34 (1760126014) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:53:52 (1760126032) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:54:11 (1760126051) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 15:54:13 (1760126053) mirror create pid 52250 mirror split pid 52251 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52250 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52251 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 15:55:17 (1760126117) Seed set to 6117 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x2dd0d 000006 1760126117.992913 trunc from 00000000 to 0x02dd0c (0x2dd0d bytes) 000007 1760126118.005660 trunc from 0x02dd0d to 0x008d24 (0x24fe8 bytes) 000008 1760126118.014301 read 0x008501 thru 0x008d24 (0x824 bytes) 000009 1760126118.018323 write 0x0075e0 thru 0x0099b4 (0x23d5 bytes) 000010 1760126118.021733 mapwrite 0x01e844 thru 0x0269f0 (0x81ad bytes) 000013 1760126118.081547 read 0x02283f thru 0x0269f0 (0x41b2 bytes) 000014 1760126118.082188 mapread 0x00013e thru 0x003d38 (0x3bfb bytes) 000015 1760126118.089973 trunc from 0x0269f1 to 0x02339c (0x3654 bytes) 000016 1760126118.099456 mapread 0x011014 thru 0x019e54 (0x8e41 bytes) 000017 1760126118.100376 read 0x0017a2 thru 0x006e09 (0x5668 bytes) 000023 1760126118.103315 trunc from 0x02339d to 0x017cc5 (0xb6d7 bytes) truncating to largest ever: 0x355ec 000025 1760126118.112551 trunc from 0x017cc6 to 0x0355eb (0x1d926 bytes) 000026 1760126118.121725 mapread 0x00ea0b thru 0x01978a (0xad80 bytes) 000027 1760126118.128792 mapread 0x0142ba thru 0x01b2b1 (0x6ff8 bytes) 000030 1760126118.130564 read 0x033590 thru 0x0355eb (0x205c bytes) 000031 1760126118.131235 write 0x00f51e thru 0x019208 (0x9ceb bytes) 000033 1760126118.132757 write 0x010065 thru 0x018147 (0x80e3 bytes) 000034 1760126118.133874 mapwrite 0x001f59 thru 0x0057f3 (0x389b bytes) 000035 1760126118.163153 trunc from 0x0355ec to 0x0092d8 (0x2c313 bytes) 000036 1760126118.172463 read 0x00860c thru 0x0092d8 (0xccd bytes) 000038 1760126118.173464 trunc from 0x0092d9 to 0x0221e3 (0x18f0b bytes) 000039 1760126118.182565 mapread 0x010817 thru 0x016ff6 (0x67e0 bytes) 000040 1760126118.188337 mapread 0x00a156 thru 0x00f5d2 (0x547d bytes) 000041 1760126118.189023 write 0x0287fd thru 0x033133 (0xa937 bytes) 000044 1760126118.190811 read 0x031ad3 thru 0x033133 (0x1661 bytes) 000046 1760126118.191937 trunc from 0x033134 to 0x00ce46 (0x262ed bytes) 000049 1760126118.202997 write 0x030bec thru 0x0330de (0x24f3 bytes) 000050 1760126118.204115 read 0x012bb8 thru 0x0190cd (0x6516 bytes) 000052 1760126118.215019 write 0x010690 thru 0x01f564 (0xeed5 bytes) 000053 1760126118.216455 read 0x019758 thru 0x01c694 (0x2f3d bytes) 000054 1760126118.217063 mapwrite 0x03166b thru 0x034af5 (0x348b bytes) 000059 1760126118.274421 mapread 0x01309b thru 0x01dfb1 (0xaf17 bytes) 000060 1760126118.275495 mapwrite 0x03d679 thru 0x03ffff (0x2987 bytes) 000061 1760126118.315983 mapread 0x01dc44 thru 0x029af7 (0xbeb4 bytes) 000062 1760126118.317013 trunc from 0x040000 to 0x02e7a5 (0x1185a bytes) 000063 1760126118.334282 write 0x00e201 thru 0x0178a3 (0x96a3 bytes) 000068 1760126118.337307 mapread 0x01ac52 thru 0x027bda (0xcf89 bytes) 000071 1760126118.339159 mapwrite 0x02baf1 thru 0x0381e0 (0xc6f0 bytes) 000075 1760126118.417766 trunc from 0x0381e1 to 0x025041 (0x1319f bytes) 000078 1760126118.428517 trunc from 0x025042 to 0x0024e0 (0x22b61 bytes) 000083 1760126118.437612 mapread 0x001d13 thru 0x0024e0 (0x7ce bytes) 000084 1760126118.438459 read 0x001294 thru 0x0024e0 (0x124d bytes) 000085 1760126118.439242 trunc from 0x0024e1 to 0x00b6dc (0x91fc bytes) 000089 1760126118.448084 trunc from 0x00b6dd to 0x01e15e (0x12a82 bytes) 000090 1760126118.456187 trunc from 0x01e15f to 0x011b60 (0xc5fe bytes) 000092 1760126118.465072 write 0x01637b thru 0x0252b2 (0xef38 bytes) 000093 1760126118.466667 write 0x0398e1 thru 0x03ffff (0x671f bytes) 000094 1760126118.467797 mapwrite 0x02cacf thru 0x02e3e2 (0x1914 bytes) 000095 1760126118.506995 mapread 0x0273db thru 0x02a688 (0x32ae bytes) 000096 1760126118.507952 trunc from 0x040000 to 0x023416 (0x1cbe9 bytes) 000097 1760126118.518207 trunc from 0x023417 to 0x01d8b2 (0x5b64 bytes) 000100 1760126118.528722 mapread 0x01a9e3 thru 0x01d8b2 (0x2ed0 bytes) 000105 1760126118.535205 MIRROR_EXTEND to 2 mirrors 000107 1760126118.826227 MIRROR_RESYNC 2 mirrors 000109 1760126118.955560 mapread 0x01776b thru 0x01d8b2 (0x6148 bytes) 000110 1760126118.969590 trunc from 0x01d8b3 to 0x01e26f (0x9bd bytes) 000111 1760126119.039308 write 0x03a80a thru 0x03ffff (0x57f6 bytes) 000112 1760126119.040450 mapwrite 0x03c19b thru 0x03ffff (0x3e65 bytes) 000113 1760126119.075107 mapwrite 0x029590 thru 0x038d72 (0xf7e3 bytes) 000114 1760126119.121562 write 0x01f2ae thru 0x02c5e6 (0xd339 bytes) 000116 1760126119.129438 mapwrite 0x02d2ea thru 0x035f33 (0x8c4a bytes) 000118 1760126119.162402 mapwrite 0x01e4fc thru 0x0256f3 (0x71f8 bytes) 000119 1760126119.187365 trunc from 0x040000 to 0x0228c9 (0x1d736 bytes) 000120 1760126119.199496 read 0x02061d thru 0x0228c9 (0x22ad bytes) 000123 1760126119.201042 mapread 0x01310f thru 0x014ed8 (0x1dca bytes) 000125 1760126119.208608 read 0x022235 thru 0x0228c9 (0x695 bytes) 000127 1760126119.209685 mapread 0x0022e6 thru 0x00a3fe (0x8119 bytes) 000128 1760126119.215370 mapread 0x01cfdc thru 0x01e562 (0x1587 bytes) 000129 1760126119.216274 read 0x00d110 thru 0x0174b8 (0xa3a9 bytes) 000130 1760126119.216897 mapread 0x00b8de thru 0x012b08 (0x722b bytes) 000131 1760126119.217829 write 0x037255 thru 0x03dcee (0x6a9a bytes) 000133 1760126119.219730 mapread 0x02c662 thru 0x03773e (0xb0dd bytes) 000134 1760126119.225225 mapwrite 0x02b2ef thru 0x038cfb (0xda0d bytes) 000135 1760126119.258668 write 0x01704e thru 0x021f02 (0xaeb5 bytes) 000137 1760126119.260564 mapread 0x03ca85 thru 0x03ccd7 (0x253 bytes) 000139 1760126119.265197 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000142 1760126119.353442 write 0x01f098 thru 0x02bdf2 (0xcd5b bytes) 000143 1760126119.360813 write 0x02ae3c thru 0x02c480 (0x1645 bytes) 000147 1760126119.368307 read 0x007847 thru 0x00d019 (0x57d3 bytes) 000148 1760126119.374998 mapwrite 0x007e32 thru 0x00b1aa (0x3379 bytes) 000149 1760126119.406581 trunc from 0x03dcef to 0x01418e (0x29b60 bytes) 000156 1760126119.423117 trunc from 0x01418f to 0x02a73c (0x165ae bytes) 000159 1760126119.434251 mapwrite 0x03e795 thru 0x03ffff (0x186b bytes) 000160 1760126119.476252 trunc from 0x040000 to 0x013a9c (0x2c563 bytes) 000161 1760126119.489257 write 0x033c84 thru 0x03ffff (0xc37c bytes) 000162 1760126119.490739 mapwrite 0x007541 thru 0x013567 (0xc027 bytes) 000164 1760126119.556072 write 0x02727e thru 0x02fd4f (0x8ad2 bytes) 000166 1760126119.566607 mapwrite 0x00e5ad thru 0x016d78 (0x87cc bytes) 000167 1760126119.624015 mapread 0x0394f5 thru 0x03ffff (0x6b0b bytes) 000170 1760126119.625702 read 0x03e652 thru 0x03ffff (0x19ae bytes) 000171 1760126119.626352 mapwrite 0x03d9bc thru 0x03ffff (0x2644 bytes) 000173 1760126119.661091 MIRROR_EXTEND to 2 mirrors 000174 1760126120.024253 mapread 0x023437 thru 0x026edf (0x3aa9 bytes) 000175 1760126120.040012 trunc from 0x040000 to 0x004074 (0x3bf8b bytes) 000176 1760126120.118470 trunc from 0x004075 to 0x000315 (0x3d5f bytes) 000177 1760126120.132535 trunc from 0x000316 to 0x022cc1 (0x229ac bytes) 000178 1760126120.147131 trunc from 0x022cc2 to 0x0338c6 (0x10c05 bytes) 000181 1760126120.163404 trunc from 0x0338c7 to 0x013b6a (0x1fd5c bytes) 000184 1760126120.178778 mapwrite 0x037512 thru 0x0386b5 (0x11a4 bytes) 000185 1760126120.224170 trunc from 0x0386b6 to 0x007100 (0x315b5 bytes) 000186 1760126120.231916 trunc from 0x007101 to 0x011607 (0xa507 bytes) 000187 1760126120.247568 MIRROR_EXTEND to 3 mirrors 000188 1760126120.576513 MIRROR_EXTEND to 4 mirrors 000189 1760126120.934100 read 0x008a3d thru 0x011607 (0x8bcb bytes) 000190 1760126120.943164 write 0x013b4c thru 0x021b00 (0xdfb5 bytes) 000192 1760126120.978545 write 0x00ddce thru 0x01be58 (0xe08b bytes) 000194 1760126120.984880 write 0x022858 thru 0x023f13 (0x16bc bytes) 000195 1760126120.990405 MIRROR_RESYNC 4 mirrors 000196 1760126121.451504 read 0x014a9a thru 0x016007 (0x156e bytes) 000197 1760126121.454959 read 0x01938c thru 0x01f5c8 (0x623d bytes) 000198 1760126121.461688 MIRROR_RESYNC 4 mirrors 000199 1760126121.588723 write 0x001db9 thru 0x006818 (0x4a60 bytes) 000200 1760126121.634406 MIRROR_RESYNC 4 mirrors 000201 1760126122.025977 write 0x004e86 thru 0x00bb68 (0x6ce3 bytes) 000202 1760126122.065546 mapwrite 0x02dd95 thru 0x038ab3 (0xad1f bytes) 000204 1760126122.122629 write 0x02d4b2 thru 0x03ba92 (0xe5e1 bytes) 000206 1760126122.124534 read 0x02b0a1 thru 0x02d1cd (0x212d bytes) 000209 1760126122.126167 mapwrite 0x039e40 thru 0x03fa0a (0x5bcb bytes) 000210 1760126122.174126 write 0x03f4ff thru 0x03fe12 (0x914 bytes) 000213 1760126122.175948 mapread 0x03aa0f thru 0x03fe12 (0x5404 bytes) 000216 1760126122.181748 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000218 1760126122.292801 MIRROR_RESYNC 3 mirrors 000219 1760126122.673119 mapread 0x037fcb thru 0x03fe12 (0x7e48 bytes) 000221 1760126122.685917 mapread 0x006292 thru 0x00e304 (0x8073 bytes) 000222 1760126122.694416 write 0x01486e thru 0x016aad (0x2240 bytes) 000226 1760126122.752388 mapwrite 0x00091b thru 0x004696 (0x3d7c bytes) 000227 1760126122.785269 write 0x033c2c thru 0x034cb7 (0x108c bytes) 000229 1760126122.794602 MIRROR_EXTEND to 4 mirrors 000230 1760126123.119529 mapwrite 0x02e8a8 thru 0x02fe52 (0x15ab bytes) 000231 1760126123.212741 trunc from 0x03fe13 to 0x0092ea (0x36b28 bytes) 000233 1760126123.241104 read 0x005907 thru 0x0092ea (0x39e4 bytes) 000234 1760126123.247807 write 0x039d9e thru 0x03ffff (0x6262 bytes) 000237 1760126123.250019 read 0x001f10 thru 0x00dcb5 (0xbda6 bytes) 000239 1760126123.256791 mapwrite 0x02e65f thru 0x02ec04 (0x5a6 bytes) 000241 1760126123.291800 write 0x008223 thru 0x00cbd1 (0x49af bytes) 000243 1760126123.293519 mapwrite 0x034aa5 thru 0x039779 (0x4cd5 bytes) 000244 1760126123.331970 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000245 1760126123.531830 mapwrite 0x00b682 thru 0x00c1b5 (0xb34 bytes) 000249 1760126123.570307 read 0x01d183 thru 0x01ebda (0x1a58 bytes) 000250 1760126123.577081 mapread 0x00112f thru 0x005d28 (0x4bfa bytes) 000252 1760126123.584989 mapread 0x0082cb thru 0x0091b1 (0xee7 bytes) 000255 1760126123.589436 MIRROR_EXTEND to 4 mirrors 000256 1760126123.925402 mapwrite 0x0319f5 thru 0x03b3f1 (0x99fd bytes) 000258 1760126124.070238 mapread 0x02c38a thru 0x03a41c (0xe093 bytes) 000259 1760126124.078639 MIRROR_EXTEND to 5 mirrors 000260 1760126124.403202 read 0x02fc5c thru 0x0383dd (0x8782 bytes) 000261 1760126124.415270 read 0x00d14b thru 0x0122ba (0x5170 bytes) 000262 1760126124.425962 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000266 1760126124.536055 MIRROR_EXTEND to 5 mirrors 000267 1760126124.788025 write 0x0050bd thru 0x0128a6 (0xd7ea bytes) 000270 1760126124.828399 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000271 1760126124.941716 read 0x037a64 thru 0x03ffff (0x859c bytes) 000272 1760126124.947199 trunc from 0x040000 to 0x0162bf (0x29d40 bytes) 000273 1760126124.962120 read 0x007c75 thru 0x01552f (0xd8bb bytes) 000278 1760126124.968943 write 0x02fc2b thru 0x038236 (0x860c bytes) 000279 1760126124.970208 read 0x020a22 thru 0x028d15 (0x82f4 bytes) 000280 1760126124.974610 write 0x0376f2 thru 0x03ffff (0x890e bytes) 000281 1760126124.975931 read 0x0073a9 thru 0x016fe2 (0xfc3a bytes) 000282 1760126124.976507 mapwrite 0x025547 thru 0x02f2c6 (0x9d80 bytes) 000283 1760126125.004885 mapread 0x0032b2 thru 0x01178f (0xe4de bytes) 000284 1760126125.008842 MIRROR_RESYNC 4 mirrors 000286 1760126125.460096 write 0x005c44 thru 0x01462b (0xe9e8 bytes) 000288 1760126125.501837 read 0x02c785 thru 0x035538 (0x8db4 bytes) 000289 1760126125.508587 write 0x032ecd thru 0x03fae1 (0xcc15 bytes) 000291 1760126125.515143 trunc from 0x040000 to 0x033ffb (0xc004 bytes) 000294 1760126125.535041 MIRROR_EXTEND to 5 mirrors 000296 1760126125.870677 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000297 1760126126.064520 mapwrite 0x035f86 thru 0x0394c8 (0x3543 bytes) 000298 1760126126.145761 mapread 0x01c2e1 thru 0x029dfa (0xdb1a bytes) 000299 1760126126.155412 read 0x00f56e thru 0x015bf2 (0x6685 bytes) 000301 1760126126.163533 MIRROR_EXTEND to 5 mirrors 000302 1760126126.445121 write 0x004ce8 thru 0x008580 (0x3899 bytes) 000303 1760126126.494057 mapwrite 0x00a067 thru 0x00c8f1 (0x288b bytes) 000305 1760126126.530771 read 0x0042d7 thru 0x011f89 (0xdcb3 bytes) 000306 1760126126.531482 read 0x036b03 thru 0x0394c8 (0x29c6 bytes) 000308 1760126126.539085 mapwrite 0x03f898 thru 0x03ffff (0x768 bytes) 000309 1760126126.594682 mapwrite 0x03b451 thru 0x03ffff (0x4baf bytes) 000311 1760126126.620138 mapwrite 0x034395 thru 0x03e91f (0xa58b bytes) 000312 1760126126.648286 mapread 0x02d767 thru 0x02d9cf (0x269 bytes) 000316 1760126126.650376 trunc from 0x040000 to 0x01b50f (0x24af0 bytes) 000317 1760126126.661899 mapwrite 0x00e9cc thru 0x010161 (0x1796 bytes) 000318 1760126126.686974 mapwrite 0x018ec5 thru 0x01f694 (0x67d0 bytes) 000319 1760126126.736824 write 0x02ff6d thru 0x03bf05 (0xbf99 bytes) 000320 1760126126.738703 mapread 0x01d9de thru 0x02273f (0x4d62 bytes) 000322 1760126126.747492 mapread 0x0175f4 thru 0x01d5c5 (0x5fd2 bytes) 000323 1760126126.748388 mapwrite 0x027f51 thru 0x033bfe (0xbcae bytes) 000324 1760126126.794176 MIRROR_RESYNC 5 mirrors 000325 1760126127.287964 mapwrite 0x00b1d7 thru 0x00ba2a (0x854 bytes) 000327 1760126127.369161 trunc from 0x03bf06 to 0x02e201 (0xdd04 bytes) 000328 1760126127.379815 mapwrite 0x02dfca thru 0x03ba6b (0xdaa2 bytes) 000329 1760126127.425934 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000331 1760126127.552529 mapread 0x014bbc thru 0x01d2d4 (0x8719 bytes) 000333 1760126127.566694 trunc from 0x03ba6c to 0x01bc77 (0x1fdf4 bytes) 000335 1760126127.578077 mapwrite 0x01040c thru 0x019fdc (0x9bd1 bytes) 000336 1760126127.604314 write 0x028a4e thru 0x0291cc (0x77f bytes) 000337 1760126127.605316 trunc from 0x0291cd to 0x007180 (0x2204c bytes) 000338 1760126127.614706 write 0x01e6e2 thru 0x02e63f (0xff5e bytes) 000339 1760126127.616385 mapwrite 0x0057ab thru 0x00f07b (0x98d1 bytes) 000340 1760126127.651325 mapread 0x015609 thru 0x01e446 (0x8e3e bytes) 000341 1760126127.657973 read 0x02cc40 thru 0x02e63f (0x1a00 bytes) 000342 1760126127.658812 mapwrite 0x019028 thru 0x019523 (0x4fc bytes) 000346 1760126127.695261 MIRROR_EXTEND to 5 mirrors 000348 1760126127.996679 MIRROR_EXTEND to 6 mirrors 000351 1760126128.333218 trunc from 0x02e640 to 0x01979c (0x14ea3 bytes) 000354 1760126128.389958 mapwrite 0x0155e6 thru 0x0207b1 (0xb1cc bytes) 000355 1760126128.449544 mapwrite 0x000ea2 thru 0x005643 (0x47a2 bytes) 000358 1760126128.480223 mapwrite 0x020d36 thru 0x02f9eb (0xecb6 bytes) 000359 1760126128.527017 trunc from 0x02f9ec to 0x034543 (0x4b58 bytes) 000361 1760126128.537314 write 0x0099cd thru 0x0111c1 (0x77f5 bytes) 000363 1760126128.538630 trunc from 0x034544 to 0x00f46b (0x250d8 bytes) 000364 1760126128.550244 mapread 0x00a582 thru 0x00f46b (0x4eea bytes) 000365 1760126128.554018 MIRROR_EXTEND to 7 mirrors 000367 1760126128.904949 MIRROR_RESYNC 7 mirrors 000369 1760126129.394223 MIRROR_EXTEND to 8 mirrors 000370 1760126129.728360 mapwrite 0x01919f thru 0x01f83f (0x66a1 bytes) 000376 1760126129.825130 mapwrite 0x00a6ef thru 0x00c5d1 (0x1ee3 bytes) 000378 1760126129.861659 mapwrite 0x021615 thru 0x023b29 (0x2515 bytes) 000380 1760126129.904035 trunc from 0x023b2a to 0x0330c9 (0xf5a0 bytes) 000381 1760126129.914020 read 0x0074f6 thru 0x00d412 (0x5f1d bytes) 000383 1760126129.915217 read 0x0110ee thru 0x020f0a (0xfe1d bytes) 000389 1760126129.923634 trunc from 0x0330ca to 0x018382 (0x1ad47 bytes) 000390 1760126129.935740 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000391 1760126130.049611 mapwrite 0x00717e thru 0x0160ae (0xef31 bytes) 000392 1760126130.097875 read 0x0110ce thru 0x018382 (0x72b5 bytes) 000394 1760126130.098827 write 0x021b4b thru 0x027c61 (0x6117 bytes) 000395 1760126130.105420 MIRROR_EXTEND to 8 mirrors 000398 1760126130.420215 mapread 0x01332a thru 0x01e371 (0xb048 bytes) 000401 1760126130.448942 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000402 1760126130.650399 mapread 0x00f487 thru 0x00f5b3 (0x12d bytes) 000407 1760126130.694264 MIRROR_RESYNC 7 mirrors 000411 1760126131.166562 mapwrite 0x00f6a0 thru 0x0164db (0x6e3c bytes) 000413 1760126131.256725 mapwrite 0x02bc7c thru 0x03a36b (0xe6f0 bytes) 000414 1760126131.332731 mapwrite 0x01cc41 thru 0x0282bd (0xb67d bytes) 000417 1760126131.370660 read 0x02742f thru 0x0310dd (0x9caf bytes) 000421 1760126131.372824 mapread 0x00f604 thru 0x01efe6 (0xf9e3 bytes) 000423 1760126131.374157 mapwrite 0x005e39 thru 0x0085ad (0x2775 bytes) 000425 1760126131.406620 write 0x009678 thru 0x010852 (0x71db bytes) 000427 1760126131.408182 mapread 0x009674 thru 0x0139f3 (0xa380 bytes) 000428 1760126131.409035 read 0x028cea thru 0x02ced7 (0x41ee bytes) 000430 1760126131.410033 write 0x025145 thru 0x027a79 (0x2935 bytes) 000431 1760126131.413832 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000432 1760126131.513906 mapread 0x01c8fd thru 0x0223fc (0x5b00 bytes) 000433 1760126131.523161 trunc from 0x03a36c to 0x009ba7 (0x307c4 bytes) 000434 1760126131.531259 mapwrite 0x02cc4a thru 0x033205 (0x65bc bytes) 000437 1760126131.566116 mapwrite 0x014fc9 thru 0x01671a (0x1752 bytes) 000438 1760126131.589571 read 0x0216a3 thru 0x02ae5d (0x97bb bytes) 000439 1760126131.592922 mapread 0x014108 thru 0x0233d1 (0xf2ca bytes) 000442 1760126131.597904 read 0x008aa6 thru 0x00cbbb (0x4116 bytes) 000443 1760126131.602909 mapread 0x000f8f thru 0x00e1f5 (0xd267 bytes) 000444 1760126131.603817 read 0x02e164 thru 0x033205 (0x50a2 bytes) 000445 1760126131.606822 MIRROR_RESYNC 6 mirrors 000446 1760126132.045256 read 0x02872b thru 0x02ef82 (0x6858 bytes) truncating to largest ever: 0x3e3b7 000447 1760126132.049768 trunc from 0x033206 to 0x03e3b6 (0xb1b1 bytes) 000448 1760126132.092030 read 0x001ad1 thru 0x00e2fd (0xc82d bytes) 000449 1760126132.098691 mapwrite 0x0092f0 thru 0x012172 (0x8e83 bytes) 000451 1760126132.127235 MIRROR_EXTEND to 7 mirrors 000454 1760126132.405222 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000457 1760126132.554260 mapread 0x02654d thru 0x0335ab (0xd05f bytes) 000458 1760126132.568681 write 0x02eca8 thru 0x033a30 (0x4d89 bytes) 000461 1760126132.607776 write 0x02e2b8 thru 0x031ec7 (0x3c10 bytes) 000462 1760126132.608726 write 0x03687a thru 0x0385fd (0x1d84 bytes) 000463 1760126132.613807 mapread 0x01f4a9 thru 0x02958b (0xa0e3 bytes) 000464 1760126132.621128 mapread 0x037988 thru 0x03a6de (0x2d57 bytes) 000465 1760126132.621964 mapwrite 0x034673 thru 0x03f214 (0xaba2 bytes) 000466 1760126132.675287 MIRROR_RESYNC 6 mirrors 000467 1760126133.107815 read 0x01de23 thru 0x02b6e6 (0xd8c4 bytes) 000468 1760126133.112111 read 0x021ccc thru 0x02502b (0x3360 bytes) 000469 1760126133.112501 read 0x0057dd thru 0x00698e (0x11b2 bytes) 000472 1760126133.120447 MIRROR_EXTEND to 7 mirrors 000475 1760126133.435395 write 0x01fc03 thru 0x0275b6 (0x79b4 bytes) 000476 1760126133.473989 MIRROR_RESYNC 7 mirrors 000477 1760126133.903643 trunc from 0x03f215 to 0x036c0b (0x8609 bytes) 000478 1760126133.927606 read 0x01d281 thru 0x02aa85 (0xd805 bytes) 000479 1760126133.934652 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000481 1760126134.026836 MIRROR_RESYNC 6 mirrors 000482 1760126134.425236 write 0x03ff23 thru 0x03ffff (0xdd bytes) 000483 1760126134.454079 trunc from 0x040000 to 0x0370ca (0x8f35 bytes) 000484 1760126134.464786 write 0x005166 thru 0x00a597 (0x5432 bytes) 000486 1760126134.471336 read 0x00c629 thru 0x019f2e (0xd906 bytes) 000488 1760126134.477452 read 0x01e499 thru 0x028d47 (0xa8af bytes) 000490 1760126134.483733 mapread 0x01c9c1 thru 0x021f16 (0x5556 bytes) 000491 1760126134.487259 MIRROR_RESYNC 6 mirrors 000492 1760126134.926556 read 0x011cad thru 0x01b398 (0x96ec bytes) truncating to largest ever: 0x3e6fb 000493 1760126134.932282 trunc from 0x0370cb to 0x03e6fa (0x7630 bytes) 000496 1760126134.978230 trunc from 0x03e6fb to 0x003599 (0x3b161 bytes) 000497 1760126134.991239 MIRROR_RESYNC 6 mirrors 000498 1760126135.496631 mapwrite 0x00e568 thru 0x011615 (0x30ae bytes) 000499 1760126135.567635 trunc from 0x011616 to 0x0378b1 (0x2629c bytes) 000501 1760126135.585768 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000502 1760126135.709115 read 0x026712 thru 0x027892 (0x1181 bytes) 000503 1760126135.715312 write 0x026f06 thru 0x030a45 (0x9b40 bytes) 000504 1760126135.720076 read 0x02254d thru 0x02daaf (0xb563 bytes) 000505 1760126135.728287 trunc from 0x0378b2 to 0x00c261 (0x2b650 bytes) 000507 1760126135.741138 write 0x023e5c thru 0x027958 (0x3afd bytes) 000508 1760126135.742245 mapwrite 0x03bf85 thru 0x03ffff (0x407b bytes) truncating to largest ever: 0x3fdf5 000510 1760126135.793266 trunc from 0x040000 to 0x03fdf4 (0x20b bytes) 000512 1760126135.813098 read 0x030760 thru 0x03fdf4 (0xf695 bytes) 000513 1760126135.824027 write 0x02ac7b thru 0x02f7f7 (0x4b7d bytes) 000515 1760126135.825484 trunc from 0x03fdf5 to 0x0382ac (0x7b48 bytes) 000520 1760126135.839064 write 0x038124 thru 0x03cda6 (0x4c83 bytes) 000522 1760126135.840921 mapread 0x02a652 thru 0x033712 (0x90c1 bytes) 000523 1760126135.842716 trunc from 0x03cda7 to 0x02b0ea (0x11cbc bytes) 000524 1760126135.856299 mapread 0x0167e8 thru 0x01e6d5 (0x7eee bytes) 000526 1760126135.863296 MIRROR_EXTEND to 6 mirrors 000527 1760126136.228236 mapread 0x010031 thru 0x01ef05 (0xeed5 bytes) 000528 1760126136.255272 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000529 1760126136.459697 MIRROR_RESYNC 5 mirrors 000531 1760126136.836013 trunc from 0x02b0eb to 0x018749 (0x129a1 bytes) 000532 1760126136.870740 mapread 0x00fc30 thru 0x01658d (0x695e bytes) 000533 1760126136.882597 read 0x002aa3 thru 0x004c03 (0x2161 bytes) 000534 1760126136.886249 read 0x004eea thru 0x00f13c (0xa253 bytes) 000535 1760126136.891802 mapwrite 0x019684 thru 0x028902 (0xf27f bytes) 000536 1760126136.929595 read 0x006b8a thru 0x00bea5 (0x531c bytes) 000537 1760126136.930221 mapwrite 0x03fcbd thru 0x03ffff (0x343 bytes) 000538 1760126136.960110 mapwrite 0x026217 thru 0x0341f2 (0xdfdc bytes) 000543 1760126136.989474 trunc from 0x040000 to 0x026ba0 (0x1945f bytes) 000546 1760126136.999551 read 0x00f61a thru 0x013fa6 (0x498d bytes) 000547 1760126137.000185 mapread 0x0055c7 thru 0x00914d (0x3b87 bytes) 000548 1760126137.000982 mapwrite 0x001841 thru 0x00d9bf (0xc17f bytes) 000549 1760126137.032340 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000551 1760126137.134186 mapwrite 0x01c1e3 thru 0x0293ed (0xd20b bytes) 000552 1760126137.181572 write 0x015f3f thru 0x022d77 (0xce39 bytes) 000553 1760126137.187315 read 0x00fff7 thru 0x01003c (0x46 bytes) 000554 1760126137.187956 write 0x00c5bf thru 0x011bcb (0x560d bytes) 000556 1760126137.196487 MIRROR_RESYNC 4 mirrors 000557 1760126137.505471 read 0x00a684 thru 0x00dde6 (0x3763 bytes) 000558 1760126137.509120 write 0x03cc9e thru 0x03d07a (0x3dd bytes) 000559 1760126137.536132 trunc from 0x03d07b to 0x0353fa (0x7c80 bytes) 000560 1760126137.554559 mapread 0x01ca36 thru 0x021f14 (0x54df bytes) 000561 1760126137.566503 write 0x014d9b thru 0x01d19a (0x8400 bytes) 000562 1760126137.573131 mapread 0x03523f thru 0x0353fa (0x1bc bytes) 000563 1760126137.578560 read 0x00d443 thru 0x01052d (0x30eb bytes) 000565 1760126137.579516 write 0x016ed5 thru 0x0242a2 (0xd3ce bytes) 000566 1760126137.583735 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000567 1760126137.676703 read 0x00b5e8 thru 0x01946e (0xde87 bytes) 000568 1760126137.682006 mapread 0x034e0d thru 0x0353fa (0x5ee bytes) 000571 1760126137.688530 mapread 0x030595 thru 0x0353fa (0x4e66 bytes) 000573 1760126137.693840 read 0x0226a9 thru 0x024193 (0x1aeb bytes) 000574 1760126137.698866 read 0x02ad41 thru 0x0353fa (0xa6ba bytes) 000575 1760126137.699445 mapwrite 0x00eca2 thru 0x01836f (0x96ce bytes) 000576 1760126137.723402 mapwrite 0x030054 thru 0x036601 (0x65ae bytes) 000577 1760126137.757118 read 0x00395b thru 0x00f6a4 (0xbd4a bytes) truncating to largest ever: 0x3ff76 000579 1760126137.758415 trunc from 0x036602 to 0x03ff75 (0x9974 bytes) 000580 1760126137.767421 mapwrite 0x03c73f thru 0x03ffff (0x38c1 bytes) 000581 1760126137.804428 write 0x01a989 thru 0x01e143 (0x37bb bytes) 000582 1760126137.808209 MIRROR_EXTEND to 4 mirrors 000583 1760126138.054763 mapwrite 0x01554d thru 0x01cec3 (0x7977 bytes) 000584 1760126138.143411 write 0x0296c9 thru 0x0308aa (0x71e2 bytes) 000586 1760126138.151410 MIRROR_EXTEND to 5 mirrors 000587 1760126138.435888 write 0x03939e thru 0x03ec8e (0x58f1 bytes) 000588 1760126138.468891 mapwrite 0x01ac34 thru 0x01c5f0 (0x19bd bytes) 000589 1760126138.502334 MIRROR_EXTEND to 6 mirrors 000590 1760126138.729145 trunc from 0x040000 to 0x018ab3 (0x2754c bytes) 000591 1760126138.788180 MIRROR_EXTEND to 7 mirrors 000592 1760126139.032361 MIRROR_RESYNC 7 mirrors 000593 1760126139.456366 MIRROR_EXTEND to 8 mirrors 000594 1760126139.736657 read 0x013d87 thru 0x018ab3 (0x4d2d bytes) 000595 1760126139.740194 trunc from 0x018ab4 to 0x01c0cf (0x361c bytes) 000597 1760126139.778690 trunc from 0x01c0d0 to 0x0398ab (0x1d7dc bytes) 000598 1760126139.790680 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000599 1760126139.887557 write 0x0376e8 thru 0x0379c0 (0x2d9 bytes) 000600 1760126139.892813 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000601 1760126139.995972 mapread 0x0047a0 thru 0x013823 (0xf084 bytes) 000603 1760126140.009598 mapread 0x009c18 thru 0x018f37 (0xf320 bytes) 000606 1760126140.011275 mapread 0x023f86 thru 0x0288f3 (0x496e bytes) 000610 1760126140.017463 trunc from 0x0398ac to 0x021d5e (0x17b4d bytes) 000612 1760126140.028801 MIRROR_EXTEND to 7 mirrors 000613 1760126140.310160 read 0x011b9e thru 0x015660 (0x3ac3 bytes) 000614 1760126140.323377 mapread 0x012ec8 thru 0x01339e (0x4d7 bytes) 000616 1760126140.336871 trunc from 0x021d5f to 0x017a2f (0xa32f bytes) 000617 1760126140.379673 mapwrite 0x0236aa thru 0x02d6da (0xa031 bytes) 000618 1760126140.430730 read 0x00b7b9 thru 0x01b351 (0xfb99 bytes) 000621 1760126140.440921 MIRROR_RESYNC 7 mirrors 000623 1760126141.012080 write 0x032870 thru 0x038030 (0x57c1 bytes) 000624 1760126141.046027 mapwrite 0x02a459 thru 0x036a45 (0xc5ed bytes) 000626 1760126141.089449 mapwrite 0x016fb0 thru 0x026c09 (0xfc5a bytes) 000627 1760126141.142241 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000628 1760126141.277503 write 0x01c52b thru 0x029c4a (0xd720 bytes) 000629 1760126141.285638 read 0x006205 thru 0x0153db (0xf1d7 bytes) 000630 1760126141.292234 read 0x033fd5 thru 0x035a2e (0x1a5a bytes) 000634 1760126141.299388 mapwrite 0x01a58f thru 0x0247d6 (0xa248 bytes) 000635 1760126141.328793 mapwrite 0x0066d9 thru 0x012c25 (0xc54d bytes) 000636 1760126141.359314 mapwrite 0x03b9de thru 0x03ffff (0x4622 bytes) 000638 1760126141.441643 mapread 0x035da1 thru 0x03ac9b (0x4efb bytes) 000639 1760126141.452218 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000640 1760126141.565405 mapread 0x02f93f thru 0x038792 (0x8e54 bytes) 000645 1760126141.581435 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000646 1760126141.680300 write 0x034022 thru 0x03546d (0x144c bytes) 000647 1760126141.686443 mapwrite 0x0122e3 thru 0x0147ed (0x250b bytes) 000648 1760126141.720504 read 0x025d4e thru 0x0352e0 (0xf593 bytes) 000649 1760126141.726405 mapread 0x03db59 thru 0x03ffff (0x24a7 bytes) 000650 1760126141.731067 mapread 0x0369a3 thru 0x03cc9f (0x62fd bytes) 000651 1760126141.731931 mapwrite 0x02ccda thru 0x03a368 (0xd68f bytes) 000654 1760126141.772319 trunc from 0x040000 to 0x030bdc (0xf423 bytes) 000655 1760126141.786407 trunc from 0x030bdd to 0x00e8a0 (0x2233c bytes) 000656 1760126141.812782 MIRROR_EXTEND to 5 mirrors 000657 1760126142.106538 mapwrite 0x00aeba thru 0x0127f3 (0x793a bytes) 000660 1760126142.201547 read 0x0117ef thru 0x0127f3 (0x1005 bytes) 000661 1760126142.202169 trunc from 0x0127f4 to 0x029f1d (0x1772a bytes) 000662 1760126142.217923 read 0x027f8e thru 0x029f1d (0x1f90 bytes) 000663 1760126142.227352 mapread 0x000fc2 thru 0x00189f (0x8de bytes) 000666 1760126142.229217 write 0x022b97 thru 0x0280f5 (0x555f bytes) 000667 1760126142.230257 mapwrite 0x01f4aa thru 0x01fb8a (0x6e1 bytes) 000668 1760126142.254076 trunc from 0x029f1e to 0x03654e (0xc631 bytes) 000670 1760126142.268907 MIRROR_RESYNC 5 mirrors 000672 1760126142.684091 read 0x014d69 thru 0x0232c4 (0xe55c bytes) 000673 1760126142.691507 MIRROR_RESYNC 5 mirrors 000675 1760126142.799299 mapread 0x00be2f thru 0x01573b (0x990d bytes) 000676 1760126142.807563 trunc from 0x03654f to 0x032c4f (0x38ff bytes) 000679 1760126142.846338 MIRROR_EXTEND to 6 mirrors 000680 1760126143.100878 write 0x03b67b thru 0x03c99b (0x1321 bytes) 000682 1760126143.128593 mapread 0x030ca4 thru 0x03c99b (0xbcf8 bytes) 000683 1760126143.137386 mapwrite 0x035738 thru 0x03976e (0x4037 bytes) 000684 1760126143.156787 mapread 0x03152e thru 0x03c99b (0xb46e bytes) 000689 1760126143.159205 mapwrite 0x014330 thru 0x01ebfb (0xa8cc bytes) 000690 1760126143.189391 mapread 0x01efcf thru 0x02e06a (0xf09c bytes) 000692 1760126143.195116 write 0x03cef2 thru 0x03ffff (0x310e bytes) 000693 1760126143.196190 read 0x027213 thru 0x031933 (0xa721 bytes) 000695 1760126143.197169 mapread 0x01b804 thru 0x021fc7 (0x67c4 bytes) 000699 1760126143.199070 write 0x0153bd thru 0x01a488 (0x50cc bytes) 000702 1760126143.200778 write 0x03bb07 thru 0x03ffff (0x44f9 bytes) 000703 1760126143.201692 read 0x019a97 thru 0x020019 (0x6583 bytes) 000704 1760126143.202205 mapread 0x01ad51 thru 0x02657b (0xb82b bytes) 000708 1760126143.204202 write 0x00c581 thru 0x01481c (0x829c bytes) 000712 1760126143.213946 MIRROR_RESYNC 6 mirrors 000714 1760126143.603377 read 0x0010d1 thru 0x0060b4 (0x4fe4 bytes) 000715 1760126143.611473 MIRROR_RESYNC 6 mirrors 000717 1760126143.736131 trunc from 0x040000 to 0x038b22 (0x74dd bytes) 000719 1760126143.779697 mapwrite 0x02bc90 thru 0x02d408 (0x1779 bytes) 000721 1760126143.822008 read 0x0179cc thru 0x01cf6c (0x55a1 bytes) 000722 1760126143.827145 read 0x0291f4 thru 0x037d59 (0xeb66 bytes) 000723 1760126143.834058 trunc from 0x038b23 to 0x03329c (0x5886 bytes) 000724 1760126143.843187 trunc from 0x03329d to 0x011c25 (0x21677 bytes) 000725 1760126143.850240 write 0x01f639 thru 0x02157c (0x1f44 bytes) 000726 1760126143.853399 MIRROR_EXTEND to 7 mirrors 000727 1760126144.075204 mapread 0x01b263 thru 0x01d674 (0x2412 bytes) 000728 1760126144.098496 MIRROR_RESYNC 7 mirrors 000730 1760126144.511584 write 0x029f17 thru 0x0317b7 (0x78a1 bytes) 000731 1760126144.543577 trunc from 0x0317b8 to 0x007859 (0x29f5e bytes) 000732 1760126144.565629 MIRROR_RESYNC 7 mirrors 000733 1760126144.991556 trunc from 0x00785a to 0x03d05c (0x35803 bytes) 000734 1760126145.020974 mapread 0x03cd94 thru 0x03d05c (0x2c9 bytes) 000736 1760126145.030362 write 0x03fcf8 thru 0x03ffff (0x308 bytes) 000737 1760126145.034603 MIRROR_EXTEND to 8 mirrors 000739 1760126145.305786 read 0x0010a3 thru 0x00dcb6 (0xcc14 bytes) 000740 1760126145.313949 mapwrite 0x008795 thru 0x0174c6 (0xed32 bytes) 000741 1760126145.400751 mapread 0x01cb14 thru 0x026c17 (0xa104 bytes) 000742 1760126145.409962 write 0x01d9da thru 0x023ad3 (0x60fa bytes) 000745 1760126145.415074 MIRROR_RESYNC 8 mirrors 000746 1760126145.906424 mapread 0x02aec4 thru 0x03286b (0x79a8 bytes) 000747 1760126145.916082 write 0x01bd0a thru 0x02afbb (0xf2b2 bytes) 000748 1760126145.955514 MIRROR_EXTEND to 9 mirrors 000749 1760126146.200980 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000753 1760126146.375697 mapread 0x035800 thru 0x0373da (0x1bdb bytes) 000755 1760126146.408713 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000756 1760126146.573691 mapread 0x034925 thru 0x03dd53 (0x942f bytes) 000758 1760126146.609701 MIRROR_EXTEND to 8 mirrors 000761 1760126146.890018 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000762 1760126147.057793 MIRROR_EXTEND to 8 mirrors 000766 1760126147.350779 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000768 1760126147.500516 write 0x006448 thru 0x0127e6 (0xc39f bytes) 000770 1760126147.532823 mapread 0x00a4d0 thru 0x0153ac (0xaedd bytes) 000772 1760126147.538472 trunc from 0x040000 to 0x00a898 (0x35767 bytes) 000774 1760126147.549160 trunc from 0x00a899 to 0x02aa54 (0x201bc bytes) 000775 1760126147.556041 trunc from 0x02aa55 to 0x0377e6 (0xcd92 bytes) 000776 1760126147.562808 trunc from 0x0377e7 to 0x02159c (0x1624a bytes) 000778 1760126147.571467 write 0x01cc88 thru 0x0205a0 (0x3919 bytes) 000779 1760126147.579453 MIRROR_RESYNC 7 mirrors 000780 1760126148.073129 read 0x0116fd thru 0x01cb37 (0xb43b bytes) 000781 1760126148.078088 read 0x00566f thru 0x008c18 (0x35aa bytes) 000784 1760126148.083526 mapread 0x00d43c thru 0x0124ae (0x5073 bytes) 000785 1760126148.084084 mapwrite 0x030051 thru 0x03b21e (0xb1ce bytes) 000787 1760126148.149668 mapwrite 0x006954 thru 0x01069f (0x9d4c bytes) 000788 1760126148.209204 mapwrite 0x028d58 thru 0x032d0d (0x9fb6 bytes) 000792 1760126148.236157 mapwrite 0x00a67a thru 0x0120a7 (0x7a2e bytes) 000794 1760126148.274722 MIRROR_EXTEND to 8 mirrors 000797 1760126148.562583 write 0x02cfeb thru 0x03a6ec (0xd702 bytes) 000798 1760126148.598966 mapwrite 0x0342c6 thru 0x03e6f8 (0xa433 bytes) 000799 1760126148.656736 trunc from 0x03e6f9 to 0x02ce48 (0x118b0 bytes) 000800 1760126148.667301 write 0x01a76d thru 0x027794 (0xd028 bytes) 000802 1760126148.679631 MIRROR_EXTEND to 9 mirrors 000803 1760126148.924565 MIRROR_EXTEND to 10 mirrors 000804 1760126149.203916 trunc from 0x02ce49 to 0x011dd7 (0x1b071 bytes) 000805 1760126149.244828 read 0x00b942 thru 0x011dd7 (0x6496 bytes) 000806 1760126149.248617 mapwrite 0x008830 thru 0x01639b (0xdb6c bytes) 000808 1760126149.294088 mapread 0x0035e6 thru 0x0123a8 (0xedc3 bytes) 000809 1760126149.297463 MIRROR_RESYNC 10 mirrors 000811 1760126149.882816 write 0x01d4cf thru 0x022b69 (0x569b bytes) 000813 1760126149.916546 mapwrite 0x03a416 thru 0x03e696 (0x4281 bytes) 000814 1760126149.983164 trunc from 0x03e697 to 0x036bdc (0x7aba bytes) 000815 1760126149.994233 trunc from 0x036bdd to 0x03f4bf (0x88e3 bytes) 000819 1760126150.006784 trunc from 0x03f4c0 to 0x002f38 (0x3c587 bytes) 000821 1760126150.019227 mapwrite 0x01e8a3 thru 0x01ee33 (0x591 bytes) 000822 1760126150.063615 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000825 1760126150.185048 MIRROR_RESYNC 9 mirrors 000826 1760126150.781327 trunc from 0x01ee34 to 0x03a28c (0x1b459 bytes) 000828 1760126150.812445 mapread 0x02518e thru 0x029c86 (0x4af9 bytes) 000830 1760126150.824980 read 0x030630 thru 0x03a28c (0x9c5d bytes) 000831 1760126150.831281 mapread 0x0316be thru 0x0331f7 (0x1b3a bytes) 000832 1760126150.835075 MIRROR_RESYNC 9 mirrors 000833 1760126151.447317 MIRROR_EXTEND to 10 mirrors 000834 1760126151.836405 trunc from 0x03a28d to 0x02f802 (0xaa8a bytes) 000836 1760126151.877048 write 0x03ad98 thru 0x03ffff (0x5268 bytes) 000837 1760126151.881277 MIRROR_RESYNC 10 mirrors 000839 1760126152.596921 write 0x03bdf3 thru 0x03f57b (0x3789 bytes) 000840 1760126152.646657 read 0x0137a5 thru 0x01f170 (0xb9cc bytes) 000841 1760126152.653661 read 0x00f7ac thru 0x01e915 (0xf16a bytes) 000842 1760126152.662579 trunc from 0x040000 to 0x014aa9 (0x2b556 bytes) 000844 1760126152.678176 mapread 0x001b19 thru 0x00eb8b (0xd073 bytes) 000845 1760126152.679201 trunc from 0x014aaa to 0x01f698 (0xabef bytes) 000847 1760126152.691108 mapwrite 0x01a05b thru 0x028dd2 (0xed78 bytes) 000848 1760126152.802798 mapread 0x01b44d thru 0x01c676 (0x122a bytes) 000850 1760126152.807776 MIRROR_EXTEND to 11 mirrors 000855 1760126153.136999 write 0x002aa2 thru 0x004b1b (0x207a bytes) 000856 1760126153.188533 mapwrite 0x01f91a thru 0x02b8a2 (0xbf89 bytes) 000857 1760126153.260295 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000860 1760126153.392374 mapread 0x019e03 thru 0x01e892 (0x4a90 bytes) 000862 1760126153.405824 mapread 0x02719b thru 0x02b8a2 (0x4708 bytes) 000867 1760126153.413353 trunc from 0x02b8a3 to 0x01b9eb (0xfeb7 bytes) 000868 1760126153.424413 mapwrite 0x023781 thru 0x02863e (0x4ebe bytes) 000872 1760126153.474211 mapread 0x006382 thru 0x01345c (0xd0db bytes) 000873 1760126153.488915 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000874 1760126153.638474 mapread 0x010039 thru 0x010a56 (0xa1e bytes) 000875 1760126153.645672 mapread 0x003f01 thru 0x0079de (0x3ade bytes) 000877 1760126153.659680 mapwrite 0x0007ae thru 0x001569 (0xdbc bytes) 000878 1760126153.686301 mapread 0x02617a thru 0x02863e (0x24c5 bytes) 000880 1760126153.692304 mapwrite 0x01c17c thru 0x01c74d (0x5d2 bytes) 000882 1760126153.719653 mapread 0x007c10 thru 0x007eda (0x2cb bytes) 000883 1760126153.720566 write 0x011114 thru 0x01b6e4 (0xa5d1 bytes) 000885 1760126153.725130 mapread 0x019f7f thru 0x01c89b (0x291d bytes) 000888 1760126153.726838 trunc from 0x02863f to 0x006c6a (0x219d4 bytes) 000890 1760126153.739490 write 0x03b41f thru 0x03ffff (0x4be1 bytes) 000891 1760126153.741456 write 0x011ac5 thru 0x015f79 (0x44b5 bytes) 000893 1760126153.748970 trunc from 0x040000 to 0x031739 (0xe8c6 bytes) 000894 1760126153.759226 mapwrite 0x025dbd thru 0x0336b0 (0xd8f4 bytes) 000895 1760126153.819163 MIRROR_EXTEND to 10 mirrors 000896 1760126154.108777 MIRROR_RESYNC 10 mirrors 000897 1760126154.721094 write 0x0388fb thru 0x03ac1c (0x2322 bytes) 000898 1760126154.773947 trunc from 0x03ac1d to 0x018c4f (0x21fcd bytes) 000899 1760126154.815967 mapwrite 0x028811 thru 0x02a781 (0x1f71 bytes) 000901 1760126154.884293 mapwrite 0x03456d thru 0x038afd (0x4591 bytes) 000902 1760126154.930269 mapread 0x03057d thru 0x0338c6 (0x334a bytes) 000903 1760126154.931189 mapread 0x00b876 thru 0x01200f (0x679a bytes) 000904 1760126154.939899 trunc from 0x038afe to 0x016394 (0x22769 bytes) 000908 1760126154.952937 trunc from 0x016395 to 0x020f78 (0xabe4 bytes) 000909 1760126154.962896 mapread 0x002c25 thru 0x008fd8 (0x63b4 bytes) 000911 1760126154.968605 mapread 0x018dc7 thru 0x01ebe2 (0x5e1c bytes) 000912 1760126154.974239 write 0x01c36b thru 0x024feb (0x8c81 bytes) 000913 1760126154.975640 trunc from 0x024fec to 0x02124d (0x3d9e bytes) 000915 1760126154.987316 mapread 0x01888d thru 0x01b3ed (0x2b61 bytes) 000917 1760126154.991606 MIRROR_RESYNC 10 mirrors 000918 1760126155.700561 read 0x013dc6 thru 0x01db09 (0x9d44 bytes) 000919 1760126155.705175 write 0x03e692 thru 0x03ffff (0x196e bytes) 000920 1760126155.733532 trunc from 0x040000 to 0x024e6b (0x1b194 bytes) 000921 1760126155.745124 write 0x03c78d thru 0x03ffff (0x3873 bytes) 000923 1760126155.746616 write 0x03e7f0 thru 0x03ffff (0x1810 bytes) 000926 1760126155.750744 MIRROR_EXTEND to 11 mirrors 000927 1760126156.098004 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000929 1760126156.305581 read 0x0266f3 thru 0x028d58 (0x2666 bytes) 000930 1760126156.321132 mapwrite 0x004c2d thru 0x012651 (0xda25 bytes) 000932 1760126156.460738 write 0x0025b0 thru 0x00b131 (0x8b82 bytes) 000935 1760126156.462714 trunc from 0x040000 to 0x020000 (0x1ffff bytes) 000937 1760126156.479680 trunc from 0x020001 to 0x03d132 (0x1d132 bytes) 000938 1760126156.489600 trunc from 0x03d133 to 0x00d348 (0x2fdea bytes) 000939 1760126156.500333 read 0x00651c thru 0x00ced8 (0x69bd bytes) 000943 1760126156.502307 mapwrite 0x033821 thru 0x03bc2a (0x840a bytes) 000944 1760126156.551892 read 0x023cbe thru 0x02a604 (0x6947 bytes) 000945 1760126156.555626 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000946 1760126156.659658 write 0x00a2ee thru 0x019783 (0xf496 bytes) 000947 1760126156.666120 mapwrite 0x01ceb6 thru 0x02b186 (0xe2d1 bytes) 000948 1760126156.707363 trunc from 0x03bc2b to 0x02e7fc (0xd42e bytes) 000950 1760126156.718627 read 0x021ccc thru 0x0294af (0x77e4 bytes) 000951 1760126156.723304 MIRROR_RESYNC 9 mirrors 000954 1760126157.263988 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x422:0x0]: Invalid argument (22) 000955 1760126157.465472 mapread 0x0018c3 thru 0x00ea4b (0xd189 bytes) 000957 1760126157.477334 mapread 0x0243ec thru 0x0249f3 (0x608 bytes) 000958 1760126157.483015 mapread 0x022481 thru 0x02e7fc (0xc37c bytes) 000959 1760126157.487779 trunc from 0x02e7fd to 0x0381d7 (0x99db bytes) 000961 1760126157.529201 mapread 0x02f434 thru 0x031309 (0x1ed6 bytes) 000962 1760126157.536126 read 0x00e336 thru 0x015fb8 (0x7c83 bytes) 000963 1760126157.542058 trunc from 0x0381d8 to 0x023f74 (0x14263 bytes) 000964 1760126157.552116 read 0x01dab1 thru 0x023f74 (0x64c4 bytes) 000965 1760126157.558805 mapread 0x005981 thru 0x00e905 (0x8f85 bytes) 000967 1760126157.560247 mapread 0x00c7a0 thru 0x01c5bd (0xfe1e bytes) 000969 1760126157.561519 mapread 0x00f285 thru 0x01a0eb (0xae67 bytes) 000970 1760126157.562374 mapread 0x014c45 thru 0x016a18 (0x1dd4 bytes) 000972 1760126157.563519 read 0x0150d9 thru 0x018467 (0x338f bytes) 000973 1760126157.564060 mapread 0x00f898 thru 0x01c4d9 (0xcc42 bytes) 000974 1760126157.564833 write 0x03d924 thru 0x03ffff (0x26dc bytes) 000976 1760126157.569198 MIRROR_RESYNC 8 mirrors 000980 1760126158.018514 write 0x0046fc thru 0x00e06e (0x9973 bytes) 000982 1760126158.052576 read 0x022824 thru 0x02aef2 (0x86cf bytes) 000985 1760126158.058082 mapwrite 0x03fd8e thru 0x03ffff (0x272 bytes) 000986 1760126158.080469 mapwrite 0x010862 thru 0x01f83f (0xefde bytes) 000987 1760126158.109415 write 0x025231 thru 0x02d92f (0x86ff bytes) 000989 1760126158.113229 write 0x00f562 thru 0x01d351 (0xddf0 bytes) 000990 1760126158.114478 read 0x0226fb thru 0x02ed6e (0xc674 bytes) 000991 1760126158.120080 write 0x01e984 thru 0x0294f3 (0xab70 bytes) 000992 1760126158.121142 mapread 0x024e43 thru 0x03219f (0xd35d bytes) 000993 1760126158.122054 trunc from 0x040000 to 0x0206ac (0x1f953 bytes) 000994 1760126158.137878 read 0x017496 thru 0x01af35 (0x3aa0 bytes) 000996 1760126158.141495 MIRROR_RESYNC 8 mirrors 000998 1760126158.562498 write 0x007088 thru 0x00cf61 (0x5eda bytes) All operations completed A-OK! PASS 70a (43s) == sanity-flr test 71: check mirror extend parent fid ==== 15:56:00 (1760126160) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 15:56:02 (1760126162) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7147744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1925152 .. done Extending file size to 4216800 .. Extending file size to 8971616 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8159488 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9114432 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9039616 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9293824 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56641 56642 56644 56646 56649 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:57:08 (1760126228) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2415360 .. Extending file size to 1482880 .. done Extending file size to 3261280 .. Extending file size to 7557376 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3821504 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5182400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6883264 .. Extending file size to 7254912 .. Extending file size to 7794912 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8192224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8734496 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8755936 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8780096 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8858432 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8825408 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9116768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 60057 60058 60060 60062 60064 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:58:14 (1760126294) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0553447 s, 37.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161336 s, 13.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:58:22 (1760126302) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0394043 s, 53.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 15:58:25 (1760126305) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x475:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:58:28 (1760126308) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0832976 s, 25.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x610:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:58:31 (1760126311) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0861773 s, 24.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x611:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x612:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x611:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x612:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60d:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:58:34 (1760126314) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0667694 s, 31.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x613:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x613:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60f:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:58:59 (1760126339) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341918 s, 30.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x614:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x611:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x610:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x614:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x611:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x610:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x615:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:59:42 (1760126382) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x613:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250844 s, 8.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x613:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x616:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x612:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x613:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x616:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:00:40 (1760126440) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217652 s, 9.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x614:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x617:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x614:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x617:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:01:57 (1760126517) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:02:03 (1760126523) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:02:09 (1760126529) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:02:30 (1760126550) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566268 s, 18.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238639 s, 43.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfd245eec Mirror 3: 0xfd245eec PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:02:38 (1760126558) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:02:41 (1760126561) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:02:44 (1760126564) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0739355 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1440000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (30s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:03:14 (1760126594) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0183819 s, 446 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:03:19 (1760126599) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531623 s, 19.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x49a lmm_fid: [0x200000404:0x49a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x49a lmm_fid: [0x200000404:0x49a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x622:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835098 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-10 16:03:20.000000000 -0400 Modify: 2025-10-10 16:03:21.000000000 -0400 Change: 2025-10-10 16:03:22.000000000 -0400 Birth: 2025-10-10 16:03:21.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:03:26 (1760126606) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:03:28 (1760126608) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.519309 s, 78.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0480318 s, 85.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 10 16:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x49c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 10 16:03 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test complete, duration 1057 sec =========== 16:03:36 (1760126616) === sanity-flr: start cleanup 16:03:37 (1760126617) === === sanity-flr: finish cleanup 16:03:42 (1760126622) ===