-----============= acceptance-small: sanity-flr ============----- Mon Mar 10 13:09:30 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg427-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:09:35 (1741626575) === oleg427-client.virtnet: executing check_config_client /mnt/lustre oleg427-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg427-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800af282800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800af282800.idle_timeout=debug disable quota as required oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:09:43 (1741626583) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7019] == sanity-flr test 0a: lfs mirror create with -N option == 13:09:43 (1741626583) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:09:47 (1741626587) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:09:48 (1741626588) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:09:49 (1741626589) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:09:56 (1741626596) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:09:57 (1741626597) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 13:09:58 (1741626598) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800af282800.stats=clear osc.lustre-OST0001-osc-ffff8800af282800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:10:00 (1741626600) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:10:02 (1741626602) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:10:04 (1741626604) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:10:05 (1741626605) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 13:10:08 (1741626608) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:10:10 (1741626610) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:10:11 (1741626611) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:10:13 (1741626613) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:10:15 (1741626615) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:10:17 (1741626617) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.602162 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042567 s, 24.6 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:10:20 (1741626620) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.0885567 s, 166 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:10:22 (1741626622) Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:10:48 (1741626648) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0534085 s, 58.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:10:57 (1741626657) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (15s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:11:12 (1741626672) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.730611 s, 71.8 MB/s -rw-r--r-- 1 root root 52428800 Mar 10 13:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.201s user 0m0.000s sys 0m0.103s Stopping /mnt/lustre-ost1 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.192s user 0m0.000s sys 0m0.094s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg427-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg427-server oleg427-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.184s user 0m0.001s sys 0m0.094s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg427-server: oleg427-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg427-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af282800.ost_server_uuid in FULL state after 0 sec PASS 33b (16s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:11:28 (1741626688) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:11:29 (1741626689) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:11:29 (1741626689) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:11:30 (1741626690) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 13:11:32 (1741626692) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0918641 s, 137 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:11:34 (1741626694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00682483 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00770184 s, 136 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:11:35 (1741626695) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:11:38 (1741626698) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000513527 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000526065 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000546266 s, 1.9 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000497302 s, 2.1 MB/s flv=10 olv=10 PASS 36d (1s) == sanity-flr test 37: mirror I/O API verification ======= 13:11:40 (1741626700) create test files with size 15758315 .. 60a67eaf841623af4db6f571c25d64f3 - 256a75ab2ef2097f80e2ef82244216ec - 8faedb0d7611ff4839d3f162c7c2b46e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 13:11:44 (1741626704) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:11:46 (1741626706) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:11:48 (1741626708) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179832 s, 58.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107701 s, 97.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 13:11:49 (1741626709) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0248676 s, 84.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0750295 s, 55.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.032751 s, 64.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 13:11:51 (1741626711) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:11:52 (1741626712) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:11:53 (1741626713) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.008343 s, 126 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 13:11:54 (1741626714) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:11:55 (1741626715) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:11:57 (1741626717) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0533368 s, 197 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:12:21 (1741626741) llite.lustre-ffff8801311f1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0486614 s, 215 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg427-client.virtnet /mnt/lustre (opts:) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800af283800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:12:24 (1741626744) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:12:30 (1741626750) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:12:35 (1741626755) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:12:36 (1741626756) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0475027 s, 66.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0374652 s, 84.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: abfe5939e9eb90ff902afd221477f58b - mirror 1 checksum: 46a72d5ab0082fb74cf8391344687701 - mirror 2 checksum: abfe5939e9eb90ff902afd221477f58b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 10 13:12:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 10 13:12:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: abfe5939e9eb90ff902afd221477f58b - mirror 1 checksum: 46a72d5ab0082fb74cf8391344687701 - mirror 2 checksum: 46a72d5ab0082fb74cf8391344687701 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 10 13:12:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 10 13:12:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:12:38 (1741626758) Failing mds1 on oleg427-server Stopping /mnt/lustre-mds1 (opts:) on oleg427-server 13:12:40 (1741626760) shut down facet: mds1 facet_host: oleg427-server facet_failover_host: oleg427-server Failover mds1 to oleg427-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 13:12:52 (1741626772) targets are mounted 13:12:52 (1741626772) facet_failover done oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00712619 s, 147 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:12:57 (1741626777) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:12:57 (1741626777) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:12:58 (1741626778) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00146335 s, 11.2 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:13:00 (1741626780) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:13:00 (1741626780) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:13:01 (1741626781) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:13:05 (1741626785) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:13:22 (1741626802) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:13:39 (1741626819) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:13:56 (1741626836) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 13:13:57 (1741626837) mirror create pid 22446 mirror split pid 22447 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22446 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22447 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 13:14:59 (1741626899) Seed set to 6899 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1741626899.644869 mapwrite 0x02f108 thru 0x0353cf (0x62c8 bytes) 000004 1741626899.681679 mapread 0x031c31 thru 0x0353cf (0x379f bytes) truncating to largest ever: 0x155f8 000010 1741626899.685012 trunc from 0x0353d0 to 0x0155f7 (0x1fdd8 bytes) 000012 1741626899.689831 write 0x018987 thru 0x0230db (0xa755 bytes) 000015 1741626899.691795 trunc from 0x0230dc to 0x00fa2b (0x136b0 bytes) 000016 1741626899.696427 mapwrite 0x02a54a thru 0x02c873 (0x232a bytes) 000017 1741626899.715570 MIRROR_EXTEND to 2 mirrors 000018 1741626899.811808 read 0x01dcc6 thru 0x021dc7 (0x4102 bytes) 000019 1741626899.813673 read 0x0132c4 thru 0x020ef5 (0xdc32 bytes) 000021 1741626899.817012 mapwrite 0x003194 thru 0x010bd1 (0xda3e bytes) 000022 1741626899.872239 write 0x02b08f thru 0x032f13 (0x7e85 bytes) 000023 1741626899.873957 mapread 0x032dfc thru 0x032f13 (0x118 bytes) 000027 1741626899.875114 mapwrite 0x02865f thru 0x02efff (0x69a1 bytes) 000028 1741626899.890459 mapread 0x01566c thru 0x024e7d (0xf812 bytes) 000029 1741626899.906867 mapwrite 0x00cb73 thru 0x00d3e3 (0x871 bytes) 000031 1741626899.917779 mapwrite 0x01be9d thru 0x02318a (0x72ee bytes) 000033 1741626899.931123 mapread 0x02ac75 thru 0x02dd8b (0x3117 bytes) 000038 1741626899.933162 mapread 0x02f8dd thru 0x0303c4 (0xae8 bytes) 000039 1741626899.933670 mapread 0x000996 thru 0x008658 (0x7cc3 bytes) 000040 1741626899.938235 write 0x003966 thru 0x008f8c (0x5627 bytes) 000041 1741626899.939064 mapwrite 0x02b392 thru 0x02e793 (0x3402 bytes) 000043 1741626899.951130 mapwrite 0x02c780 thru 0x03be1f (0xf6a0 bytes) 000044 1741626899.975723 mapwrite 0x004e7a thru 0x005c15 (0xd9c bytes) 000045 1741626899.987984 read 0x006fb4 thru 0x00db8b (0x6bd8 bytes) truncating to largest ever: 0x325d2 000049 1741626899.989672 trunc from 0x03be20 to 0x0325d1 (0x984e bytes) 000050 1741626899.993516 write 0x005252 thru 0x005d58 (0xb07 bytes) 000053 1741626899.994869 write 0x018f53 thru 0x0282b2 (0xf360 bytes) 000054 1741626899.995839 write 0x018a52 thru 0x023da2 (0xb351 bytes) 000055 1741626899.996718 read 0x00f009 thru 0x01c602 (0xd5fa bytes) 000056 1741626900.000113 MIRROR_EXTEND to 3 mirrors 000057 1741626900.122863 mapwrite 0x01ca57 thru 0x02910d (0xc6b7 bytes) 000058 1741626900.174388 trunc from 0x0325d2 to 0x0314c3 (0x110e bytes) truncating to largest ever: 0x3ac26 000061 1741626900.179300 trunc from 0x0314c4 to 0x03ac25 (0x9762 bytes) 000062 1741626900.182465 mapread 0x037a6e thru 0x03ac25 (0x31b8 bytes) 000065 1741626900.186397 MIRROR_EXTEND to 4 mirrors 000066 1741626900.283804 mapread 0x036fa6 thru 0x03a294 (0x32ef bytes) 000067 1741626900.289934 read 0x01310b thru 0x01a709 (0x75ff bytes) 000070 1741626900.295374 MIRROR_RESYNC 4 mirrors 000072 1741626900.408554 write 0x028482 thru 0x035800 (0xd37f bytes) 000073 1741626900.424509 MIRROR_EXTEND to 5 mirrors 000074 1741626900.533036 mapread 0x006137 thru 0x010519 (0xa3e3 bytes) 000075 1741626900.542041 mapread 0x02ffdc thru 0x03ac25 (0xac4a bytes) 000076 1741626900.549271 MIRROR_RESYNC 5 mirrors 000078 1741626900.683832 MIRROR_RESYNC 5 mirrors 000079 1741626900.730631 mapread 0x01bd74 thru 0x025960 (0x9bed bytes) 000080 1741626900.737091 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000081 1741626900.798570 trunc from 0x03ac26 to 0x023e9e (0x16d87 bytes) 000083 1741626900.821024 trunc from 0x023e9f to 0x01a140 (0x9d5e bytes) 000085 1741626900.826666 mapread 0x01204a thru 0x01a140 (0x80f7 bytes) 000086 1741626900.831161 mapwrite 0x00ffaf thru 0x01af24 (0xaf76 bytes) 000088 1741626900.853596 mapread 0x018d21 thru 0x01af24 (0x2204 bytes) 000089 1741626900.854207 trunc from 0x01af25 to 0x0378bb (0x1c997 bytes) 000090 1741626900.857956 mapread 0x0127a9 thru 0x021e5e (0xf6b6 bytes) 000091 1741626900.860932 read 0x0250f3 thru 0x02a4ee (0x53fc bytes) 000092 1741626900.862496 write 0x0303d7 thru 0x03629b (0x5ec5 bytes) 000093 1741626900.864956 mapread 0x01c2df thru 0x01c831 (0x553 bytes) 000094 1741626900.866908 MIRROR_RESYNC 4 mirrors 000095 1741626901.001419 write 0x024259 thru 0x02ebaf (0xa957 bytes) 000096 1741626901.017111 trunc from 0x0378bc to 0x01cb68 (0x1ad53 bytes) 000097 1741626901.025695 write 0x001010 thru 0x007a61 (0x6a52 bytes) 000101 1741626901.029598 mapwrite 0x021578 thru 0x028de3 (0x786c bytes) 000103 1741626901.051019 read 0x007f2a thru 0x00ae6c (0x2f43 bytes) 000106 1741626901.052248 trunc from 0x028de4 to 0x02fba2 (0x6dbf bytes) 000108 1741626901.057126 MIRROR_RESYNC 4 mirrors 000109 1741626901.189498 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000110 1741626901.245602 trunc from 0x02fba3 to 0x01bcd2 (0x13ed0 bytes) truncating to largest ever: 0x3fd6d 000112 1741626901.260880 trunc from 0x01bcd3 to 0x03fd6c (0x2409a bytes) 000113 1741626901.266509 mapwrite 0x01905d thru 0x020965 (0x7909 bytes) 000114 1741626901.287835 write 0x00c74b thru 0x014277 (0x7b2d bytes) 000115 1741626901.292218 MIRROR_RESYNC 3 mirrors 000117 1741626901.409841 mapwrite 0x002ca7 thru 0x006bc5 (0x3f1f bytes) 000118 1741626901.453949 mapread 0x03ba14 thru 0x03fd6c (0x4359 bytes) 000119 1741626901.456968 trunc from 0x03fd6d to 0x035774 (0xa5f8 bytes) 000121 1741626901.465625 write 0x03c623 thru 0x03f2d6 (0x2cb4 bytes) 000122 1741626901.467329 MIRROR_RESYNC 3 mirrors 000123 1741626901.578406 read 0x03f192 thru 0x03f2d6 (0x145 bytes) 000127 1741626901.579354 write 0x034c9a thru 0x03ffff (0xb366 bytes) 000128 1741626901.594257 read 0x0196b7 thru 0x0260cb (0xca15 bytes) 000131 1741626901.597835 write 0x032e80 thru 0x03ffff (0xd180 bytes) 000132 1741626901.600035 MIRROR_EXTEND to 4 mirrors 000133 1741626901.701629 read 0x02a36a thru 0x032199 (0x7e30 bytes) 000134 1741626901.704605 mapwrite 0x0275c0 thru 0x035fd9 (0xea1a bytes) 000138 1741626901.755812 read 0x00f804 thru 0x013d3f (0x453c bytes) 000141 1741626901.759439 read 0x03e231 thru 0x03ffff (0x1dcf bytes) 000144 1741626901.760752 trunc from 0x040000 to 0x03c599 (0x3a66 bytes) 000146 1741626901.770109 MIRROR_RESYNC 4 mirrors 000147 1741626901.898214 mapwrite 0x00c914 thru 0x010d20 (0x440d bytes) 000148 1741626901.940636 mapread 0x0307cd thru 0x03bd00 (0xb534 bytes) 000150 1741626901.945441 write 0x0037c7 thru 0x00bb5d (0x8397 bytes) 000151 1741626901.947912 mapread 0x015c38 thru 0x01eb2c (0x8ef5 bytes) 000152 1741626901.951865 write 0x01c082 thru 0x028016 (0xbf95 bytes) 000153 1741626901.953609 mapread 0x006242 thru 0x009f8c (0x3d4b bytes) 000156 1741626901.954920 trunc from 0x03c59a to 0x035e79 (0x6720 bytes) 000157 1741626901.963497 trunc from 0x035e7a to 0x0385b5 (0x273c bytes) 000159 1741626901.967115 mapwrite 0x01337a thru 0x0187bb (0x5442 bytes) 000160 1741626901.981172 mapread 0x0076a2 thru 0x013f80 (0xc8df bytes) 000161 1741626901.983194 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000163 1741626902.030552 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000164 1741626902.072835 trunc from 0x0385b6 to 0x023f6b (0x1464a bytes) 000165 1741626902.077274 MIRROR_RESYNC 2 mirrors 000166 1741626902.169210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000167 1741626902.196091 write 0x03f4a5 thru 0x03ffff (0xb5b bytes) 000172 1741626902.201293 write 0x027767 thru 0x02d743 (0x5fdd bytes) 000175 1741626902.205551 MIRROR_EXTEND to 2 mirrors 000177 1741626902.328908 write 0x016949 thru 0x024656 (0xdd0e bytes) 000178 1741626902.345055 read 0x000f72 thru 0x010b16 (0xfba5 bytes) 000180 1741626902.347645 write 0x005920 thru 0x0077f2 (0x1ed3 bytes) 000181 1741626902.348393 mapwrite 0x0326a8 thru 0x035ec8 (0x3821 bytes) 000183 1741626902.363894 read 0x03bd9c thru 0x03ffff (0x4264 bytes) 000184 1741626902.367615 MIRROR_RESYNC 2 mirrors 000185 1741626902.455409 write 0x02ce06 thru 0x02dc36 (0xe31 bytes) 000186 1741626902.469795 trunc from 0x040000 to 0x036b2a (0x94d5 bytes) 000188 1741626902.477600 mapread 0x0102b9 thru 0x01b23a (0xaf82 bytes) 000189 1741626902.481832 read 0x01fc14 thru 0x029578 (0x9965 bytes) 000191 1741626902.485521 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000193 1741626902.519033 write 0x012470 thru 0x013122 (0xcb3 bytes) 000195 1741626902.523616 read 0x017b86 thru 0x027962 (0xfddd bytes) 000196 1741626902.526658 read 0x033ff7 thru 0x036b2a (0x2b34 bytes) 000197 1741626902.530490 read 0x00981f thru 0x0100c1 (0x68a3 bytes) 000198 1741626902.530833 read 0x01b5c1 thru 0x022747 (0x7187 bytes) 000199 1741626902.531228 write 0x0314fb thru 0x03ffff (0xeb05 bytes) 000202 1741626902.532812 write 0x018d73 thru 0x01b7a9 (0x2a37 bytes) 000204 1741626902.533488 mapread 0x026fbe thru 0x02a0d0 (0x3113 bytes) 000205 1741626902.533989 write 0x036497 thru 0x03d0ce (0x6c38 bytes) 000206 1741626902.534740 mapread 0x01fc17 thru 0x020ea3 (0x128d bytes) 000207 1741626902.535446 trunc from 0x040000 to 0x003a93 (0x3c56c bytes) 000208 1741626902.539522 read 0x0017c8 thru 0x003a93 (0x22cc bytes) 000212 1741626902.541297 mapread 0x001477 thru 0x003a93 (0x261d bytes) 000215 1741626902.542598 read 0x003248 thru 0x003a93 (0x84c bytes) 000216 1741626902.542949 read 0x00063f thru 0x0010c2 (0xa84 bytes) 000218 1741626902.543682 mapread 0x001f2c thru 0x0025fa (0x6cf bytes) 000223 1741626902.545946 mapread 0x000db5 thru 0x003a93 (0x2cdf bytes) 000226 1741626902.547073 trunc from 0x003a94 to 0x03e1d2 (0x3a73f bytes) 000228 1741626902.552268 MIRROR_EXTEND to 2 mirrors 000229 1741626902.645972 mapread 0x007f3f thru 0x0156c8 (0xd78a bytes) 000230 1741626902.653057 read 0x01d859 thru 0x0210b8 (0x3860 bytes) 000231 1741626902.655824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000232 1741626902.683963 read 0x016fa8 thru 0x019657 (0x26b0 bytes) 000235 1741626902.686783 read 0x001f4b thru 0x008f7a (0x7030 bytes) 000237 1741626902.691282 trunc from 0x03e1d3 to 0x00706f (0x37163 bytes) 000239 1741626902.702018 MIRROR_EXTEND to 2 mirrors 000241 1741626902.792417 mapread 0x004e58 thru 0x00706f (0x2218 bytes) 000242 1741626902.796395 mapwrite 0x03eca6 thru 0x03ffff (0x135a bytes) 000244 1741626902.830565 write 0x0122a3 thru 0x015bec (0x394a bytes) 000246 1741626902.833526 mapwrite 0x013e01 thru 0x014533 (0x733 bytes) 000247 1741626902.845313 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000251 1741626902.876650 mapread 0x0389da thru 0x03b12b (0x2752 bytes) 000254 1741626902.882203 mapwrite 0x0096d3 thru 0x00e49d (0x4dcb bytes) 000255 1741626902.896626 write 0x001333 thru 0x009a30 (0x86fe bytes) 000256 1741626902.898532 mapwrite 0x03633d thru 0x03ffff (0x9cc3 bytes) 000257 1741626902.912901 MIRROR_EXTEND to 2 mirrors 000258 1741626903.004613 trunc from 0x040000 to 0x029050 (0x16faf bytes) 000261 1741626903.023169 mapread 0x023b67 thru 0x029050 (0x54ea bytes) 000263 1741626903.028601 mapwrite 0x02ae92 thru 0x036260 (0xb3cf bytes) 000264 1741626903.050228 read 0x004c6c thru 0x00bfee (0x7383 bytes) 000265 1741626903.053980 MIRROR_RESYNC 2 mirrors 000266 1741626903.145491 read 0x02863a thru 0x036038 (0xd9ff bytes) 000267 1741626903.148634 mapwrite 0x03ac95 thru 0x03ffff (0x536b bytes) 000268 1741626903.182823 read 0x02d189 thru 0x02eddc (0x1c54 bytes) 000269 1741626903.186369 write 0x010573 thru 0x010671 (0xff bytes) 000272 1741626903.187652 read 0x03fba0 thru 0x03ffff (0x460 bytes) 000274 1741626903.188478 mapwrite 0x0126a2 thru 0x019e71 (0x77d0 bytes) 000276 1741626903.201511 trunc from 0x040000 to 0x03a7d2 (0x582d bytes) 000281 1741626903.207712 trunc from 0x03a7d3 to 0x034816 (0x5fbc bytes) 000282 1741626903.210995 mapwrite 0x037954 thru 0x03b3a3 (0x3a50 bytes) 000285 1741626903.230066 trunc from 0x03b3a4 to 0x00db3f (0x2d864 bytes) 000286 1741626903.233738 mapread 0x0007c9 thru 0x005c98 (0x54d0 bytes) 000287 1741626903.235309 MIRROR_EXTEND to 3 mirrors 000289 1741626903.321122 write 0x00c879 thru 0x011d2b (0x54b3 bytes) 000292 1741626903.336874 write 0x00be72 thru 0x01050f (0x469e bytes) 000293 1741626903.338381 mapwrite 0x01641d thru 0x01e003 (0x7be7 bytes) 000294 1741626903.364505 mapread 0x00ae91 thru 0x017793 (0xc903 bytes) 000295 1741626903.367021 trunc from 0x01e004 to 0x00157e (0x1ca85 bytes) 000296 1741626903.370931 mapwrite 0x015241 thru 0x015f81 (0xd41 bytes) 000298 1741626903.389723 MIRROR_RESYNC 3 mirrors 000300 1741626903.494248 write 0x001efd thru 0x002029 (0x12d bytes) 000302 1741626903.508908 read 0x008d31 thru 0x00fe16 (0x70e6 bytes) 000304 1741626903.513007 MIRROR_EXTEND to 4 mirrors 000308 1741626903.606175 read 0x00e693 thru 0x013f76 (0x58e4 bytes) 000309 1741626903.609973 read 0x012b29 thru 0x015f81 (0x3459 bytes) 000310 1741626903.612696 mapwrite 0x038ac9 thru 0x03ffff (0x7537 bytes) 000311 1741626903.647678 read 0x034dd2 thru 0x03ffff (0xb22e bytes) 000313 1741626903.648757 mapread 0x0070e1 thru 0x016228 (0xf148 bytes) 000314 1741626903.652042 mapwrite 0x0046b5 thru 0x0144a4 (0xfdf0 bytes) 000315 1741626903.668328 mapwrite 0x005688 thru 0x0116ec (0xc065 bytes) 000316 1741626903.682994 mapread 0x00bc50 thru 0x019e67 (0xe218 bytes) 000318 1741626903.683881 mapwrite 0x0239c0 thru 0x02f3b8 (0xb9f9 bytes) 000319 1741626903.698864 MIRROR_EXTEND to 5 mirrors 000320 1741626903.793616 mapread 0x039fd1 thru 0x03ffff (0x602f bytes) 000322 1741626903.800852 mapwrite 0x021287 thru 0x02d3ef (0xc169 bytes) 000323 1741626903.848106 write 0x030b40 thru 0x03bddc (0xb29d bytes) 000325 1741626903.850615 trunc from 0x040000 to 0x0171d0 (0x28e2f bytes) 000326 1741626903.859930 read 0x015244 thru 0x0171d0 (0x1f8d bytes) 000327 1741626903.863025 MIRROR_EXTEND to 6 mirrors 000329 1741626903.952930 read 0x010b48 thru 0x0171d0 (0x6689 bytes) 000335 1741626903.960879 mapwrite 0x02652f thru 0x036060 (0xfb32 bytes) 000337 1741626904.003040 MIRROR_RESYNC 6 mirrors 000339 1741626904.189818 write 0x00db97 thru 0x0154d1 (0x793b bytes) 000340 1741626904.205616 mapread 0x03508d thru 0x036060 (0xfd4 bytes) 000341 1741626904.208081 read 0x007ce7 thru 0x00d082 (0x539c bytes) 000342 1741626904.210844 trunc from 0x036061 to 0x03e338 (0x82d8 bytes) 000343 1741626904.215410 mapwrite 0x02bc3d thru 0x03707e (0xb442 bytes) 000344 1741626904.233626 MIRROR_EXTEND to 7 mirrors 000345 1741626904.326440 read 0x038e58 thru 0x03e338 (0x54e1 bytes) 000346 1741626904.329458 MIRROR_RESYNC 7 mirrors 000349 1741626904.498055 trunc from 0x03e339 to 0x002458 (0x3bee0 bytes) 000351 1741626904.516847 mapwrite 0x00bf2e thru 0x01acca (0xed9d bytes) 000353 1741626904.542518 read 0x005ba6 thru 0x010c3f (0xb09a bytes) 000354 1741626904.544189 mapwrite 0x022286 thru 0x02662e (0x43a9 bytes) 000356 1741626904.562300 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000357 1741626904.604989 trunc from 0x02662f to 0x02bc9f (0x5671 bytes) 000359 1741626904.608596 trunc from 0x02bca0 to 0x02a68b (0x1614 bytes) 000361 1741626904.612525 read 0x01137f thru 0x02046f (0xf0f1 bytes) 000362 1741626904.614936 read 0x002b23 thru 0x010572 (0xda50 bytes) 000365 1741626904.618080 trunc from 0x02a68c to 0x026614 (0x4077 bytes) 000366 1741626904.621796 trunc from 0x026615 to 0x01467b (0x11f99 bytes) 000367 1741626904.625501 write 0x01c980 thru 0x02032c (0x39ad bytes) 000368 1741626904.626250 mapwrite 0x027732 thru 0x02939e (0x1c6d bytes) 000370 1741626904.645724 mapwrite 0x01822b thru 0x024714 (0xc4ea bytes) 000371 1741626904.667660 mapread 0x00e8e4 thru 0x01c92a (0xe047 bytes) 000373 1741626904.671430 trunc from 0x02939f to 0x009952 (0x1fa4c bytes) 000375 1741626904.676855 MIRROR_RESYNC 6 mirrors 000376 1741626904.832610 write 0x02929e thru 0x02e081 (0x4de4 bytes) 000377 1741626904.848679 read 0x02bdcb thru 0x02e081 (0x22b7 bytes) 000378 1741626904.849210 trunc from 0x02e082 to 0x039ba7 (0xbb26 bytes) 000381 1741626904.857804 write 0x0090fb thru 0x00a291 (0x1197 bytes) 000382 1741626904.860600 trunc from 0x039ba8 to 0x0304a6 (0x9701 bytes) 000383 1741626904.864888 trunc from 0x0304a7 to 0x03f8c7 (0xf421 bytes) 000384 1741626904.868495 mapwrite 0x002f8d thru 0x007a75 (0x4ae9 bytes) 000385 1741626904.886918 mapwrite 0x00f020 thru 0x016c5d (0x7c3e bytes) 000387 1741626904.901197 trunc from 0x03f8c8 to 0x03bc90 (0x3c37 bytes) 000389 1741626904.905043 mapwrite 0x030387 thru 0x036bc4 (0x683e bytes) 000391 1741626904.920464 MIRROR_EXTEND to 7 mirrors 000392 1741626905.014464 read 0x004ea7 thru 0x014bd9 (0xfd33 bytes) 000394 1741626905.017181 write 0x0131a7 thru 0x01e999 (0xb7f3 bytes) 000395 1741626905.034011 trunc from 0x03bc91 to 0x006eb2 (0x34dde bytes) 000397 1741626905.041957 mapread 0x00587e thru 0x006eb2 (0x1635 bytes) 000398 1741626905.044114 read 0x003cf7 thru 0x006eb2 (0x31bc bytes) 000400 1741626905.044937 mapwrite 0x00a610 thru 0x00d01d (0x2a0e bytes) 000401 1741626905.064963 MIRROR_RESYNC 7 mirrors 000402 1741626905.235512 read 0x0011e2 thru 0x00662d (0x544c bytes) 000405 1741626905.238172 trunc from 0x00d01e to 0x03a7a5 (0x2d788 bytes) 000410 1741626905.258864 read 0x037cec thru 0x03a7a5 (0x2aba bytes) 000413 1741626905.264012 mapwrite 0x01830e thru 0x01d863 (0x5556 bytes) 000414 1741626905.279109 write 0x03435e thru 0x03e967 (0xa60a bytes) 000417 1741626905.280454 read 0x0256ea thru 0x027e3f (0x2756 bytes) 000418 1741626905.280813 trunc from 0x03e968 to 0x02cf32 (0x11a35 bytes) 000419 1741626905.285628 write 0x01e9ce thru 0x02b1d7 (0xc80a bytes) 000423 1741626905.287569 trunc from 0x02cf33 to 0x02f453 (0x2521 bytes) 000424 1741626905.291677 mapread 0x0096e3 thru 0x016983 (0xd2a1 bytes) 000425 1741626905.292296 mapwrite 0x02faf9 thru 0x031bae (0x20b6 bytes) 000426 1741626905.311732 mapread 0x0196e0 thru 0x01b336 (0x1c57 bytes) 000429 1741626905.312708 mapwrite 0x02c920 thru 0x031b3e (0x521f bytes) 000431 1741626905.328055 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000434 1741626905.376288 mapread 0x013db7 thru 0x01fe4e (0xc098 bytes) 000437 1741626905.384757 write 0x01fd62 thru 0x02afbd (0xb25c bytes) 000438 1741626905.386865 write 0x0032e8 thru 0x00e768 (0xb481 bytes) 000439 1741626905.389750 trunc from 0x031baf to 0x039e40 (0x8292 bytes) 000440 1741626905.393828 mapread 0x0048aa thru 0x00c609 (0x7d60 bytes) 000441 1741626905.394427 trunc from 0x039e41 to 0x03a700 (0x8c0 bytes) 000442 1741626905.398007 read 0x0028ca thru 0x00304e (0x785 bytes) 000444 1741626905.398746 mapwrite 0x03e956 thru 0x03f4b9 (0xb64 bytes) 000446 1741626905.419817 mapread 0x00359a thru 0x009a75 (0x64dc bytes) 000447 1741626905.420459 read 0x00f1fc thru 0x01c1e1 (0xcfe6 bytes) 000449 1741626905.421468 trunc from 0x03f4ba to 0x01e794 (0x20d25 bytes) 000450 1741626905.425097 trunc from 0x01e795 to 0x0318fa (0x13166 bytes) 000452 1741626905.429544 MIRROR_EXTEND to 7 mirrors 000453 1741626905.535107 trunc from 0x0318fb to 0x03055d (0x139d bytes) 000455 1741626905.556717 read 0x02325f thru 0x02bb51 (0x88f3 bytes) 000456 1741626905.559660 write 0x034f3f thru 0x03ffff (0xb0c1 bytes) 000457 1741626905.562076 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000460 1741626905.615223 write 0x004d30 thru 0x00eae1 (0x9db2 bytes) 000461 1741626905.618275 mapwrite 0x03d60c thru 0x03ffff (0x29f4 bytes) 000462 1741626905.635451 trunc from 0x040000 to 0x015d7a (0x2a285 bytes) 000464 1741626905.640915 MIRROR_RESYNC 6 mirrors 000465 1741626905.816935 mapread 0x00d5ec thru 0x00ec9d (0x16b2 bytes) 000466 1741626905.819716 mapwrite 0x01d85f thru 0x0283f1 (0xab93 bytes) 000467 1741626905.857204 trunc from 0x0283f2 to 0x03577a (0xd389 bytes) 000470 1741626905.862381 read 0x026c09 thru 0x0327b0 (0xbba8 bytes) 000474 1741626905.866556 write 0x00956d thru 0x017ebf (0xe953 bytes) 000475 1741626905.869437 read 0x013a98 thru 0x023811 (0xfd7a bytes) 000476 1741626905.869841 write 0x032e6d thru 0x03c8fb (0x9a8f bytes) 000477 1741626905.872113 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000478 1741626905.923311 trunc from 0x03c8fc to 0x024895 (0x18066 bytes) 000479 1741626905.928978 MIRROR_RESYNC 5 mirrors 000480 1741626906.084976 trunc from 0x024896 to 0x00bd71 (0x18b24 bytes) 000482 1741626906.101783 mapread 0x000218 thru 0x008e91 (0x8c7a bytes) 000485 1741626906.109163 mapwrite 0x03a971 thru 0x03ffff (0x568f bytes) 000486 1741626906.129915 mapwrite 0x028365 thru 0x030ecc (0x8b68 bytes) 000488 1741626906.145028 mapwrite 0x0108b7 thru 0x0113a2 (0xaec bytes) 000491 1741626906.159595 mapwrite 0x00ff46 thru 0x01e804 (0xe8bf bytes) 000492 1741626906.176659 trunc from 0x040000 to 0x00d9ae (0x32651 bytes) 000493 1741626906.181186 trunc from 0x00d9af to 0x01a342 (0xc994 bytes) 000494 1741626906.184571 mapwrite 0x02c6f6 thru 0x03c5b3 (0xfebe bytes) 000495 1741626906.208005 mapwrite 0x008525 thru 0x00880c (0x2e8 bytes) 000496 1741626906.217332 read 0x01dc94 thru 0x0225e5 (0x4952 bytes) 000498 1741626906.221105 MIRROR_EXTEND to 6 mirrors 000499 1741626906.317537 read 0x0068dc thru 0x0098cd (0x2ff2 bytes) 000500 1741626906.319900 trunc from 0x03c5b4 to 0x0138a5 (0x28d0e bytes) 000501 1741626906.340897 trunc from 0x0138a6 to 0x00457d (0xf328 bytes) 000507 1741626906.347792 mapwrite 0x035eb9 thru 0x036a84 (0xbcc bytes) 000510 1741626906.367124 mapread 0x00125b thru 0x002220 (0xfc6 bytes) 000512 1741626906.370161 mapwrite 0x00e802 thru 0x01db91 (0xf390 bytes) 000513 1741626906.388239 write 0x02923b thru 0x02b3cc (0x2192 bytes) 000514 1741626906.391888 MIRROR_EXTEND to 7 mirrors 000517 1741626906.489576 mapread 0x00448f thru 0x00e019 (0x9b8b bytes) 000518 1741626906.497597 MIRROR_EXTEND to 8 mirrors 000519 1741626906.595195 trunc from 0x036a85 to 0x013c7e (0x22e06 bytes) 000524 1741626906.616249 trunc from 0x013c7f to 0x025b6c (0x11eee bytes) 000526 1741626906.622066 mapwrite 0x000989 thru 0x008e3f (0x84b7 bytes) 000527 1741626906.643664 read 0x00a661 thru 0x00ade7 (0x787 bytes) 000528 1741626906.644124 read 0x009c3d thru 0x00c218 (0x25dc bytes) 000529 1741626906.644446 mapwrite 0x02ab0e thru 0x02cf28 (0x241b bytes) 000532 1741626906.664463 mapwrite 0x00799c thru 0x010dcc (0x9431 bytes) 000533 1741626906.678100 write 0x03c8ab thru 0x03ffff (0x3755 bytes) 000535 1741626906.679063 trunc from 0x040000 to 0x02e7d1 (0x1182e bytes) 000537 1741626906.685010 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000542 1741626906.737660 mapread 0x00b8f4 thru 0x0105b4 (0x4cc1 bytes) 000543 1741626906.742081 read 0x01a1b4 thru 0x02155d (0x73aa bytes) 000544 1741626906.744282 mapwrite 0x032444 thru 0x03334c (0xf09 bytes) 000545 1741626906.762671 trunc from 0x03334d to 0x007bce (0x2b77e bytes) 000547 1741626906.767885 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000548 1741626906.817769 mapwrite 0x01b817 thru 0x027b73 (0xc35d bytes) 000550 1741626906.842234 mapread 0x015dfa thru 0x017278 (0x147f bytes) 000551 1741626906.846206 MIRROR_RESYNC 6 mirrors 000552 1741626907.029971 mapwrite 0x01d238 thru 0x027aa7 (0xa870 bytes) 000553 1741626907.081080 read 0x012202 thru 0x021600 (0xf3ff bytes) 000554 1741626907.083573 read 0x0241b7 thru 0x027b73 (0x39bd bytes) 000556 1741626907.085986 MIRROR_RESYNC 6 mirrors 000560 1741626907.249481 write 0x03e28c thru 0x03ffff (0x1d74 bytes) 000561 1741626907.264389 trunc from 0x040000 to 0x017963 (0x2869c bytes) 000562 1741626907.272574 mapread 0x00b152 thru 0x011ab2 (0x6961 bytes) 000563 1741626907.278408 read 0x0058bc thru 0x014d6e (0xf4b3 bytes) 000566 1741626907.281001 trunc from 0x017964 to 0x009e33 (0xdb30 bytes) 000567 1741626907.284720 trunc from 0x009e34 to 0x016c27 (0xcdf4 bytes) 000569 1741626907.288897 write 0x00702a thru 0x0112c6 (0xa29d bytes) 000571 1741626907.292796 MIRROR_EXTEND to 7 mirrors 000572 1741626907.403979 write 0x02ae8d thru 0x033eef (0x9063 bytes) 000573 1741626907.419883 write 0x00e81d thru 0x018bab (0xa38f bytes) 000575 1741626907.422866 mapwrite 0x02b846 thru 0x03959d (0xdd58 bytes) 000576 1741626907.450580 mapwrite 0x01c073 thru 0x01f4cc (0x345a bytes) 000577 1741626907.466045 MIRROR_EXTEND to 8 mirrors 000579 1741626907.563533 read 0x02eb79 thru 0x02ee38 (0x2c0 bytes) 000580 1741626907.565810 write 0x039927 thru 0x03ffff (0x66d9 bytes) 000582 1741626907.581726 read 0x001e39 thru 0x0072ca (0x5492 bytes) 000583 1741626907.585199 write 0x01f528 thru 0x020221 (0xcfa bytes) 000584 1741626907.587516 MIRROR_EXTEND to 9 mirrors 000585 1741626907.700186 mapread 0x01d11a thru 0x02b05a (0xdf41 bytes) 000587 1741626907.710855 MIRROR_EXTEND to 10 mirrors 000588 1741626907.821086 write 0x0345d0 thru 0x03addc (0x680d bytes) 000590 1741626907.841403 write 0x01ae62 thru 0x026cbc (0xbe5b bytes) 000593 1741626907.847521 MIRROR_RESYNC 10 mirrors 000595 1741626908.133595 read 0x005118 thru 0x0109d5 (0xb8be bytes) 000596 1741626908.136102 mapread 0x030437 thru 0x038aae (0x8678 bytes) 000598 1741626908.141122 read 0x018208 thru 0x0190dd (0xed6 bytes) 000599 1741626908.144626 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000606 1741626908.223167 write 0x024daf thru 0x028a7a (0x3ccc bytes) 000608 1741626908.239840 read 0x014521 thru 0x01e88d (0xa36d bytes) 000610 1741626908.242742 mapwrite 0x022c39 thru 0x03179f (0xeb67 bytes) 000613 1741626908.263438 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000615 1741626908.315768 read 0x024625 thru 0x032746 (0xe122 bytes) 000616 1741626908.318632 mapwrite 0x00cf3e thru 0x01717d (0xa240 bytes) 000617 1741626908.341332 write 0x03fff4 thru 0x03ffff (0xc bytes) 000618 1741626908.343923 read 0x00a476 thru 0x015118 (0xaca3 bytes) 000619 1741626908.344364 mapread 0x039f4f thru 0x03ffff (0x60b1 bytes) 000620 1741626908.345064 read 0x00421c thru 0x00f471 (0xb256 bytes) 000621 1741626908.347752 trunc from 0x040000 to 0x02b98f (0x14670 bytes) 000622 1741626908.353261 MIRROR_EXTEND to 9 mirrors 000624 1741626908.457045 mapwrite 0x010ee5 thru 0x01a968 (0x9a84 bytes) 000627 1741626908.506857 mapread 0x0041dd thru 0x00ae00 (0x6c24 bytes) 000628 1741626908.510466 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000629 1741626908.559558 trunc from 0x02b990 to 0x0366cd (0xad3e bytes) 000630 1741626908.567821 trunc from 0x0366ce to 0x00fc17 (0x26ab6 bytes) 000631 1741626908.571578 mapread 0x008729 thru 0x00fc17 (0x74ef bytes) 000632 1741626908.576508 write 0x03c7ee thru 0x03ffff (0x3812 bytes) 000634 1741626908.577725 mapread 0x01a22b thru 0x01da91 (0x3867 bytes) 000635 1741626908.580261 mapwrite 0x0149e3 thru 0x021ea5 (0xd4c3 bytes) 000637 1741626908.597527 mapwrite 0x02014b thru 0x02c942 (0xc7f8 bytes) 000639 1741626908.614278 mapread 0x03d84e thru 0x03ffff (0x27b2 bytes) 000640 1741626908.614754 mapwrite 0x00496e thru 0x006e46 (0x24d9 bytes) 000641 1741626908.626380 mapread 0x03ee93 thru 0x03ffff (0x116d bytes) 000642 1741626908.626859 read 0x0292fe thru 0x02dfbc (0x4cbf bytes) 000644 1741626908.627668 mapread 0x01160d thru 0x0194a7 (0x7e9b bytes) 000645 1741626908.628352 read 0x00048d thru 0x00dade (0xd652 bytes) 000646 1741626908.631248 write 0x0258ee thru 0x029e6f (0x4582 bytes) 000647 1741626908.631951 read 0x013666 thru 0x018589 (0x4f24 bytes) 000648 1741626908.632420 mapwrite 0x001478 thru 0x003da2 (0x292b bytes) 000649 1741626908.643055 write 0x03d17d thru 0x03ffff (0x2e83 bytes) 000650 1741626908.643806 write 0x02bb02 thru 0x03a233 (0xe732 bytes) 000652 1741626908.644812 mapread 0x02908d thru 0x02f756 (0x66ca bytes) 000656 1741626908.646045 mapread 0x0166b9 thru 0x01b01e (0x4966 bytes) 000657 1741626908.646464 mapwrite 0x010dc9 thru 0x014244 (0x347c bytes) 000660 1741626908.659254 mapread 0x004663 thru 0x006a85 (0x2423 bytes) 000661 1741626908.661014 MIRROR_EXTEND to 9 mirrors 000662 1741626908.756766 read 0x004182 thru 0x00c8e6 (0x8765 bytes) 000663 1741626908.759571 write 0x02f019 thru 0x03e2b6 (0xf29e bytes) 000666 1741626908.776482 mapread 0x038941 thru 0x03ffff (0x76bf bytes) 000668 1741626908.778175 mapread 0x007a5c thru 0x00d8ea (0x5e8f bytes) 000669 1741626908.781291 read 0x03348d thru 0x03ffff (0xcb73 bytes) 000670 1741626908.781834 trunc from 0x040000 to 0x02e473 (0x11b8c bytes) 000672 1741626908.790126 read 0x025347 thru 0x02a856 (0x5510 bytes) 000676 1741626908.793495 write 0x003439 thru 0x00b97a (0x8542 bytes) 000682 1741626908.796644 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000684 1741626908.847132 mapwrite 0x001828 thru 0x007847 (0x6020 bytes) 000685 1741626908.863586 read 0x0007c7 thru 0x006636 (0x5e70 bytes) 000687 1741626908.865478 read 0x019a19 thru 0x025a8d (0xc075 bytes) 000688 1741626908.868659 trunc from 0x02e474 to 0x025e02 (0x8671 bytes) 000690 1741626908.872824 mapread 0x00bb92 thru 0x011019 (0x5488 bytes) 000691 1741626908.873398 trunc from 0x025e03 to 0x0114ef (0x14913 bytes) 000695 1741626908.878818 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000697 1741626908.924986 trunc from 0x0114f0 to 0x027c96 (0x167a7 bytes) 000698 1741626908.928563 trunc from 0x027c97 to 0x022ad5 (0x51c1 bytes) 000699 1741626908.933651 read 0x0079ba thru 0x0102ab (0x88f2 bytes) 000700 1741626908.936715 write 0x0199f4 thru 0x01f765 (0x5d72 bytes) 000701 1741626908.938976 read 0x01c3ca thru 0x022ad5 (0x670c bytes) 000703 1741626908.942531 MIRROR_EXTEND to 8 mirrors 000704 1741626909.056692 mapwrite 0x025174 thru 0x027715 (0x25a2 bytes) 000706 1741626909.094117 MIRROR_EXTEND to 9 mirrors 000708 1741626909.185281 mapwrite 0x0144ee thru 0x01bd76 (0x7889 bytes) 000709 1741626909.228851 mapread 0x024e99 thru 0x027715 (0x287d bytes) 000711 1741626909.231875 write 0x011763 thru 0x013c96 (0x2534 bytes) 000712 1741626909.232503 trunc from 0x027716 to 0x028495 (0xd80 bytes) 000713 1741626909.239821 mapread 0x0253b6 thru 0x028495 (0x30e0 bytes) 000715 1741626909.241959 trunc from 0x028496 to 0x03c4f8 (0x14063 bytes) 000718 1741626909.245645 trunc from 0x03c4f9 to 0x03a1e5 (0x2313 bytes) 000719 1741626909.248774 write 0x038ea6 thru 0x03aba6 (0x1d01 bytes) 000722 1741626909.252103 read 0x01f896 thru 0x025426 (0x5b91 bytes) 000723 1741626909.253711 mapread 0x02711c thru 0x028ade (0x19c3 bytes) 000725 1741626909.254454 read 0x03a99f thru 0x03aba6 (0x208 bytes) 000727 1741626909.254994 trunc from 0x03aba7 to 0x03d609 (0x2a63 bytes) 000728 1741626909.258743 trunc from 0x03d60a to 0x021839 (0x1bdd0 bytes) 000729 1741626909.262540 mapread 0x020225 thru 0x021839 (0x1615 bytes) 000731 1741626909.264506 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000734 1741626909.313054 write 0x035400 thru 0x03cd62 (0x7963 bytes) 000735 1741626909.313876 write 0x00a639 thru 0x016630 (0xbff8 bytes) 000739 1741626909.317640 mapread 0x01f36f thru 0x0212e4 (0x1f76 bytes) 000740 1741626909.320126 mapread 0x002177 thru 0x0029c3 (0x84d bytes) 000741 1741626909.322306 mapread 0x035092 thru 0x03cd62 (0x7cd1 bytes) 000742 1741626909.322784 mapwrite 0x0231d8 thru 0x02fd85 (0xcbae bytes) 000744 1741626909.340996 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000745 1741626909.385353 MIRROR_RESYNC 7 mirrors 000746 1741626909.578384 trunc from 0x03cd63 to 0x014dd5 (0x27f8d bytes) 000753 1741626909.599887 MIRROR_RESYNC 7 mirrors 000754 1741626909.762204 mapread 0x0025c4 thru 0x00ffb2 (0xd9ef bytes) 000756 1741626909.768040 write 0x0113f8 thru 0x013b8f (0x2798 bytes) 000757 1741626909.781816 mapread 0x006357 thru 0x01497b (0xe625 bytes) 000758 1741626909.784261 mapread 0x000de6 thru 0x00e399 (0xd5b4 bytes) 000759 1741626909.787362 trunc from 0x014dd6 to 0x031d22 (0x1cf4d bytes) 000760 1741626909.795624 mapwrite 0x013b2a thru 0x013e6b (0x342 bytes) 000761 1741626909.808392 write 0x022807 thru 0x023680 (0xe7a bytes) 000762 1741626909.811310 trunc from 0x031d23 to 0x0058d3 (0x2c44f bytes) 000764 1741626909.815744 mapwrite 0x01151c thru 0x013c41 (0x2726 bytes) 000766 1741626909.832674 write 0x00002b thru 0x00d328 (0xd2fe bytes) 000768 1741626909.833763 trunc from 0x013c42 to 0x017cec (0x40ab bytes) 000769 1741626909.837412 read 0x00110e thru 0x008679 (0x756c bytes) 000772 1741626909.839176 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000773 1741626909.885637 read 0x001ac4 thru 0x003ef1 (0x242e bytes) 000774 1741626909.887219 mapwrite 0x026cec thru 0x02b88c (0x4ba1 bytes) 000775 1741626909.909023 mapread 0x01252f thru 0x014a90 (0x2562 bytes) 000781 1741626909.913888 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000782 1741626909.956443 mapwrite 0x00106d thru 0x0063cc (0x5360 bytes) 000784 1741626909.973312 trunc from 0x02b88d to 0x038ae6 (0xd25a bytes) 000785 1741626909.977038 mapwrite 0x035860 thru 0x038809 (0x2faa bytes) 000786 1741626909.991484 mapwrite 0x03e790 thru 0x03ffff (0x1870 bytes) 000787 1741626910.008006 mapread 0x02a8f1 thru 0x02e3c6 (0x3ad6 bytes) 000788 1741626910.010297 mapwrite 0x0035b3 thru 0x003e1b (0x869 bytes) 000789 1741626910.020566 mapread 0x021f16 thru 0x02bc2f (0x9d1a bytes) 000791 1741626910.023425 trunc from 0x040000 to 0x011804 (0x2e7fb bytes) 000794 1741626910.027689 trunc from 0x011805 to 0x0315aa (0x1fda6 bytes) 000796 1741626910.031463 mapwrite 0x038cf5 thru 0x03f91f (0x6c2b bytes) 000797 1741626910.051181 read 0x01fba1 thru 0x02a4c7 (0xa927 bytes) 000798 1741626910.053734 trunc from 0x03f920 to 0x01dac4 (0x21e5b bytes) 000799 1741626910.057649 trunc from 0x01dac5 to 0x0289af (0xaeeb bytes) 000801 1741626910.061100 write 0x02b728 thru 0x034d33 (0x960c bytes) 000802 1741626910.063251 MIRROR_RESYNC 5 mirrors 000804 1741626910.203850 trunc from 0x034d34 to 0x02abad (0xa186 bytes) 000805 1741626910.221304 mapread 0x00e6bc thru 0x01bf5d (0xd8a2 bytes) 000808 1741626910.229795 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000809 1741626910.277186 read 0x029a61 thru 0x02abad (0x114d bytes) 000815 1741626910.280987 trunc from 0x02abae to 0x03a1e0 (0xf633 bytes) 000816 1741626910.285505 write 0x038ed6 thru 0x03ffff (0x712a bytes) 000819 1741626910.288647 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000820 1741626910.332233 mapwrite 0x03153d thru 0x036e8e (0x5952 bytes) 000822 1741626910.351048 mapread 0x010d78 thru 0x0171de (0x6467 bytes) 000824 1741626910.353993 mapwrite 0x0224a5 thru 0x027c20 (0x577c bytes) 000826 1741626910.370320 MIRROR_RESYNC 3 mirrors 000828 1741626910.475230 trunc from 0x040000 to 0x021116 (0x1eee9 bytes) 000829 1741626910.492927 read 0x00e65c thru 0x01c6d8 (0xe07d bytes) 000830 1741626910.495496 read 0x004a45 thru 0x01296f (0xdf2b bytes) 000831 1741626910.498900 mapread 0x0169a7 thru 0x018dfb (0x2455 bytes) 000832 1741626910.499509 write 0x02698b thru 0x035517 (0xeb8d bytes) 000833 1741626910.500546 read 0x020919 thru 0x02c7b5 (0xbe9d bytes) 000836 1741626910.503024 write 0x03b863 thru 0x03c394 (0xb32 bytes) 000838 1741626910.504259 mapread 0x00ea0f thru 0x01bdf8 (0xd3ea bytes) 000839 1741626910.505003 read 0x010e29 thru 0x01300f (0x21e7 bytes) 000842 1741626910.506005 trunc from 0x03c395 to 0x027e63 (0x14531 bytes) 000843 1741626910.510662 write 0x01834a thru 0x01b2ca (0x2f81 bytes) 000844 1741626910.511280 write 0x001008 thru 0x008b03 (0x7afc bytes) 000845 1741626910.513287 MIRROR_RESYNC 3 mirrors 000849 1741626910.622322 read 0x0110e6 thru 0x014198 (0x30b3 bytes) 000850 1741626910.623783 mapwrite 0x01c5d9 thru 0x02729e (0xacc6 bytes) 000855 1741626910.670603 read 0x00cefb thru 0x010cf8 (0x3dfe bytes) 000857 1741626910.673804 mapwrite 0x01a6de thru 0x020fad (0x68d0 bytes) 000858 1741626910.686843 mapread 0x007fbc thru 0x016399 (0xe3de bytes) 000859 1741626910.687514 mapread 0x0101f3 thru 0x01daa8 (0xd8b6 bytes) 000861 1741626910.688465 write 0x0140c7 thru 0x01a6e2 (0x661c bytes) 000863 1741626910.690634 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000864 1741626910.737486 read 0x01ac96 thru 0x01d3dd (0x2748 bytes) 000865 1741626910.739253 trunc from 0x027e64 to 0x01a82e (0xd635 bytes) 000866 1741626910.746394 write 0x0138f5 thru 0x0149e9 (0x10f5 bytes) 000867 1741626910.748882 trunc from 0x01a82f to 0x01b566 (0xd38 bytes) 000869 1741626910.753805 trunc from 0x01b567 to 0x03e309 (0x22da3 bytes) 000870 1741626910.757389 read 0x02c63a thru 0x03a486 (0xde4d bytes) 000871 1741626910.761394 read 0x0144ba thru 0x016972 (0x24b9 bytes) 000872 1741626910.763051 MIRROR_EXTEND to 3 mirrors 000873 1741626910.873745 read 0x027a8c thru 0x0296a5 (0x1c1a bytes) 000874 1741626910.875899 mapread 0x01d487 thru 0x0251be (0x7d38 bytes) 000876 1741626910.883513 trunc from 0x03e30a to 0x0167ec (0x27b1d bytes) 000878 1741626910.901975 mapwrite 0x009117 thru 0x00cda4 (0x3c8e bytes) 000884 1741626910.918424 trunc from 0x0167ed to 0x03e7e6 (0x27ffa bytes) 000885 1741626910.923311 mapread 0x0081f5 thru 0x00cf28 (0x4d34 bytes) 000888 1741626910.924497 mapread 0x02bd49 thru 0x039118 (0xd3d0 bytes) 000889 1741626910.928807 mapread 0x03408a thru 0x03a3ea (0x6361 bytes) 000892 1741626910.929919 mapread 0x035ac1 thru 0x03c49b (0x69db bytes) 000893 1741626910.930364 read 0x019f57 thru 0x02417b (0xa225 bytes) 000896 1741626910.933785 write 0x022206 thru 0x02588b (0x3686 bytes) 000897 1741626910.934501 mapread 0x007724 thru 0x016c3f (0xf51c bytes) 000898 1741626910.935299 read 0x00c73b thru 0x016b2e (0xa3f4 bytes) 000900 1741626910.936096 mapread 0x02f72a thru 0x03ce5b (0xd732 bytes) 000901 1741626910.936720 mapwrite 0x0209b4 thru 0x0291b3 (0x8800 bytes) 000902 1741626910.950127 mapread 0x00e410 thru 0x017ee6 (0x9ad7 bytes) 000904 1741626910.951073 trunc from 0x03e7e7 to 0x017cfe (0x26ae8 bytes) 000905 1741626910.956273 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000906 1741626911.000225 mapwrite 0x015bd1 thru 0x0250de (0xf50e bytes) 000909 1741626911.027531 mapread 0x0214be thru 0x0250de (0x3c21 bytes) 000910 1741626911.028334 trunc from 0x0250df to 0x023f37 (0x11a7 bytes) 000911 1741626911.032152 write 0x012bcc thru 0x020961 (0xdd96 bytes) 000912 1741626911.032893 trunc from 0x023f38 to 0x0350de (0x111a7 bytes) 000913 1741626911.036597 trunc from 0x0350df to 0x02fc0c (0x54d2 bytes) 000915 1741626911.040180 mapread 0x02d17d thru 0x02fc0c (0x2a90 bytes) 000916 1741626911.043819 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000917 1741626911.080693 mapread 0x00bb71 thru 0x01adac (0xf23c bytes) 000919 1741626911.087920 mapwrite 0x02b50a thru 0x03222b (0x6d22 bytes) 000922 1741626911.107919 trunc from 0x03222c to 0x038864 (0x6639 bytes) 000923 1741626911.111818 trunc from 0x038865 to 0x02ee6b (0x99f9 bytes) 000924 1741626911.115345 write 0x01a656 thru 0x026075 (0xba20 bytes) 000925 1741626911.116046 read 0x02dbdf thru 0x02ee6b (0x128d bytes) 000926 1741626911.116381 read 0x00a8e0 thru 0x011a96 (0x71b7 bytes) 000927 1741626911.116735 read 0x01637a thru 0x0196af (0x3336 bytes) 000929 1741626911.117262 mapwrite 0x03abfb thru 0x03ffff (0x5405 bytes) 000931 1741626911.138783 mapread 0x0196a9 thru 0x0216ff (0x8057 bytes) 000934 1741626911.140213 read 0x01d99b thru 0x02760f (0x9c75 bytes) 000935 1741626911.140610 write 0x028998 thru 0x03300c (0xa675 bytes) 000937 1741626911.141615 trunc from 0x040000 to 0x028c26 (0x173d9 bytes) 000938 1741626911.146012 mapread 0x01e5ba thru 0x01edcf (0x816 bytes) 000941 1741626911.147394 mapread 0x00f583 thru 0x015381 (0x5dff bytes) 000943 1741626911.148587 mapread 0x0062bf thru 0x007554 (0x1296 bytes) 000944 1741626911.149096 trunc from 0x028c27 to 0x01a295 (0xe991 bytes) 000945 1741626911.152801 mapwrite 0x0203cc thru 0x029a8c (0x96c1 bytes) 000947 1741626911.172954 mapread 0x026e3a thru 0x029a8c (0x2c53 bytes) 000949 1741626911.173756 mapread 0x0013f8 thru 0x0051c7 (0x3dd0 bytes) 000951 1741626911.174896 mapread 0x023f24 thru 0x029a8c (0x5b69 bytes) 000954 1741626911.176241 trunc from 0x029a8d to 0x00b9e3 (0x1e0a9 bytes) 000955 1741626911.179976 read 0x00597f thru 0x00b9e3 (0x6065 bytes) 000958 1741626911.182347 MIRROR_EXTEND to 2 mirrors 000960 1741626911.270071 write 0x0112ff thru 0x01dfc5 (0xccc7 bytes) 000961 1741626911.284103 write 0x008cf6 thru 0x0166ff (0xda0a bytes) 000963 1741626911.286283 read 0x011f8e thru 0x01b0ee (0x9161 bytes) 000964 1741626911.286660 mapwrite 0x00772f thru 0x011baf (0xa481 bytes) 000965 1741626911.302223 write 0x0297f1 thru 0x02d6ac (0x3ebc bytes) 000966 1741626911.302833 mapread 0x0014c4 thru 0x0039ed (0x252a bytes) 000970 1741626911.305743 trunc from 0x02d6ad to 0x03e39c (0x10cf0 bytes) 000971 1741626911.314005 write 0x026e54 thru 0x0293f7 (0x25a4 bytes) 000972 1741626911.317324 mapread 0x02c380 thru 0x03ba54 (0xf6d5 bytes) 000973 1741626911.322073 mapwrite 0x02c2d9 thru 0x02fac1 (0x37e9 bytes) 000974 1741626911.335178 write 0x03ff3a thru 0x03ffff (0xc6 bytes) 000975 1741626911.335690 mapread 0x03bf0b thru 0x03ffff (0x40f5 bytes) 000976 1741626911.336111 read 0x03279d thru 0x0359ee (0x3252 bytes) 000977 1741626911.336416 trunc from 0x040000 to 0x008177 (0x37e88 bytes) 000978 1741626911.341506 MIRROR_RESYNC 2 mirrors 000979 1741626911.420050 write 0x03f3a2 thru 0x03ffff (0xc5e bytes) 000980 1741626911.433210 read 0x0101d3 thru 0x01aca3 (0xaad1 bytes) 000982 1741626911.437007 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x933:0x0]: Invalid argument (22) 000986 1741626911.473382 mapread 0x02df62 thru 0x0309a0 (0x2a3f bytes) 000987 1741626911.478427 mapwrite 0x01032a thru 0x01e4f4 (0xe1cb bytes) 000990 1741626911.498954 mapread 0x00bcd3 thru 0x014493 (0x87c1 bytes) 000991 1741626911.499644 read 0x014fac thru 0x022cf3 (0xdd48 bytes) 000992 1741626911.501257 mapwrite 0x03eb1f thru 0x03ffff (0x14e1 bytes) 000994 1741626911.514992 write 0x00948d thru 0x018c78 (0xf7ec bytes) 000995 1741626911.515852 mapwrite 0x02b045 thru 0x033345 (0x8301 bytes) 000997 1741626911.530407 MIRROR_EXTEND to 2 mirrors 001000 1741626911.628450 read 0x03d0ca thru 0x03ffff (0x2f36 bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 13:15:12 (1741626912) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 13:15:13 (1741626913) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2201792 .. Extending file size to 1428224 .. failed Extending file size to 5619712 .. Extending file size to 6919392 .. Extending file size to 8412320 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8537248 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7986560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9265184 .. done Extending file size to 9376928 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 30728 30729 30731 30732 30735 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:16:16 (1741626976) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6116224 .. Extending file size to 1283328 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6180640 .. failed Extending file size to 8836480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8604640 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8763904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8907936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8797504 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9134368 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9001376 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9269248 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 2562 2563 2565 2566 2569 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) 192.168.204.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:17:19 (1741627039) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0340027 s, 61.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0794403 s, 26.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:17:24 (1741627044) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0248677 s, 84.3 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 13:17:25 (1741627045) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x97c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd2e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd2e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd2f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xd29:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:17:27 (1741627047) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0250596 s, 83.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:17:29 (1741627049) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0310716 s, 67.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd32:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd32:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2d:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:17:30 (1741627050) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0224583 s, 93.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd2f:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:17:53 (1741627073) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152959 s, 68.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd35:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:18:37 (1741627117) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd33:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0238771 s, 87.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:19:22 (1741627162) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg427-server mds-ost sync done. Creating new pool test_204f oleg427-server: Pool lustre.test_204f created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.023277 s, 90.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg427-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:20:13 (1741627213) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:20:15 (1741627215) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:20:16 (1741627216) oleg427-server: Pool lustre.test_206 created Adding targets to pool oleg427-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg427-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg427-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg427-server: Pool lustre.test_206 destroyed PASS 206 (10s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:20:26 (1741627226) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019263 s, 54.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914409 s, 115 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x5aef2937 Mirror 3: 0x5aef2937 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:20:28 (1741627228) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:20:29 (1741627229) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:20:29 (1741627229) Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0178533 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x13c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x13c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x13c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9c0000 Read this (4 bytes): BBBB 192.168.204.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg427-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:20:42 (1741627242) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00257684 s, 3.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:20:44 (1741627244) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109006 s, 96.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x9a1 lmm_fid: [0x200000404:0x9a1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x9a1 lmm_fid: [0x200000404:0x9a1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd42:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836385 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-10 13:20:44.000000000 -0400 Modify: 2025-03-10 13:20:44.000000000 -0400 Change: 2025-03-10 13:20:45.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:20:46 (1741627246) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:20:46 (1741627246) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.102272 s, 401 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0129512 s, 316 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 10 13:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9a3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 10 13:20 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 679 sec ============ 13:20:49 (1741627249) === sanity-flr: start cleanup 13:20:49 (1741627249) === === sanity-flr: finish cleanup 13:20:50 (1741627250) ===