-----============= acceptance-small: sanity-flr ============----- Tue Jun 4 23:04:46 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 23:04:51 (1717556691) === oleg147-client.virtnet: executing check_config_client /mnt/lustre oleg147-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg147-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a4e0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a4e0000.idle_timeout=debug disable quota as required oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:05:00 (1717556700) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7538] == sanity-flr test 0a: lfs mirror create with -N option == 23:05:00 (1717556700) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:05:04 (1717556704) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:05:05 (1717556705) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:05:06 (1717556706) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:05:12 (1717556712) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:05:13 (1717556713) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:05:14 (1717556714) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a4e0000.stats=clear osc.lustre-OST0001-osc-ffff88012a4e0000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:05:16 (1717556716) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:05:19 (1717556719) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:05:22 (1717556722) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:05:23 (1717556723) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 23:05:26 (1717556726) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:05:28 (1717556728) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:05:30 (1717556730) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:05:32 (1717556732) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:05:35 (1717556735) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:05:36 (1717556736) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.213907 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293735 s, 35.7 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:05:39 (1717556739) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0275791 s, 304 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:05:41 (1717556741) Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:06:07 (1717556767) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0230912 s, 90.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:06:16 (1717556776) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:06:37 (1717556797) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.613487 s, 85.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 4 23:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.218s user 0m0.000s sys 0m0.103s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.218s user 0m0.000s sys 0m0.096s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.238s user 0m0.000s sys 0m0.094s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a4e0000.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:06:59 (1717556819) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:06:59 (1717556819) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:07:00 (1717556820) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:07:01 (1717556821) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 23:07:03 (1717556823) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0812356 s, 155 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:07:05 (1717556825) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00587703 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00538377 s, 195 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:07:07 (1717556827) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:07:10 (1717556830) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000557868 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000748943 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000611342 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000542038 s, 1.9 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:07:12 (1717556832) create test files with size 15756113 .. 7529c37c121c71ac49987b9fa8bd1ef1 - e1993e0ae5949ea64f86c3af9eb40185 - f76bfef4599c130271f4005fd1339895 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 23:07:18 (1717556838) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:07:23 (1717556843) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:07:25 (1717556845) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0310163 s, 33.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00606866 s, 173 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:07:27 (1717556847) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0185812 s, 113 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.090909 s, 46.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0510385 s, 41.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:07:30 (1717556850) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:07:31 (1717556851) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:07:32 (1717556852) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00630906 s, 166 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:07:34 (1717556854) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.074212 s, 42.4 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:07:37 (1717556857) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:07:39 (1717556859) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0394668 s, 266 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:07:42 (1717556862) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aad38800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0744471 s, 141 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aad3b000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:07:46 (1717556866) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:07:54 (1717556874) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:08:01 (1717556881) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:08:03 (1717556883) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0994612 s, 31.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0454446 s, 69.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: f0aed708579857891de5611fdfbde74e - mirror 1 checksum: 2957f722662afac129743c3e1e73272c - mirror 2 checksum: f0aed708579857891de5611fdfbde74e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 4 23:08:03 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 4 23:08:03 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f0aed708579857891de5611fdfbde74e - mirror 1 checksum: 2957f722662afac129743c3e1e73272c - mirror 2 checksum: 2957f722662afac129743c3e1e73272c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 4 23:08:03 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 4 23:08:03 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:08:06 (1717556886) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00626137 s, 167 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:08:08 (1717556888) Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 23:08:15 (1717556895) shut down Failover mds1 to oleg147-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 23:08:29 (1717556909) targets are mounted 23:08:29 (1717556909) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00658051 s, 159 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:08:36 (1717556916) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.014763 s, 1.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00704437 s, 2.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:08:39 (1717556919) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0128414 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00261315 s, 7.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:08:42 (1717556922) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00199532 s, 8.2 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:08:45 (1717556925) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0152711 s, 1.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:08:47 (1717556927) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00326925 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.014532 s, 282 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:08:49 (1717556929) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:08:55 (1717556935) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:09:13 (1717556953) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:09:30 (1717556970) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:09:47 (1717556987) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 23:09:49 (1717556989) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 24285 mirror split pid 24286 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4717: 24285 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4717: 24286 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 23:10:50 (1717557050) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7050 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole 000008 1717557050.840341 mapwrite 0x01a0fa thru 0x01b3e5 (0x12ec bytes) 000009 1717557050.851524 mapwrite 0x0279bb thru 0x02bb2c (0x4172 bytes) truncating to largest ever: 0x7b6d 000010 1717557050.860276 trunc from 0x02bb2d to 0x007b6c (0x23fc0 bytes) fallocating to largest ever: 0x40000 000011 1717557050.863389 fallocate 0x03e895 thru 0x03ffff (0x176b bytes) 000014 1717557050.865000 read 0x000ede thru 0x0036b0 (0x27d3 bytes) 000016 1717557050.867493 MIRROR_EXTEND to 2 mirrors 000017 1717557050.940037 MIRROR_RESYNC 2 mirrors 000019 1717557050.969937 trunc from 0x007b6d to 0x00e8b3 (0x6d47 bytes) 000020 1717557050.980743 trunc from 0x00e8b4 to 0x00fc4a (0x1397 bytes) 000022 1717557050.984314 trunc from 0x00fc4b to 0x016c63 (0x7019 bytes) 000023 1717557050.986972 fallocate 0x009fbf thru 0x013345 (0x9387 bytes) 000025 1717557050.988150 mapwrite 0x0169f6 thru 0x0242a8 (0xd8b3 bytes) 000026 1717557051.001176 fallocate 0x03d7e1 thru 0x03ffff (0x281f bytes) 000027 1717557051.002494 fallocate 0x026379 thru 0x0297a3 (0x342b bytes) 000028 1717557051.003539 trunc from 0x0242a9 to 0x01b6a5 (0x8c03 bytes) 000029 1717557051.006611 trunc from 0x01b6a6 to 0x03fb2d (0x24488 bytes) 000030 1717557051.009683 read 0x00bfbd thru 0x015358 (0x939c bytes) 000031 1717557051.011514 write 0x0245eb thru 0x027364 (0x2d7a bytes) 000032 1717557051.013725 trunc from 0x03fb2e to 0x01620f (0x2991e bytes) 000033 1717557051.016740 fallocate 0x032276 thru 0x03338f (0x111a bytes) 000034 1717557051.017758 punch from 0x015efe to 0x01a9b2 (0x4ab5 bytes) 000035 1717557051.019581 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000037 1717557051.045729 mapwrite 0x02cb7f thru 0x02ef7d (0x23ff bytes) 000038 1717557051.053269 read 0x017d19 thru 0x01c8bf (0x4ba7 bytes) 000039 1717557051.054930 mapread 0x0332ac thru 0x03338f (0xe4 bytes) 000040 1717557051.055267 mapwrite 0x02fe44 thru 0x03f50c (0xf6c9 bytes) 000041 1717557051.067131 fallocate 0x012870 thru 0x018b05 (0x6296 bytes) 000042 1717557051.068339 mapread 0x030cf6 thru 0x03d21b (0xc526 bytes) 000043 1717557051.068762 punch from 0x01f194 to 0x0253ce (0x623b bytes) 000044 1717557051.069716 mapwrite 0x03ab20 thru 0x03ffff (0x54e0 bytes) 000045 1717557051.076856 punch from 0x0319cb to 0x033fa9 (0x25df bytes) 000046 1717557051.077856 fallocate 0x0054fc thru 0x013768 (0xe26d bytes) 000048 1717557051.078920 punch from 0x037b7b to 0x03ba08 (0x3e8e bytes) 000049 1717557051.080003 fallocate 0x0052ee thru 0x0103fc (0xb10f bytes) 000050 1717557051.081011 trunc from 0x040000 to 0x037aa5 (0x855a bytes) 000051 1717557051.084918 MIRROR_EXTEND to 2 mirrors 000052 1717557051.170926 punch from 0x019f0b to 0x027d9c (0xde92 bytes) 000053 1717557051.177407 write 0x0232db thru 0x02d88e (0xa5b4 bytes) 000054 1717557051.180491 read 0x0119c8 thru 0x012180 (0x7b9 bytes) 000055 1717557051.181723 read 0x007d12 thru 0x014197 (0xc486 bytes) 000056 1717557051.184434 trunc from 0x037aa6 to 0x013980 (0x24125 bytes) 000057 1717557051.191013 punch from 0x00de95 to 0x00f62b (0x1797 bytes) 000059 1717557051.192388 mapread 0x005bcf thru 0x0085cc (0x29fe bytes) 000060 1717557051.192816 write 0x0308c5 thru 0x03a75d (0x9e99 bytes) 000061 1717557051.194454 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000063 1717557051.228915 fallocate 0x021b9e thru 0x0308b9 (0xed1c bytes) 000064 1717557051.231051 MIRROR_EXTEND to 2 mirrors 000066 1717557051.317045 mapwrite 0x0202f3 thru 0x027143 (0x6e51 bytes) 000067 1717557051.345718 write 0x016f82 thru 0x01e29c (0x731b bytes) 000068 1717557051.348581 read 0x02183c thru 0x029f86 (0x874b bytes) 000070 1717557051.348892 trunc from 0x03a75e to 0x020f49 (0x19814 bytes) 000071 1717557051.356575 MIRROR_EXTEND to 3 mirrors 000072 1717557051.448286 punch from 0x01a0d4 to 0x01f842 (0x576f bytes) 000073 1717557051.454974 write 0x018b74 thru 0x0211d9 (0x8666 bytes) 000074 1717557051.457108 mapwrite 0x007882 thru 0x011363 (0x9ae2 bytes) 000075 1717557051.468682 MIRROR_EXTEND to 4 mirrors 000076 1717557051.549560 read 0x020cef thru 0x0211d9 (0x4eb bytes) 000077 1717557051.552913 MIRROR_RESYNC 4 mirrors 000078 1717557051.648520 fallocate 0x024ed0 thru 0x024eff (0x30 bytes) 000079 1717557051.654820 fallocate 0x00d4da thru 0x019c91 (0xc7b8 bytes) 000080 1717557051.655894 read 0x011958 thru 0x016be8 (0x5291 bytes) 000081 1717557051.657607 write 0x029bfc thru 0x032435 (0x883a bytes) 000082 1717557051.657964 read 0x01d92f thru 0x021026 (0x36f8 bytes) 000083 1717557051.660369 read 0x01ab27 thru 0x028a4a (0xdf24 bytes) 000084 1717557051.660679 read 0x0129fd thru 0x01ff2f (0xd533 bytes) 000085 1717557051.660880 trunc from 0x032436 to 0x013145 (0x1f2f0 bytes) 000086 1717557051.668139 mapread 0x00a049 thru 0x00be89 (0x1e41 bytes) 000088 1717557051.669643 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000089 1717557051.712862 mapread 0x008703 thru 0x00cc7a (0x4578 bytes) 000090 1717557051.716656 write 0x03b22e thru 0x03ffff (0x4dd2 bytes) 000091 1717557051.717918 read 0x02714e thru 0x030997 (0x984a bytes) 000093 1717557051.720481 MIRROR_RESYNC 3 mirrors 000095 1717557051.809073 mapwrite 0x028f62 thru 0x0379b7 (0xea56 bytes) 000096 1717557051.836537 mapwrite 0x026304 thru 0x026821 (0x51e bytes) 000097 1717557051.839286 mapread 0x003726 thru 0x0057dc (0x20b7 bytes) 000098 1717557051.841060 punch from 0x001da0 to 0x00d6f5 (0xb956 bytes) 000099 1717557051.842235 mapread 0x001a02 thru 0x00d878 (0xbe77 bytes) 000101 1717557051.845710 trunc from 0x040000 to 0x02ab5f (0x154a0 bytes) 000102 1717557051.853108 fallocate 0x0368eb thru 0x03f93f (0x9055 bytes) 000103 1717557051.854072 write 0x027140 thru 0x02d863 (0x6724 bytes) 000105 1717557051.855416 write 0x01027c thru 0x01383b (0x35c0 bytes) 000106 1717557051.855671 trunc from 0x03f940 to 0x02e219 (0x11726 bytes) 000107 1717557051.858625 trunc from 0x02e21a to 0x01bcbf (0x1255a bytes) 000108 1717557051.861651 read 0x016a07 thru 0x01bcbf (0x52b9 bytes) 000109 1717557051.863423 read 0x010493 thru 0x017e5c (0x79ca bytes) 000110 1717557051.863591 fallocate 0x005b03 thru 0x00ed75 (0x9273 bytes) 000111 1717557051.864617 trunc from 0x01bcc0 to 0x03debf (0x22200 bytes) 000113 1717557051.867706 read 0x022872 thru 0x02ed94 (0xc523 bytes) 000114 1717557051.871044 MIRROR_EXTEND to 4 mirrors 000115 1717557051.958409 fallocate 0x0292a8 thru 0x0296b6 (0x40f bytes) 000117 1717557051.965231 trunc from 0x03dec0 to 0x00dc79 (0x30246 bytes) 000118 1717557051.972460 read 0x00211b thru 0x00dc79 (0xbb5f bytes) 000119 1717557051.974624 mapread 0x00db65 thru 0x00dc79 (0x115 bytes) 000120 1717557051.974996 trunc from 0x00dc7a to 0x03d10c (0x2f493 bytes) 000121 1717557051.978069 mapread 0x014f6c thru 0x0164cb (0x1560 bytes) 000123 1717557051.982159 write 0x03751e thru 0x03ccd7 (0x57ba bytes) 000126 1717557051.984290 fallocate 0x027112 thru 0x0320cd (0xafbc bytes) 000128 1717557051.985585 trunc from 0x03d10d to 0x03c3ca (0xd42 bytes) 000129 1717557051.988520 mapread 0x035132 thru 0x035b94 (0xa63 bytes) 000130 1717557051.988880 read 0x035c30 thru 0x03c3ca (0x679b bytes) 000131 1717557051.989123 mapread 0x03517d thru 0x03c3ca (0x724e bytes) 000132 1717557051.989371 trunc from 0x03c3cb to 0x038b8e (0x383c bytes) 000133 1717557051.993075 MIRROR_RESYNC 4 mirrors 000134 1717557052.130085 mapwrite 0x03630e thru 0x03ffff (0x9cf2 bytes) 000136 1717557052.145772 trunc from 0x040000 to 0x0235bb (0x1ca44 bytes) 000137 1717557052.149540 fallocate 0x01f424 thru 0x029b91 (0xa76e bytes) 000138 1717557052.150642 read 0x005064 thru 0x00c234 (0x71d1 bytes) 000139 1717557052.152724 trunc from 0x0235bc to 0x00b819 (0x17da2 bytes) 000140 1717557052.155668 write 0x00c008 thru 0x00f407 (0x3400 bytes) 000141 1717557052.155979 fallocate 0x004a0b thru 0x012ec4 (0xe4ba bytes) 000142 1717557052.156959 mapwrite 0x03277f thru 0x036ef9 (0x477b bytes) 000143 1717557052.165334 fallocate 0x020539 thru 0x02d94a (0xd412 bytes) 000144 1717557052.168072 mapwrite 0x015cb1 thru 0x016dd3 (0x1123 bytes) 000145 1717557052.172054 trunc from 0x036efa to 0x014285 (0x22c74 bytes) 000146 1717557052.174760 read 0x00c9d9 thru 0x01040b (0x3a33 bytes) 000147 1717557052.174970 read 0x0098ab thru 0x014285 (0xa9db bytes) 000149 1717557052.175244 mapwrite 0x008cef thru 0x00bdde (0x30f0 bytes) 000150 1717557052.178996 write 0x00ea0f thru 0x01caae (0xe0a0 bytes) 000151 1717557052.179490 write 0x0380f4 thru 0x03d516 (0x5423 bytes) 000152 1717557052.179771 write 0x014e8f thru 0x019357 (0x44c9 bytes) 000153 1717557052.181124 MIRROR_EXTEND to 5 mirrors 000154 1717557052.265882 fallocate 0x030c0a thru 0x03ffff (0xf3f6 bytes) 000155 1717557052.272268 punch from 0x0286b0 to 0x028824 (0x175 bytes) 000156 1717557052.274146 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000157 1717557052.317727 mapwrite 0x000aeb thru 0x00d2cd (0xc7e3 bytes) 000158 1717557052.327869 mapwrite 0x006079 thru 0x010cf2 (0xac7a bytes) 000159 1717557052.331801 mapwrite 0x001c40 thru 0x004d6d (0x312e bytes) 000160 1717557052.335916 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000161 1717557052.379117 trunc from 0x03d517 to 0x0301c1 (0xd355 bytes) 000162 1717557052.385624 trunc from 0x0301c2 to 0x035b16 (0x5955 bytes) 000163 1717557052.388135 mapwrite 0x013b9b thru 0x022fb4 (0xf41a bytes) 000165 1717557052.398615 write 0x0362b3 thru 0x038e22 (0x2b70 bytes) 000166 1717557052.399048 mapwrite 0x01db61 thru 0x01f3de (0x187e bytes) 000167 1717557052.402581 mapwrite 0x01cf8f thru 0x01e546 (0x15b8 bytes) 000168 1717557052.404967 fallocate 0x00428e thru 0x01318a (0xeefd bytes) 000170 1717557052.405929 write 0x01e7b2 thru 0x02348e (0x4cdd bytes) 000171 1717557052.406158 write 0x0006c1 thru 0x00c212 (0xbb52 bytes) 000172 1717557052.408003 mapwrite 0x030dd0 thru 0x0311a7 (0x3d8 bytes) 000173 1717557052.413754 write 0x015ecc thru 0x01a2d4 (0x4409 bytes) 000174 1717557052.414003 fallocate 0x0214ec thru 0x025035 (0x3b4a bytes) 000177 1717557052.415301 write 0x03b3f1 thru 0x03ffff (0x4c0f bytes) 000178 1717557052.415760 mapwrite 0x005810 thru 0x00e991 (0x9182 bytes) 000179 1717557052.420131 fallocate 0x03d387 thru 0x03ffff (0x2c79 bytes) 000182 1717557052.421372 mapread 0x02d528 thru 0x0399f5 (0xc4ce bytes) 000183 1717557052.422654 mapread 0x0348c8 thru 0x03d1ed (0x8926 bytes) 000184 1717557052.423015 mapread 0x02717a thru 0x02be49 (0x4cd0 bytes) 000185 1717557052.425297 MIRROR_EXTEND to 4 mirrors 000186 1717557052.521603 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000190 1717557052.592464 write 0x03a9e7 thru 0x03edcf (0x43e9 bytes) 000191 1717557052.601271 trunc from 0x040000 to 0x0348c2 (0xb73d bytes) 000192 1717557052.608218 read 0x02d046 thru 0x0320f8 (0x50b3 bytes) 000193 1717557052.610076 read 0x019c37 thru 0x0266fb (0xcac5 bytes) 000194 1717557052.613710 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000196 1717557052.655557 mapread 0x033ff1 thru 0x0348c2 (0x8d2 bytes) 000197 1717557052.657824 punch from 0x0269e8 to 0x031615 (0xac2e bytes) 000198 1717557052.659026 write 0x0252e5 thru 0x028d39 (0x3a55 bytes) 000199 1717557052.661748 write 0x008a8c thru 0x013ac2 (0xb037 bytes) 000200 1717557052.664553 mapwrite 0x00e634 thru 0x012de7 (0x47b4 bytes) 000201 1717557052.669373 mapwrite 0x0150a8 thru 0x019cfb (0x4c54 bytes) 000202 1717557052.674157 punch from 0x020dd8 to 0x027bf8 (0x6e21 bytes) 000203 1717557052.675315 mapwrite 0x005e6c thru 0x0064f2 (0x687 bytes) 000205 1717557052.678259 trunc from 0x0348c3 to 0x0321a6 (0x271c bytes) 000206 1717557052.681428 mapread 0x0247e3 thru 0x029e46 (0x5664 bytes) 000207 1717557052.683675 trunc from 0x0321a7 to 0x01198f (0x20817 bytes) 000208 1717557052.687781 MIRROR_EXTEND to 3 mirrors 000209 1717557052.752641 trunc from 0x011990 to 0x026bc6 (0x15237 bytes) 000210 1717557052.762839 trunc from 0x026bc7 to 0x00dee6 (0x18ce0 bytes) 000212 1717557052.766560 fallocate 0x031c84 thru 0x03392f (0x1cac bytes) 000213 1717557052.768496 MIRROR_RESYNC 3 mirrors 000215 1717557052.838661 trunc from 0x00dee7 to 0x02ce69 (0x1ef83 bytes) 000218 1717557052.850419 trunc from 0x02ce6a to 0x004c96 (0x281d3 bytes) 000220 1717557052.854592 mapread 0x000fa5 thru 0x002349 (0x13a5 bytes) 000223 1717557052.858834 trunc from 0x004c97 to 0x018ecf (0x14239 bytes) 000224 1717557052.861934 read 0x00efb5 thru 0x018ecf (0x9f1b bytes) 000225 1717557052.864551 MIRROR_EXTEND to 4 mirrors 000226 1717557052.945462 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000227 1717557053.003909 write 0x006c85 thru 0x0129fe (0xbd7a bytes) 000228 1717557053.012376 mapwrite 0x0120f6 thru 0x02042b (0xe336 bytes) 000229 1717557053.027322 write 0x011c79 thru 0x01f0e0 (0xd468 bytes) 000231 1717557053.027901 fallocate 0x010d17 thru 0x011aa1 (0xd8b bytes) 000232 1717557053.029926 MIRROR_RESYNC 3 mirrors 000233 1717557053.114824 read 0x00047a thru 0x0099b2 (0x9539 bytes) 000234 1717557053.116710 mapread 0x01f8a4 thru 0x02042b (0xb88 bytes) 000235 1717557053.117814 trunc from 0x02042c to 0x018a31 (0x79fa bytes) 000236 1717557053.128810 read 0x01489a thru 0x018a31 (0x4198 bytes) 000237 1717557053.130985 fallocate 0x00e851 thru 0x013f11 (0x56c1 bytes) 000238 1717557053.132026 fallocate 0x008a88 thru 0x00dc37 (0x51b0 bytes) 000239 1717557053.132927 mapwrite 0x025e1c thru 0x027c60 (0x1e45 bytes) 000240 1717557053.142503 mapwrite 0x01d42a thru 0x025ef2 (0x8ac9 bytes) 000241 1717557053.146655 write 0x004f8d thru 0x00be14 (0x6e88 bytes) 000242 1717557053.147121 mapread 0x024c36 thru 0x027c60 (0x302b bytes) 000243 1717557053.147485 fallocate 0x03cf48 thru 0x03ffff (0x30b8 bytes) 000244 1717557053.148593 mapwrite 0x036324 thru 0x03ffff (0x9cdc bytes) 000245 1717557053.156653 MIRROR_EXTEND to 4 mirrors 000246 1717557053.252789 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000247 1717557053.316939 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000248 1717557053.383996 punch from 0x0285e3 to 0x02a7f1 (0x220f bytes) 000250 1717557053.391283 fallocate 0x03029b thru 0x03b905 (0xb66b bytes) 000252 1717557053.393488 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000253 1717557053.422765 MIRROR_EXTEND to 2 mirrors 000254 1717557053.517369 punch from 0x030ae3 to 0x035c9b (0x51b9 bytes) 000255 1717557053.523552 trunc from 0x040000 to 0x00fae5 (0x3051a bytes) 000256 1717557053.530529 mapread 0x00498b thru 0x00fae5 (0xb15b bytes) 000257 1717557053.535133 mapread 0x00f8cf thru 0x00fae5 (0x217 bytes) 000259 1717557053.536435 MIRROR_EXTEND to 3 mirrors 000260 1717557053.601500 mapwrite 0x00fa88 thru 0x01aafb (0xb074 bytes) 000261 1717557053.621323 mapwrite 0x02cc4a thru 0x039466 (0xc81d bytes) 000262 1717557053.633985 read 0x01f3f0 thru 0x024d84 (0x5995 bytes) 000263 1717557053.635735 mapread 0x0393fa thru 0x039466 (0x6d bytes) 000264 1717557053.635941 read 0x015a14 thru 0x020b2e (0xb11b bytes) 000265 1717557053.637807 mapread 0x00679c thru 0x00af45 (0x47aa bytes) 000266 1717557053.638937 MIRROR_RESYNC 3 mirrors 000267 1717557053.726754 mapwrite 0x00ec22 thru 0x0180e5 (0x94c4 bytes) 000268 1717557053.755336 mapwrite 0x0016a8 thru 0x001c31 (0x58a bytes) 000269 1717557053.759710 mapwrite 0x005565 thru 0x00d4b7 (0x7f53 bytes) 000270 1717557053.763636 mapread 0x028730 thru 0x029b7d (0x144e bytes) 000271 1717557053.765566 fallocate 0x00201e thru 0x008671 (0x6654 bytes) 000273 1717557053.766773 write 0x0377ca thru 0x03b061 (0x3898 bytes) 000275 1717557053.768567 mapread 0x0069ea thru 0x008f1f (0x2536 bytes) 000276 1717557053.769853 MIRROR_RESYNC 3 mirrors 000278 1717557053.854798 mapread 0x005d19 thru 0x0093ba (0x36a2 bytes) 000279 1717557053.858068 mapwrite 0x008ce2 thru 0x00a4fa (0x1819 bytes) 000280 1717557053.881036 mapread 0x012658 thru 0x012755 (0xfe bytes) 000282 1717557053.883062 read 0x035159 thru 0x03b061 (0x5f09 bytes) 000283 1717557053.884706 write 0x000f71 thru 0x00b1d1 (0xa261 bytes) 000285 1717557053.886232 trunc from 0x03b062 to 0x005649 (0x35a18 bytes) 000286 1717557053.893611 read 0x003590 thru 0x005649 (0x20ba bytes) 000289 1717557053.894219 mapread 0x0018ae thru 0x005649 (0x3d9c bytes) 000290 1717557053.894457 punch from 0x002bbe to 0x005649 (0x2a8c bytes) 000291 1717557053.897575 read 0x0054a9 thru 0x005649 (0x1a1 bytes) 000292 1717557053.900094 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000293 1717557053.939608 read 0x004ca7 thru 0x005649 (0x9a3 bytes) 000295 1717557053.941681 MIRROR_EXTEND to 3 mirrors 000296 1717557054.015684 write 0x030706 thru 0x0339b3 (0x32ae bytes) 000297 1717557054.022573 fallocate 0x020d6c thru 0x023999 (0x2c2e bytes) 000298 1717557054.023750 fallocate 0x028680 thru 0x034104 (0xba85 bytes) 000299 1717557054.025010 write 0x0272b9 thru 0x02b956 (0x469e bytes) 000300 1717557054.027006 fallocate 0x038433 thru 0x03f68d (0x725b bytes) 000301 1717557054.028104 trunc from 0x0339b4 to 0x00e3ee (0x255c5 bytes) 000302 1717557054.035369 MIRROR_EXTEND to 4 mirrors 000303 1717557054.125172 MIRROR_EXTEND to 5 mirrors 000304 1717557054.203744 write 0x007d65 thru 0x0098de (0x1b7a bytes) 000306 1717557054.211724 mapwrite 0x03ead5 thru 0x03ffff (0x152b bytes) 000307 1717557054.221391 fallocate 0x03599f thru 0x03dd16 (0x8378 bytes) 000308 1717557054.222688 trunc from 0x040000 to 0x001902 (0x3e6fd bytes) 000309 1717557054.226028 punch from 0x000e93 to 0x001902 (0xa70 bytes) 000310 1717557054.228275 MIRROR_RESYNC 5 mirrors 000311 1717557054.313541 punch from 0x0005d7 to 0x001902 (0x132c bytes) 000313 1717557054.321102 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000314 1717557054.365608 mapwrite 0x028947 thru 0x031fd9 (0x9693 bytes) 000315 1717557054.378053 trunc from 0x031fda to 0x01119d (0x20e3c bytes) 000316 1717557054.380656 mapwrite 0x00927f thru 0x011429 (0x81ab bytes) 000317 1717557054.391598 trunc from 0x01142a to 0x02f425 (0x1dffc bytes) 000319 1717557054.394709 mapread 0x023a05 thru 0x02f425 (0xba21 bytes) 000320 1717557054.398074 punch from 0x0242b4 to 0x02f425 (0xb172 bytes) 000321 1717557054.399049 punch from 0x021a15 to 0x027c11 (0x61fd bytes) 000322 1717557054.400020 read 0x020911 thru 0x029bbd (0x92ad bytes) 000323 1717557054.401748 mapwrite 0x00a903 thru 0x00d135 (0x2833 bytes) 000324 1717557054.405210 fallocate 0x02788c thru 0x0279ef (0x164 bytes) 000326 1717557054.406522 mapread 0x020a6a thru 0x02ca54 (0xbfeb bytes) 000327 1717557054.408395 mapread 0x00c636 thru 0x00f2eb (0x2cb6 bytes) 000328 1717557054.408620 fallocate 0x033cb0 thru 0x03549e (0x17ef bytes) 000329 1717557054.409616 fallocate 0x00622d thru 0x00df66 (0x7d3a bytes) 000330 1717557054.410694 mapwrite 0x02a4b5 thru 0x031789 (0x72d5 bytes) 000331 1717557054.419885 fallocate 0x034c0c thru 0x03ffff (0xb3f4 bytes) 000332 1717557054.421171 read 0x02ee52 thru 0x0308c7 (0x1a76 bytes) 000333 1717557054.421387 read 0x005a95 thru 0x00ae14 (0x5380 bytes) 000334 1717557054.422776 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000335 1717557054.466388 MIRROR_EXTEND to 4 mirrors 000336 1717557054.553124 MIRROR_EXTEND to 5 mirrors 000337 1717557054.639687 trunc from 0x03178a to 0x03342f (0x1ca6 bytes) 000338 1717557054.650430 trunc from 0x033430 to 0x011afb (0x21934 bytes) 000339 1717557054.654315 read 0x00cdf4 thru 0x01000e (0x321b bytes) 000340 1717557054.656079 trunc from 0x011afc to 0x0326d4 (0x20bd9 bytes) 000341 1717557054.658849 trunc from 0x0326d5 to 0x01e9a3 (0x13d31 bytes) 000344 1717557054.661678 trunc from 0x01e9a4 to 0x031be0 (0x1323d bytes) 000345 1717557054.664197 read 0x0222ed thru 0x02e738 (0xc44c bytes) 000346 1717557054.666656 mapwrite 0x0288da thru 0x03269e (0x9dc5 bytes) 000347 1717557054.675464 fallocate 0x0020b8 thru 0x0022fa (0x243 bytes) 000348 1717557054.676736 mapwrite 0x0082db thru 0x014d56 (0xca7c bytes) 000349 1717557054.680882 fallocate 0x0336a1 thru 0x03d24f (0x9baf bytes) 000350 1717557054.682840 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000351 1717557054.725253 punch from 0x02a3a7 to 0x03269e (0x82f8 bytes) 000353 1717557054.727022 mapread 0x02b9dc thru 0x02c013 (0x638 bytes) 000355 1717557054.729584 fallocate 0x03737e thru 0x0384de (0x1161 bytes) 000356 1717557054.730640 mapwrite 0x000694 thru 0x00c161 (0xbace bytes) 000357 1717557054.737272 mapread 0x0058a9 thru 0x007998 (0x20f0 bytes) 000358 1717557054.737525 trunc from 0x03269f to 0x02b96e (0x6d30 bytes) 000359 1717557054.741569 trunc from 0x02b96f to 0x020a36 (0xaf38 bytes) 000360 1717557054.744508 fallocate 0x032c62 thru 0x038df7 (0x6196 bytes) 000361 1717557054.745814 mapwrite 0x01d31e thru 0x029114 (0xbdf7 bytes) 000362 1717557054.756963 mapread 0x00696f thru 0x00f66d (0x8cff bytes) 000365 1717557054.758600 MIRROR_EXTEND to 5 mirrors 000366 1717557054.845112 trunc from 0x029115 to 0x000521 (0x28bf3 bytes) 000367 1717557054.856827 mapread 0x000457 thru 0x000521 (0xcb bytes) 000368 1717557054.859188 MIRROR_EXTEND to 6 mirrors 000369 1717557054.921636 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000370 1717557054.969706 punch from 0x0003d8 to 0x000521 (0x14a bytes) 000371 1717557054.977735 MIRROR_RESYNC 5 mirrors 000372 1717557055.058987 punch from 0x000309 to 0x000521 (0x219 bytes) 000373 1717557055.066765 MIRROR_EXTEND to 6 mirrors 000374 1717557055.137163 read 0x000120 thru 0x000521 (0x402 bytes) 000375 1717557055.139740 MIRROR_EXTEND to 7 mirrors 000376 1717557055.204274 punch from 0x0001ce to 0x000521 (0x354 bytes) 000377 1717557055.212830 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000378 1717557055.254368 fallocate 0x02e6f2 thru 0x033d09 (0x5618 bytes) 000380 1717557055.255709 write 0x01e2ab thru 0x02aecf (0xcc25 bytes) 000382 1717557055.258828 trunc from 0x033d0a to 0x032596 (0x1773 bytes) 000383 1717557055.261778 trunc from 0x032597 to 0x031af1 (0xaa5 bytes) 000384 1717557055.264672 mapread 0x013516 thru 0x015a83 (0x256e bytes) 000385 1717557055.267988 MIRROR_EXTEND to 7 mirrors 000386 1717557055.368284 fallocate 0x03636d thru 0x03ffff (0x9c93 bytes) 000387 1717557055.374881 mapwrite 0x00eef1 thru 0x00ef95 (0xa5 bytes) 000388 1717557055.379133 punch from 0x02f38b to 0x0324cc (0x3142 bytes) 000389 1717557055.380190 trunc from 0x040000 to 0x005fc9 (0x3a036 bytes) 000390 1717557055.384521 write 0x026bfc thru 0x0304c8 (0x98cd bytes) 000391 1717557055.384962 read 0x01dc87 thru 0x02c76b (0xeae5 bytes) 000392 1717557055.386578 punch from 0x019533 to 0x01a162 (0xc30 bytes) 000393 1717557055.387515 fallocate 0x0094dd thru 0x015881 (0xc3a5 bytes) 000394 1717557055.388512 write 0x022007 thru 0x02a65f (0x8659 bytes) 000395 1717557055.388814 fallocate 0x01cc61 thru 0x01f1a8 (0x2548 bytes) 000396 1717557055.389736 trunc from 0x0304c9 to 0x03eb62 (0xe69a bytes) 000397 1717557055.392474 fallocate 0x0274c1 thru 0x02cd03 (0x5843 bytes) 000399 1717557055.393556 mapread 0x01f964 thru 0x022030 (0x26cd bytes) 000404 1717557055.394123 fallocate 0x039ab4 thru 0x03ffff (0x654c bytes) 000405 1717557055.395188 mapwrite 0x02d230 thru 0x039bde (0xc9af bytes) 000406 1717557055.403013 write 0x013f83 thru 0x020e32 (0xceb0 bytes) 000407 1717557055.405463 write 0x014d92 thru 0x01e0f8 (0x9367 bytes) 000408 1717557055.405848 trunc from 0x040000 to 0x01427f (0x2bd80 bytes) 000410 1717557055.409095 trunc from 0x014280 to 0x0294e6 (0x15267 bytes) 000411 1717557055.411869 fallocate 0x0296f3 thru 0x037c30 (0xe53e bytes) 000413 1717557055.413356 read 0x01b9eb thru 0x026625 (0xac3b bytes) 000414 1717557055.415849 read 0x00fff7 thru 0x01a2d1 (0xa2db bytes) 000415 1717557055.417052 MIRROR_RESYNC 7 mirrors 000416 1717557055.600408 trunc from 0x0294e7 to 0x0136df (0x15e07 bytes) 000420 1717557055.611626 read 0x009b52 thru 0x00dce3 (0x4192 bytes) 000421 1717557055.613537 mapread 0x00ea05 thru 0x012bdd (0x41d9 bytes) 000423 1717557055.617420 mapread 0x002e72 thru 0x00af0a (0x8099 bytes) 000424 1717557055.619260 fallocate 0x03731b thru 0x039293 (0x1f79 bytes) 000425 1717557055.620427 read 0x0021df thru 0x00d0b7 (0xaed9 bytes) 000426 1717557055.620606 write 0x019193 thru 0x01cb2f (0x399d bytes) 000427 1717557055.620870 mapread 0x017ba1 thru 0x01c402 (0x4862 bytes) 000428 1717557055.622750 trunc from 0x01cb30 to 0x017a88 (0x50a7 bytes) 000429 1717557055.626855 mapread 0x0110c3 thru 0x017a88 (0x69c6 bytes) 000430 1717557055.627173 fallocate 0x00dbb7 thru 0x010895 (0x2cdf bytes) 000431 1717557055.628159 fallocate 0x038507 thru 0x03ffff (0x7af9 bytes) 000432 1717557055.629338 mapread 0x030289 thru 0x032535 (0x22ad bytes) 000433 1717557055.631891 mapread 0x0240f7 thru 0x02f16b (0xb075 bytes) 000434 1717557055.632284 mapwrite 0x0183a1 thru 0x01f349 (0x6fa9 bytes) 000435 1717557055.641830 trunc from 0x040000 to 0x014ca5 (0x2b35a bytes) 000436 1717557055.645315 trunc from 0x014ca6 to 0x033278 (0x1e5d3 bytes) 000437 1717557055.648087 punch from 0x02f745 to 0x033278 (0x3b34 bytes) 000438 1717557055.649060 mapwrite 0x01ebb3 thru 0x026663 (0x7ab1 bytes) 000439 1717557055.664190 MIRROR_RESYNC 7 mirrors 000440 1717557055.896519 write 0x000bcb thru 0x008b03 (0x7f39 bytes) 000441 1717557055.903809 fallocate 0x01bf34 thru 0x01c475 (0x542 bytes) 000442 1717557055.904914 fallocate 0x01376a thru 0x016296 (0x2b2d bytes) 000443 1717557055.905990 fallocate 0x036a38 thru 0x03ffff (0x95c8 bytes) 000444 1717557055.908016 MIRROR_EXTEND to 8 mirrors 000445 1717557056.004765 MIRROR_RESYNC 8 mirrors 000446 1717557056.210556 mapwrite 0x015a76 thru 0x017906 (0x1e91 bytes) 000450 1717557056.237808 trunc from 0x040000 to 0x032c9d (0xd362 bytes) 000451 1717557056.245352 mapread 0x0088b3 thru 0x00b34d (0x2a9b bytes) 000452 1717557056.248143 trunc from 0x032c9e to 0x00f91b (0x23382 bytes) 000453 1717557056.252514 trunc from 0x00f91c to 0x02616d (0x16852 bytes) 000455 1717557056.257311 MIRROR_EXTEND to 9 mirrors 000456 1717557056.361053 mapread 0x0217dd thru 0x02616d (0x4991 bytes) 000457 1717557056.366945 mapwrite 0x03829b thru 0x03eae7 (0x684d bytes) 000458 1717557056.383049 punch from 0x00a6f3 to 0x010e65 (0x6773 bytes) 000460 1717557056.384503 trunc from 0x03eae8 to 0x01d390 (0x21757 bytes) 000463 1717557056.388967 mapwrite 0x036388 thru 0x03bbf5 (0x586e bytes) 000464 1717557056.400211 write 0x0099a4 thru 0x010161 (0x67be bytes) 000465 1717557056.402523 write 0x03d0f0 thru 0x03f8c9 (0x27da bytes) 000466 1717557056.402874 fallocate 0x010369 thru 0x01ea69 (0xe701 bytes) 000468 1717557056.404387 mapwrite 0x027fd1 thru 0x037a2e (0xfa5e bytes) 000469 1717557056.412038 punch from 0x002622 to 0x00d1ff (0xabde bytes) 000470 1717557056.413612 mapread 0x017fce thru 0x02396f (0xb9a2 bytes) 000471 1717557056.420003 mapwrite 0x00fbc2 thru 0x01f3cc (0xf80b bytes) 000473 1717557056.427509 punch from 0x0161d0 to 0x01f5ee (0x941f bytes) 000474 1717557056.428938 trunc from 0x03f8ca to 0x013640 (0x2c289 bytes) 000475 1717557056.432573 punch from 0x00d04b to 0x013640 (0x65f6 bytes) 000476 1717557056.433714 fallocate 0x03c562 thru 0x03ffff (0x3a9e bytes) 000477 1717557056.434613 trunc from 0x013641 to 0x037917 (0x242d7 bytes) 000478 1717557056.437406 mapread 0x016014 thru 0x01abd0 (0x4bbd bytes) 000479 1717557056.440009 mapwrite 0x02a634 thru 0x03261e (0x7feb bytes) 000481 1717557056.446967 read 0x0280f8 thru 0x02a11f (0x2028 bytes) 000482 1717557056.447233 read 0x031b6e thru 0x037917 (0x5daa bytes) 000483 1717557056.449540 mapread 0x0268ce thru 0x031c67 (0xb39a bytes) 000484 1717557056.451006 MIRROR_RESYNC 9 mirrors 000485 1717557056.846166 mapread 0x01e65f thru 0x0253fb (0x6d9d bytes) 000486 1717557056.850555 mapwrite 0x03c675 thru 0x03ffff (0x398b bytes) 000487 1717557056.869826 trunc from 0x040000 to 0x029ad8 (0x16527 bytes) 000489 1717557056.874599 mapread 0x007635 thru 0x01281d (0xb1e9 bytes) 000490 1717557056.878775 fallocate 0x01572e thru 0x024d98 (0xf66b bytes) 000493 1717557056.880239 punch from 0x00c642 to 0x01c22e (0xfbed bytes) 000494 1717557056.881467 read 0x00845a thru 0x014f88 (0xcb2f bytes) 000495 1717557056.883287 read 0x016a63 thru 0x026731 (0xfccf bytes) 000496 1717557056.886623 MIRROR_RESYNC 9 mirrors 000497 1717557057.121899 punch from 0x0092bd to 0x00e54d (0x5291 bytes) 000498 1717557057.129688 write 0x03148a thru 0x034f92 (0x3b09 bytes) 000499 1717557057.131191 MIRROR_EXTEND to 10 mirrors 000500 1717557057.236210 trunc from 0x034f93 to 0x028577 (0xca1b bytes) 000503 1717557057.246056 mapread 0x0259f7 thru 0x028577 (0x2b81 bytes) 000505 1717557057.251486 MIRROR_EXTEND to 11 mirrors 000506 1717557057.339732 write 0x01ca7e thru 0x02a7eb (0xdd6e bytes) 000507 1717557057.348281 read 0x019ba1 thru 0x023011 (0x9471 bytes) 000508 1717557057.349560 mapread 0x02958c thru 0x02a7eb (0x1260 bytes) 000510 1717557057.349967 fallocate 0x019ca6 thru 0x02304a (0x93a5 bytes) 000511 1717557057.351992 MIRROR_EXTEND to 12 mirrors 000512 1717557057.443675 fallocate 0x02c947 thru 0x037aaf (0xb169 bytes) 000513 1717557057.453014 read 0x0148fb thru 0x01f5df (0xace5 bytes) 000514 1717557057.454961 trunc from 0x02a7ec to 0x0050e5 (0x25706 bytes) 000516 1717557057.463617 read 0x004e5b thru 0x0050e5 (0x28b bytes) 000517 1717557057.465415 punch from 0x000c05 to 0x0050e5 (0x44e1 bytes) 000518 1717557057.466645 read 0x000398 thru 0x0050e5 (0x4d4e bytes) 000519 1717557057.468592 punch from 0x004bb9 to 0x0050e5 (0x52d bytes) 000521 1717557057.470696 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000522 1717557057.520472 write 0x0236f7 thru 0x0241b6 (0xac0 bytes) 000523 1717557057.522681 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000525 1717557057.567135 MIRROR_RESYNC 10 mirrors 000527 1717557057.753126 read 0x0018d4 thru 0x00781d (0x5f4a bytes) 000528 1717557057.755829 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000529 1717557057.820841 mapread 0x016c4a thru 0x020268 (0x961f bytes) 000530 1717557057.826244 mapwrite 0x007d80 thru 0x00a0ff (0x2380 bytes) 000531 1717557057.855798 write 0x02b5bd thru 0x02d9ef (0x2433 bytes) 000532 1717557057.857416 mapwrite 0x007fdb thru 0x00e32e (0x6354 bytes) 000533 1717557057.861979 punch from 0x025bbf to 0x02d9ef (0x7e31 bytes) 000534 1717557057.863261 mapread 0x015876 thru 0x01d747 (0x7ed2 bytes) 000536 1717557057.865710 fallocate 0x030e84 thru 0x034a31 (0x3bae bytes) 000537 1717557057.866920 write 0x01d0b5 thru 0x028b7f (0xbacb bytes) 000538 1717557057.868532 fallocate 0x02718a thru 0x02d207 (0x607e bytes) 000539 1717557057.869878 write 0x00763b thru 0x007f58 (0x91e bytes) 000540 1717557057.870212 trunc from 0x02d9f0 to 0x01dfe9 (0xfa06 bytes) 000541 1717557057.877286 trunc from 0x01dfea to 0x028162 (0xa179 bytes) 000542 1717557057.879911 write 0x0115e2 thru 0x0206ed (0xf10c bytes) 000543 1717557057.881869 trunc from 0x028163 to 0x037b8b (0xfa29 bytes) 000544 1717557057.884788 mapread 0x015358 thru 0x022888 (0xd531 bytes) 000545 1717557057.885060 punch from 0x00f32c to 0x017851 (0x8526 bytes) 000548 1717557057.888321 fallocate 0x024339 thru 0x02ec41 (0xa909 bytes) 000549 1717557057.889420 read 0x02801e thru 0x029177 (0x115a bytes) 000550 1717557057.890726 mapread 0x032f0f thru 0x037b8b (0x4c7d bytes) 000551 1717557057.892577 fallocate 0x01a38d thru 0x0258c8 (0xb53c bytes) 000553 1717557057.893781 trunc from 0x037b8c to 0x005222 (0x32969 bytes) 000554 1717557057.897830 MIRROR_EXTEND to 10 mirrors 000555 1717557057.981140 MIRROR_EXTEND to 11 mirrors 000556 1717557058.065066 read 0x000a87 thru 0x005222 (0x479c bytes) 000557 1717557058.066841 mapread 0x002df2 thru 0x005222 (0x2431 bytes) 000558 1717557058.067159 trunc from 0x005223 to 0x027e00 (0x22bde bytes) 000559 1717557058.079776 fallocate 0x0249f1 thru 0x02f234 (0xa844 bytes) 000560 1717557058.080970 trunc from 0x027e01 to 0x00df7f (0x19e81 bytes) 000561 1717557058.084632 read 0x00a969 thru 0x00df7f (0x3617 bytes) 000563 1717557058.086898 read 0x00b110 thru 0x00cc08 (0x1af9 bytes) 000566 1717557058.087290 mapread 0x0090f8 thru 0x00df7f (0x4e88 bytes) 000567 1717557058.087535 fallocate 0x02e820 thru 0x03e180 (0xf961 bytes) 000569 1717557058.088851 read 0x004ea2 thru 0x00f2ff (0xa45e bytes) 000571 1717557058.091558 mapread 0x034bb2 thru 0x03e180 (0x95cf bytes) 000572 1717557058.091815 punch from 0x02cd9a to 0x02fc70 (0x2ed7 bytes) 000573 1717557058.092952 write 0x01e598 thru 0x0232a5 (0x4d0e bytes) 000574 1717557058.094598 MIRROR_RESYNC 11 mirrors 000576 1717557058.388246 MIRROR_EXTEND to 12 mirrors 000577 1717557058.498137 punch from 0x02ce55 to 0x037284 (0xa430 bytes) 000578 1717557058.505484 mapwrite 0x028b9c thru 0x0300fd (0x7562 bytes) 000579 1717557058.513801 trunc from 0x03e181 to 0x014c3c (0x29544 bytes) 000580 1717557058.520880 write 0x03d4e3 thru 0x03ffff (0x2b1d bytes) 000581 1717557058.522116 MIRROR_RESYNC 12 mirrors 000582 1717557058.696900 mapread 0x012b77 thru 0x015446 (0x28d0 bytes) 000583 1717557058.700789 mapread 0x013b48 thru 0x021c82 (0xe13b bytes) 000584 1717557058.702685 mapwrite 0x0347b2 thru 0x03ebb6 (0xa405 bytes) 000585 1717557058.730868 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000587 1717557058.776914 write 0x0113cd thru 0x01bb26 (0xa75a bytes) 000588 1717557058.779877 mapwrite 0x0196db thru 0x021165 (0x7a8b bytes) 000589 1717557058.786210 mapwrite 0x03269f thru 0x03366c (0xfce bytes) 000590 1717557058.790498 fallocate 0x01585f thru 0x01b3b0 (0x5b52 bytes) 000591 1717557058.792457 MIRROR_EXTEND to 12 mirrors 000592 1717557058.885482 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000593 1717557058.948877 trunc from 0x040000 to 0x0244a9 (0x1bb56 bytes) 000594 1717557058.961510 mapwrite 0x02df0d thru 0x030ef3 (0x2fe7 bytes) 000596 1717557058.972816 mapwrite 0x011170 thru 0x0193e4 (0x8275 bytes) 000597 1717557058.980685 mapread 0x01ef64 thru 0x02363e (0x46db bytes) 000599 1717557058.983318 MIRROR_RESYNC 11 mirrors 000600 1717557059.251123 trunc from 0x030ef4 to 0x0318bc (0x9c9 bytes) 000601 1717557059.258583 mapread 0x0169bb thru 0x0251d7 (0xe81d bytes) 000602 1717557059.263881 punch from 0x0226c9 to 0x02bc46 (0x957e bytes) 000603 1717557059.264935 read 0x00d492 thru 0x010063 (0x2bd2 bytes) 000604 1717557059.265993 read 0x017af9 thru 0x020dd5 (0x92dd bytes) 000605 1717557059.267034 MIRROR_RESYNC 11 mirrors 000606 1717557059.585694 mapread 0x02cd4c thru 0x0318bc (0x4b71 bytes) 000607 1717557059.589342 trunc from 0x0318bd to 0x0088fa (0x28fc2 bytes) 000608 1717557059.601238 mapread 0x0017e1 thru 0x0088fa (0x711a bytes) 000609 1717557059.604324 MIRROR_EXTEND to 12 mirrors 000610 1717557059.683892 trunc from 0x0088fb to 0x01ab95 (0x1229b bytes) 000611 1717557059.695820 read 0x00e6cf thru 0x014ca1 (0x65d3 bytes) 000612 1717557059.697524 punch from 0x0119f3 to 0x01ab95 (0x91a3 bytes) 000613 1717557059.698625 trunc from 0x01ab96 to 0x0157bb (0x53da bytes) 000614 1717557059.702225 mapwrite 0x025852 thru 0x030cfa (0xb4a9 bytes) 000615 1717557059.715198 fallocate 0x03d693 thru 0x03ffff (0x296d bytes) 000616 1717557059.716399 trunc from 0x040000 to 0x007fb2 (0x3804d bytes) 000617 1717557059.719624 mapread 0x0032c1 thru 0x007fb2 (0x4cf2 bytes) 000618 1717557059.721409 punch from 0x002bc0 to 0x007fb2 (0x53f3 bytes) 000619 1717557059.722412 fallocate 0x013f89 thru 0x020460 (0xc4d8 bytes) 000621 1717557059.723757 trunc from 0x007fb3 to 0x03342b (0x2b479 bytes) 000622 1717557059.726818 read 0x02294c thru 0x02f3f4 (0xcaa9 bytes) 000624 1717557059.729762 trunc from 0x03342c to 0x01168d (0x21d9e bytes) 000627 1717557059.733027 mapwrite 0x017d3b thru 0x01ef4e (0x7214 bytes) 000628 1717557059.747530 trunc from 0x01ef4f to 0x01501f (0x9f2f bytes) 000629 1717557059.750202 mapwrite 0x01a028 thru 0x01d180 (0x3159 bytes) 000630 1717557059.760601 MIRROR_RESYNC 12 mirrors 000631 1717557059.961339 write 0x0050b3 thru 0x006eb3 (0x1e01 bytes) 000633 1717557059.968837 mapread 0x0068c0 thru 0x00a6e5 (0x3e26 bytes) 000634 1717557059.970786 mapwrite 0x030db5 thru 0x032ed0 (0x211c bytes) 000635 1717557059.982735 write 0x00a2e2 thru 0x00fb44 (0x5863 bytes) 000636 1717557059.983026 mapread 0x01c4ba thru 0x01d860 (0x13a7 bytes) 000637 1717557059.985130 mapread 0x00d168 thru 0x014fb8 (0x7e51 bytes) 000638 1717557059.986295 mapread 0x00f6f9 thru 0x016615 (0x6f1d bytes) 000640 1717557059.986628 trunc from 0x032ed1 to 0x01b7c6 (0x1770a bytes) 000641 1717557059.989681 read 0x002619 thru 0x00a70b (0x80f3 bytes) 000642 1717557059.990845 write 0x025ad2 thru 0x03155c (0xba8b bytes) 000644 1717557059.992311 MIRROR_RESYNC 12 mirrors 000646 1717557060.415510 fallocate 0x02f108 thru 0x03ed8c (0xfc85 bytes) 000647 1717557060.422365 mapread 0x02416e thru 0x030612 (0xc4a5 bytes) 000648 1717557060.427805 read 0x033cfa thru 0x03adf1 (0x70f8 bytes) 000650 1717557060.429156 punch from 0x01f5cb to 0x024341 (0x4d77 bytes) 000651 1717557060.430137 read 0x00f330 thru 0x01424c (0x4f1d bytes) 000656 1717557060.432932 trunc from 0x03ed8d to 0x030f41 (0xde4b bytes) 000658 1717557060.436965 read 0x0078d5 thru 0x00ce22 (0x554e bytes) 000660 1717557060.437334 trunc from 0x030f42 to 0x023f55 (0xcfec bytes) 000661 1717557060.440545 trunc from 0x023f56 to 0x03a995 (0x16a40 bytes) 000663 1717557060.443616 trunc from 0x03a996 to 0x00d858 (0x2d13d bytes) 000664 1717557060.447360 MIRROR_EXTEND to 13 mirrors 000665 1717557060.527836 read 0x00430d thru 0x00b579 (0x726d bytes) 000666 1717557060.529774 trunc from 0x00d859 to 0x035cb1 (0x28459 bytes) 000668 1717557060.541916 mapwrite 0x01559d thru 0x0161a5 (0xc09 bytes) 000670 1717557060.547227 write 0x011608 thru 0x013584 (0x1f7d bytes) 000672 1717557060.549165 punch from 0x01b2c2 to 0x01c10c (0xe4b bytes) 000673 1717557060.550170 mapread 0x01fe16 thru 0x02bea6 (0xc091 bytes) 000674 1717557060.553844 read 0x010474 thru 0x01b65f (0xb1ec bytes) 000675 1717557060.556660 MIRROR_EXTEND to 14 mirrors 000676 1717557060.664947 mapwrite 0x03048f thru 0x03bcf9 (0xb86b bytes) 000677 1717557060.687183 mapread 0x034e42 thru 0x03af92 (0x6151 bytes) 000678 1717557060.687530 punch from 0x011bc7 to 0x01cd86 (0xb1c0 bytes) 000679 1717557060.688625 read 0x02dba9 thru 0x02fe42 (0x229a bytes) 000680 1717557060.688849 trunc from 0x03bcfa to 0x021799 (0x1a560 bytes) 000681 1717557060.692584 read 0x007d8a thru 0x010743 (0x89ba bytes) 000682 1717557060.694812 mapread 0x012498 thru 0x01f879 (0xd3e2 bytes) 000684 1717557060.696245 MIRROR_EXTEND to 15 mirrors 000685 1717557060.788681 MIRROR_EXTEND to 16 mirrors 000686 1717557060.880081 write 0x014885 thru 0x02069b (0xbe17 bytes) 000688 1717557060.889175 trunc from 0x02179a to 0x030725 (0xef8c bytes) 000689 1717557060.895622 mapread 0x022af1 thru 0x02d32f (0xa83f bytes) 000690 1717557060.898873 trunc from 0x030726 to 0x0284ec (0x8239 bytes) 000691 1717557060.901466 mapread 0x01f201 thru 0x0284ec (0x92ec bytes) 000692 1717557060.901876 write 0x01e4c7 thru 0x02b060 (0xcb9a bytes) 000694 1717557060.902491 fallocate 0x00f5f9 thru 0x011a6a (0x2472 bytes) 000695 1717557060.904536 MIRROR_RESYNC 16 mirrors 000696 1717557061.456678 MIRROR_RESYNC 16 mirrors 000697 1717557061.519821 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000698 1717557061.598745 write 0x004a2f thru 0x012f0e (0xe4e0 bytes) 000699 1717557061.607609 mapwrite 0x026454 thru 0x02be36 (0x59e3 bytes) 000701 1717557061.618134 fallocate 0x0102f4 thru 0x01ede7 (0xeaf4 bytes) 000702 1717557061.619232 fallocate 0x03ecab thru 0x03ffff (0x1355 bytes) 000703 1717557061.621389 MIRROR_EXTEND to 16 mirrors 000705 1717557061.718176 read 0x0047bf thru 0x00e957 (0xa199 bytes) 000706 1717557061.722231 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000707 1717557061.787951 fallocate 0x005cb5 thru 0x00b099 (0x53e5 bytes) 000708 1717557061.796792 trunc from 0x040000 to 0x020585 (0x1fa7a bytes) 000709 1717557061.801454 mapwrite 0x01ea1c thru 0x02976a (0xad4f bytes) 000710 1717557061.814737 write 0x005fc3 thru 0x00aa9a (0x4ad8 bytes) 000711 1717557061.816847 fallocate 0x01498e thru 0x01a7c5 (0x5e38 bytes) 000712 1717557061.818624 MIRROR_EXTEND to 16 mirrors 000713 1717557061.914327 mapread 0x012bbe thru 0x020c51 (0xe094 bytes) 000714 1717557061.921816 trunc from 0x02976b to 0x012be9 (0x16b81 bytes) 000715 1717557061.932535 read 0x00cdad thru 0x00fcc5 (0x2f19 bytes) 000716 1717557061.933698 fallocate 0x0200cb thru 0x021869 (0x179f bytes) 000717 1717557061.935655 MIRROR_RESYNC 16 mirrors 000719 1717557062.289088 fallocate 0x002788 thru 0x003552 (0xdcb bytes) 000720 1717557062.296700 mapwrite 0x001274 thru 0x007a6e (0x67fb bytes) 000721 1717557062.304356 mapwrite 0x00a73f thru 0x016704 (0xbfc6 bytes) 000723 1717557062.310815 trunc from 0x02186a to 0x000a56 (0x20e13 bytes) 000724 1717557062.314842 write 0x00bbcc thru 0x013093 (0x74c8 bytes) 000726 1717557062.315362 trunc from 0x013094 to 0x008fb9 (0xa0da bytes) 000728 1717557062.318554 read 0x00478c thru 0x005253 (0xac8 bytes) 000731 1717557062.319997 read 0x002d17 thru 0x007d74 (0x505e bytes) 000732 1717557062.321694 trunc from 0x008fba to 0x037cb2 (0x2ecf9 bytes) 000733 1717557062.324522 mapread 0x026ee7 thru 0x02cb53 (0x5c6d bytes) 000734 1717557062.326942 mapwrite 0x01b9b2 thru 0x028929 (0xcf78 bytes) 000736 1717557062.332114 mapwrite 0x027aaa thru 0x02f382 (0x78d9 bytes) 000737 1717557062.335915 mapread 0x00029b thru 0x009edb (0x9c41 bytes) 000738 1717557062.337131 MIRROR_RESYNC 16 mirrors 000739 1717557062.735176 trunc from 0x037cb3 to 0x00d88b (0x2a427 bytes) 000741 1717557062.745835 punch from 0x009164 to 0x00d88b (0x4728 bytes) 000742 1717557062.746845 mapread 0x0022a1 thru 0x008ffd (0x6d5d bytes) 000743 1717557062.750310 write 0x02f49b thru 0x03d5f3 (0xe159 bytes) 000744 1717557062.750955 mapread 0x009a8e thru 0x00e722 (0x4c95 bytes) 000745 1717557062.752639 write 0x02c32c thru 0x031792 (0x5467 bytes) 000748 1717557062.755457 MIRROR_RESYNC 16 mirrors 000749 1717557063.041713 write 0x016d5e thru 0x0193c1 (0x2664 bytes) 000750 1717557063.050360 punch from 0x0214ce to 0x02773f (0x6272 bytes) 000752 1717557063.051694 mapwrite 0x006324 thru 0x00b394 (0x5071 bytes) 000753 1717557063.058606 read 0x02a5c4 thru 0x02f057 (0x4a94 bytes) 000754 1717557063.060403 read 0x011337 thru 0x012613 (0x12dd bytes) 000755 1717557063.062900 write 0x03bb50 thru 0x03ffff (0x44b0 bytes) 000757 1717557063.063546 read 0x00c019 thru 0x00eb05 (0x2aed bytes) 000758 1717557063.063785 read 0x021d59 thru 0x02b77b (0x9a23 bytes) 000759 1717557063.064071 write 0x009d65 thru 0x010cdb (0x6f77 bytes) 000760 1717557063.064512 mapwrite 0x00ae83 thru 0x019620 (0xe79e bytes) 000761 1717557063.069818 read 0x00f076 thru 0x01cc99 (0xdc24 bytes) 000762 1717557063.071346 MIRROR_RESYNC 16 mirrors 000764 1717557063.422328 trunc from 0x040000 to 0x011a93 (0x2e56c bytes) 000765 1717557063.434421 write 0x03465b thru 0x035e0a (0x17b0 bytes) 000766 1717557063.434818 write 0x029d4b thru 0x035268 (0xb51e bytes) 000767 1717557063.436145 write 0x02eb9f thru 0x0336f3 (0x4b55 bytes) 000768 1717557063.436495 read 0x000f8c thru 0x003a01 (0x2a76 bytes) 000769 1717557063.437935 mapread 0x013e6d thru 0x020bf8 (0xcd8c bytes) 000772 1717557063.442632 write 0x014f4c thru 0x01debb (0x8f70 bytes) 000773 1717557063.442936 punch from 0x019bc5 to 0x024dbd (0xb1f9 bytes) 000774 1717557063.446866 mapread 0x00e25b thru 0x017b80 (0x9926 bytes) 000775 1717557063.449117 trunc from 0x035e0b to 0x00d7ad (0x2865d bytes) 000776 1717557063.453677 trunc from 0x00d7ae to 0x037ba8 (0x2a3fb bytes) 000777 1717557063.456654 mapwrite 0x019fdc thru 0x0241cd (0xa1f2 bytes) 000778 1717557063.466612 punch from 0x01bcdf to 0x01c5d1 (0x8f3 bytes) 000779 1717557063.467671 punch from 0x032493 to 0x03641b (0x3f89 bytes) 000780 1717557063.468851 trunc from 0x037ba9 to 0x0165ce (0x215da bytes) 000781 1717557063.472295 punch from 0x0003c8 to 0x00327c (0x2eb5 bytes) 000782 1717557063.473356 mapread 0x0121b8 thru 0x0165ce (0x4417 bytes) 000783 1717557063.473640 mapread 0x011b3b thru 0x016240 (0x4706 bytes) 000784 1717557063.474940 write 0x0349b5 thru 0x03ffff (0xb64b bytes) 000785 1717557063.475521 read 0x038865 thru 0x03ffff (0x779b bytes) 000786 1717557063.475742 punch from 0x03153b to 0x03daad (0xc573 bytes) 000787 1717557063.478681 mapwrite 0x037810 thru 0x03ec1e (0x740f bytes) 000788 1717557063.485974 fallocate 0x030ae6 thru 0x037aea (0x7005 bytes) 000789 1717557063.487158 fallocate 0x02689d thru 0x033b03 (0xd267 bytes) 000790 1717557063.488172 trunc from 0x040000 to 0x008823 (0x377dc bytes) 000793 1717557063.492748 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000795 1717557063.541493 fallocate 0x034372 thru 0x03b732 (0x73c1 bytes) 000796 1717557063.542847 mapwrite 0x032bf2 thru 0x03aa08 (0x7e17 bytes) 000797 1717557063.556489 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000798 1717557063.600896 mapread 0x0113c9 thru 0x0118bd (0x4f5 bytes) 000799 1717557063.602089 write 0x03b28a thru 0x03ffff (0x4d76 bytes) 000801 1717557063.603469 read 0x02fc66 thru 0x038bdc (0x8f77 bytes) 000802 1717557063.606386 MIRROR_RESYNC 14 mirrors 000803 1717557063.892870 mapwrite 0x024bfd thru 0x024ecd (0x2d1 bytes) 000804 1717557063.917800 trunc from 0x040000 to 0x011188 (0x2ee77 bytes) 000805 1717557063.926615 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000806 1717557063.969625 MIRROR_RESYNC 13 mirrors 000808 1717557064.251541 mapwrite 0x015d6b thru 0x01f516 (0x97ac bytes) 000809 1717557064.266840 read 0x013cee thru 0x018c3a (0x4f4d bytes) 000811 1717557064.267231 trunc from 0x01f517 to 0x0263b2 (0x6e9c bytes) 000813 1717557064.270500 mapread 0x0222a6 thru 0x0263b2 (0x410d bytes) 000814 1717557064.272217 punch from 0x01a90b to 0x0243d8 (0x9ace bytes) 000815 1717557064.274407 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000816 1717557064.317204 read 0x0004c9 thru 0x005bf2 (0x572a bytes) 000818 1717557064.319209 fallocate 0x029165 thru 0x029cd6 (0xb72 bytes) 000819 1717557064.321044 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000822 1717557064.362225 mapread 0x002527 thru 0x010aeb (0xe5c5 bytes) 000823 1717557064.367434 read 0x01dbf5 thru 0x02217d (0x4589 bytes) 000824 1717557064.369043 write 0x01cd73 thru 0x01cefd (0x18b bytes) 000826 1717557064.371475 punch from 0x022dde to 0x0263b2 (0x35d5 bytes) 000827 1717557064.373311 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000829 1717557064.419282 MIRROR_EXTEND to 11 mirrors 000830 1717557064.516653 fallocate 0x026fe4 thru 0x02ebc7 (0x7be4 bytes) 000831 1717557064.523268 mapread 0x004f4c thru 0x01143a (0xc4ef bytes) 000832 1717557064.528319 mapread 0x025e05 thru 0x02cf44 (0x7140 bytes) 000833 1717557064.530066 read 0x014617 thru 0x01b878 (0x7262 bytes) 000834 1717557064.531259 fallocate 0x021685 thru 0x02cc32 (0xb5ae bytes) 000835 1717557064.532332 read 0x027d9e thru 0x02ebc7 (0x6e2a bytes) 000836 1717557064.533491 MIRROR_EXTEND to 12 mirrors 000837 1717557064.631077 trunc from 0x02ebc8 to 0x026755 (0x8472 bytes) 000838 1717557064.641644 fallocate 0x00d2d3 thru 0x0134ee (0x621c bytes) 000839 1717557064.642717 fallocate 0x031dcc thru 0x036088 (0x42bd bytes) 000840 1717557064.643724 fallocate 0x030859 thru 0x03fa67 (0xf20f bytes) 000841 1717557064.644818 write 0x03448e thru 0x03ffff (0xbb72 bytes) 000842 1717557064.647144 MIRROR_RESYNC 12 mirrors 000843 1717557065.111525 mapwrite 0x0134f4 thru 0x01ab87 (0x7694 bytes) 000844 1717557065.138752 fallocate 0x008aa5 thru 0x00f46a (0x69c6 bytes) 000845 1717557065.139758 write 0x01f210 thru 0x02d2e2 (0xe0d3 bytes) 000846 1717557065.143431 fallocate 0x008916 thru 0x016201 (0xd8ec bytes) 000848 1717557065.144714 fallocate 0x019119 thru 0x01b309 (0x21f1 bytes) 000849 1717557065.145770 mapwrite 0x034896 thru 0x03e5c4 (0x9d2f bytes) 000850 1717557065.152931 trunc from 0x040000 to 0x005712 (0x3a8ed bytes) 000851 1717557065.157124 trunc from 0x005713 to 0x03c246 (0x36b34 bytes) 000853 1717557065.160213 write 0x0087b0 thru 0x0148e2 (0xc133 bytes) 000854 1717557065.164567 MIRROR_EXTEND to 13 mirrors 000855 1717557065.286130 fallocate 0x01fbf7 thru 0x02aee4 (0xb2ee bytes) 000856 1717557065.299198 fallocate 0x035fa9 thru 0x03ffff (0xa057 bytes) 000857 1717557065.300980 trunc from 0x03c247 to 0x033646 (0x8c00 bytes) 000858 1717557065.308005 mapwrite 0x013929 thru 0x01dd2c (0xa404 bytes) 000859 1717557065.322729 MIRROR_RESYNC 13 mirrors 000860 1717557065.643015 trunc from 0x033647 to 0x02d4b5 (0x6191 bytes) 000861 1717557065.655197 fallocate 0x036c2f thru 0x03ffff (0x93d1 bytes) 000862 1717557065.656403 mapread 0x00cd91 thru 0x00d168 (0x3d8 bytes) 000863 1717557065.658724 read 0x010421 thru 0x01bce5 (0xb8c5 bytes) 000864 1717557065.660773 fallocate 0x01b789 thru 0x020a41 (0x52b9 bytes) 000865 1717557065.661799 read 0x00fe51 thru 0x014b7b (0x4d2b bytes) 000866 1717557065.663973 write 0x00fd48 thru 0x01c8f8 (0xcbb1 bytes) 000867 1717557065.664514 trunc from 0x02d4b6 to 0x024d90 (0x8725 bytes) 000868 1717557065.668651 mapwrite 0x029ab8 thru 0x029f71 (0x4ba bytes) 000869 1717557065.677407 write 0x018ffd thru 0x01ac77 (0x1c7b bytes) 000870 1717557065.677670 trunc from 0x029f72 to 0x00851c (0x21a55 bytes) 000871 1717557065.681310 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000872 1717557065.725102 write 0x02cd1d thru 0x036924 (0x9c08 bytes) 000873 1717557065.726487 mapwrite 0x01fa72 thru 0x026568 (0x6af7 bytes) 000874 1717557065.734675 read 0x011033 thru 0x0164cd (0x549b bytes) 000875 1717557065.736213 punch from 0x005a5a to 0x00ca8c (0x7033 bytes) 000876 1717557065.737113 read 0x01fe5d thru 0x0232e5 (0x3489 bytes) 000878 1717557065.737346 mapread 0x0342ba thru 0x034f99 (0xce0 bytes) 000879 1717557065.737568 trunc from 0x036925 to 0x004fb4 (0x31970 bytes) 000880 1717557065.740920 punch from 0x004e62 to 0x004fb4 (0x153 bytes) 000881 1717557065.741912 mapwrite 0x03190f thru 0x0395b2 (0x7ca4 bytes) 000882 1717557065.752398 mapwrite 0x009c64 thru 0x010aa3 (0x6e40 bytes) 000883 1717557065.757103 mapread 0x011f97 thru 0x016fa7 (0x5011 bytes) 000884 1717557065.757465 punch from 0x0312a7 to 0x0395b2 (0x830c bytes) 000885 1717557065.758645 trunc from 0x0395b3 to 0x00ef9b (0x2a617 bytes) 000886 1717557065.762025 punch from 0x00d036 to 0x00ef9b (0x1f66 bytes) 000887 1717557065.763116 read 0x00d3f9 thru 0x00ef9b (0x1ba3 bytes) 000888 1717557065.765563 MIRROR_RESYNC 12 mirrors 000889 1717557065.969663 mapread 0x005ec3 thru 0x00ab0d (0x4c4b bytes) 000890 1717557065.972963 trunc from 0x00ef9c to 0x021f36 (0x12f9b bytes) 000891 1717557065.983700 punch from 0x00e941 to 0x017cc2 (0x9382 bytes) 000892 1717557065.984766 punch from 0x010f89 to 0x01238c (0x1404 bytes) 000894 1717557065.985898 write 0x00717f thru 0x01108f (0x9f11 bytes) 000895 1717557065.988866 fallocate 0x02abb7 thru 0x03a7aa (0xfbf4 bytes) 000896 1717557065.989971 read 0x01a223 thru 0x01d8eb (0x36c9 bytes) 000898 1717557065.991225 write 0x030bb4 thru 0x0381e7 (0x7634 bytes) 000899 1717557065.991636 mapread 0x01f2cd thru 0x02572a (0x645e bytes) 000901 1717557065.993792 fallocate 0x01da20 thru 0x02008e (0x266f bytes) 000902 1717557065.994853 write 0x027489 thru 0x02bbfb (0x4773 bytes) 000903 1717557065.996094 MIRROR_RESYNC 12 mirrors 000904 1717557066.376746 MIRROR_RESYNC 12 mirrors 000905 1717557066.434479 mapwrite 0x00b7d2 thru 0x01462b (0x8e5a bytes) 000906 1717557066.463348 write 0x018343 thru 0x0248ab (0xc569 bytes) 000907 1717557066.466635 punch from 0x034342 to 0x035801 (0x14c0 bytes) 000908 1717557066.467849 read 0x00976a thru 0x013fdc (0xa873 bytes) 000909 1717557066.468108 write 0x01ca2e thru 0x02c00a (0xf5dd bytes) 000910 1717557066.470637 MIRROR_RESYNC 12 mirrors 000911 1717557066.784473 read 0x0150c7 thru 0x022f64 (0xde9e bytes) 000912 1717557066.786530 mapread 0x018739 thru 0x0264a9 (0xdd71 bytes) 000914 1717557066.790520 trunc from 0x0381e8 to 0x0075e7 (0x30c00 bytes) 000915 1717557066.802084 mapwrite 0x023aa9 thru 0x02e397 (0xa8ef bytes) 000916 1717557066.814144 write 0x031087 thru 0x0345eb (0x3565 bytes) 000917 1717557066.814554 read 0x02a7d3 thru 0x0345eb (0x9e19 bytes) 000918 1717557066.816710 trunc from 0x0345ec to 0x019bbd (0x1aa2e bytes) 000919 1717557066.819829 fallocate 0x037d8a thru 0x03f880 (0x7af7 bytes) 000921 1717557066.822369 MIRROR_RESYNC 12 mirrors 000922 1717557067.117728 fallocate 0x01a2ce thru 0x029de7 (0xfb1a bytes) 000923 1717557067.125336 trunc from 0x03f881 to 0x014b1c (0x2ad64 bytes) 000924 1717557067.131268 MIRROR_RESYNC 12 mirrors 000926 1717557067.406843 mapwrite 0x03b0e7 thru 0x03ffff (0x4f19 bytes) 000927 1717557067.421746 punch from 0x028e55 to 0x02bbf7 (0x2da3 bytes) 000928 1717557067.423452 MIRROR_RESYNC 12 mirrors 000929 1717557067.679585 fallocate 0x0391fd thru 0x03c52f (0x3333 bytes) 000930 1717557067.688690 write 0x006bf2 thru 0x00c05f (0x546e bytes) 000931 1717557067.692055 trunc from 0x040000 to 0x01ad8b (0x25274 bytes) 000933 1717557067.699650 mapwrite 0x03c9e7 thru 0x03ffff (0x3619 bytes) 000934 1717557067.709118 fallocate 0x02f8f1 thru 0x031dab (0x24bb bytes) 000935 1717557067.710295 trunc from 0x040000 to 0x01cecc (0x23133 bytes) 000936 1717557067.713421 trunc from 0x01cecd to 0x01d249 (0x37d bytes) 000937 1717557067.716958 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x948:0x0]: Invalid argument (22) 000939 1717557067.764098 read 0x01adf3 thru 0x01d249 (0x2457 bytes) 000940 1717557067.765657 read 0x007ea3 thru 0x014fe2 (0xd140 bytes) 000941 1717557067.767875 punch from 0x018529 to 0x01b9c3 (0x349b bytes) 000942 1717557067.769842 MIRROR_RESYNC 11 mirrors 000943 1717557068.038766 read 0x00d7a5 thru 0x012f48 (0x57a4 bytes) 000944 1717557068.040603 mapread 0x010d4c thru 0x01adb7 (0xa06c bytes) 000945 1717557068.044057 punch from 0x00289b to 0x00c4aa (0x9c10 bytes) 000946 1717557068.051601 mapread 0x01ae6f thru 0x01d249 (0x23db bytes) 000949 1717557068.054016 mapread 0x002ebf thru 0x00b21f (0x8361 bytes) 000950 1717557068.057110 write 0x02e60b thru 0x0348da (0x62d0 bytes) 000951 1717557068.057480 mapread 0x01b3ab thru 0x027e13 (0xca69 bytes) 000952 1717557068.060175 fallocate 0x034b6a thru 0x03f347 (0xa7de bytes) 000953 1717557068.062550 MIRROR_RESYNC 11 mirrors 000954 1717557068.379229 mapread 0x00f5ca thru 0x01e558 (0xef8f bytes) 000955 1717557068.384321 read 0x0222aa thru 0x02cf45 (0xac9c bytes) 000958 1717557068.386055 punch from 0x03eea7 to 0x03f347 (0x4a1 bytes) 000959 1717557068.394375 mapwrite 0x005463 thru 0x006dce (0x196c bytes) 000960 1717557068.399957 trunc from 0x03f348 to 0x03bce9 (0x365e bytes) 000961 1717557068.406047 fallocate 0x015e5e thru 0x01f88e (0x9a31 bytes) 000962 1717557068.406949 read 0x0134ed thru 0x014255 (0xd69 bytes) 000963 1717557068.409266 mapwrite 0x00783c thru 0x01214f (0xa914 bytes) 000965 1717557068.414374 mapread 0x02813b thru 0x02fe0a (0x7cd0 bytes) 000966 1717557068.414751 write 0x03d4d1 thru 0x03ffff (0x2b2f bytes) 000968 1717557068.415354 mapwrite 0x00122b thru 0x002c90 (0x1a66 bytes) 000969 1717557068.418643 mapwrite 0x01d99b thru 0x01db29 (0x18f bytes) 000970 1717557068.421266 trunc from 0x040000 to 0x01ddee (0x22211 bytes) 000971 1717557068.424550 fallocate 0x0351b8 thru 0x03cfb8 (0x7e01 bytes) 000973 1717557068.425764 mapread 0x023244 thru 0x025d75 (0x2b32 bytes) 000975 1717557068.429314 punch from 0x032035 to 0x0333ec (0x13b8 bytes) 000976 1717557068.430379 punch from 0x022534 to 0x02724e (0x4d1b bytes) 000977 1717557068.431274 trunc from 0x03cfb9 to 0x00c265 (0x30d53 bytes) 000980 1717557068.434436 mapwrite 0x01e87f thru 0x026909 (0x808b bytes) 000981 1717557068.450083 punch from 0x02604e to 0x026718 (0x6cb bytes) 000982 1717557068.451362 mapwrite 0x03d4e6 thru 0x03ffff (0x2b1a bytes) 000983 1717557068.459621 read 0x02da06 thru 0x034130 (0x672b bytes) 000985 1717557068.463350 MIRROR_EXTEND to 12 mirrors 000986 1717557068.561561 read 0x000ffb thru 0x00636c (0x5372 bytes) 000988 1717557068.566466 trunc from 0x040000 to 0x02963c (0x169c3 bytes) 000989 1717557068.579755 mapread 0x0017b5 thru 0x008df8 (0x7644 bytes) 000990 1717557068.583580 trunc from 0x02963d to 0x023975 (0x5cc7 bytes) 000991 1717557068.587650 mapwrite 0x015d07 thru 0x01f219 (0x9513 bytes) 000993 1717557068.596312 trunc from 0x023976 to 0x00f866 (0x1410f bytes) 000994 1717557068.599880 punch from 0x006a8b to 0x00c657 (0x5bcd bytes) 000996 1717557068.601045 punch from 0x00eaba to 0x00f866 (0xdad bytes) 000998 1717557068.602041 mapread 0x001f12 thru 0x005c85 (0x3d74 bytes) 000999 1717557068.602319 punch from 0x002f0c to 0x009afc (0x6bf1 bytes) All operations completed A-OK! PASS 70a (19s) == sanity-flr test 200a: stress test ===================== 23:11:09 (1717557069) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6338144 .. Extending file size to 5018336 .. failed Extending file size to 6607104 .. Extending file size to 8037792 .. Extending file size to 7406336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9043040 .. Extending file size to 8416032 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9091040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8578912 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9071232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9113440 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9263360 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9264544 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 7634 7635 7637 7639 7641 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:12:13 (1717557133) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4278272 .. Extending file size to 3911424 .. failed Extending file size to 7296704 .. Extending file size to 8926720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7759424 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8051584 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8053472 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8295040 .. failed Extending file size to 8956288 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8304480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8377888 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8858336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8970720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9028736 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9124480 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9148352 .. Extending file size to 9191456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9249088 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9368576 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 10959 10960 10962 10964 10966 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (62s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:13:15 (1717557195) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2616 1285072 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3760 1283928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15940 3584320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22104 3584916 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38044 7169236 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2616 1285072 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3760 1283928 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15940 3584320 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22104 3584916 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38044 7169236 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.023797 s, 88.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0428378 s, 49.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:13:19 (1717557199) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0157773 s, 133 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 23:13:20 (1717557200) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:13:22 (1717557202) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0130601 s, 161 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1277:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1278:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:13:23 (1717557203) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0144437 s, 145 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:13:25 (1717557205) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0169667 s, 124 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:13:48 (1717557228) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00728438 s, 144 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x127a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x127a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1279:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x127a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x127a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1279:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x127b:0x0] } PASS 204d (31s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:19 (1717557259) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0124541 s, 168 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:14:52 (1717557292) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg147-server: Pool lustre.test_204f created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0132993 s, 158 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg147-server: Pool lustre.test_204f destroyed PASS 204f (42s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 23:15:34 (1717557334) PASS 205 (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:15:36 (1717557336) oleg147-server: Pool lustre.test_206 created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg147-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:15:50 (1717557350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012281 s, 85.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00585699 s, 179 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x6733a774 Mirror 3: 0x6733a774 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:15:52 (1717557352) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:15:53 (1717557353) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:15:54 (1717557354) Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0070526 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1090000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ab0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ab0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ab0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1960000 Read this (4 bytes): BBBB 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:16:07 (1717557367) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00156875 s, 5.2 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:16:08 (1717557368) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00588588 s, 178 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-06-04 23:16:09.000000000 -0400 Modify: 2024-06-04 23:16:09.000000000 -0400 Change: 2024-06-04 23:16:09.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:16:10 (1717557370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00419774 s, 250 MB/s oleg147-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg147-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg147-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg147-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg147-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (9s) == sanity-flr test complete, duration 693 sec ============ 23:16:19 (1717557379) === sanity-flr: start cleanup 23:16:20 (1717557380) === === sanity-flr: finish cleanup 23:16:20 (1717557380) ===