-----============= acceptance-small: sanity-flr ============----- Fri May 9 07:48:28 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg307-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:48:33 (1746791313) === oleg307-client.virtnet: executing check_config_client /mnt/lustre oleg307-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg307-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5995000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5995000.idle_timeout=debug disable quota as required oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:48:42 (1746791322) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 07:48:42 (1746791322) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:48:45 (1746791325) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:48:46 (1746791326) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:48:47 (1746791327) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:48:51 (1746791331) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:48:51 (1746791331) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:48:52 (1746791332) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5995000.stats=clear osc.lustre-OST0001-osc-ffff8800b5995000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:48:54 (1746791334) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:48:56 (1746791336) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:48:57 (1746791337) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:48:58 (1746791338) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 07:49:00 (1746791340) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:49:01 (1746791341) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:49:03 (1746791343) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:49:05 (1746791345) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:49:07 (1746791347) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:49:09 (1746791349) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.346651 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255948 s, 41.0 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:49:11 (1746791351) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0276696 s, 114 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:49:13 (1746791353) Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:49:38 (1746791378) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0371928 s, 56.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:49:46 (1746791386) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:50:07 (1746791407) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.722942 s, 72.5 MB/s -rw-r--r-- 1 root root 52428800 May 9 07:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.200s user 0m0.000s sys 0m0.092s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.200s user 0m0.000s sys 0m0.094s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.232s user 0m0.000s sys 0m0.101s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg307-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5995000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:50:28 (1746791428) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:50:29 (1746791429) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:50:30 (1746791430) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 07:50:31 (1746791431) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 07:50:32 (1746791432) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.117991 s, 107 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:50:34 (1746791434) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00958498 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0088662 s, 118 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:50:36 (1746791436) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:50:38 (1746791438) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000579714 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000525269 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000400146 s, 2.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000527243 s, 1.9 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 07:50:40 (1746791440) create test files with size 15749278 .. acbd34fa9449558cbaa8b6eb9129c286 - 8108d60562319f9c2e71365e291dd01e - f5dbde99193b0094e2133ccf301e1cb5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 07:50:44 (1746791444) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:50:47 (1746791447) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:50:48 (1746791448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165712 s, 63.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00949658 s, 110 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 07:50:50 (1746791450) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.017579 s, 119 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0697144 s, 60.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0321717 s, 65.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 07:50:52 (1746791452) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 07:50:53 (1746791453) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:50:53 (1746791453) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101401 s, 103 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 07:50:55 (1746791455) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0661697 s, 47.5 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:50:57 (1746791457) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:50:59 (1746791459) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0637853 s, 164 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:51:22 (1746791482) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b5a4c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0581223 s, 180 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b5990800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:51:25 (1746791485) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:51:33 (1746791493) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:51:39 (1746791499) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 07:51:39 (1746791499) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0606848 s, 51.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0467973 s, 67.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 48d9870966417328b297263a20f950e1 - mirror 1 checksum: 8d192c4f3ddf8bd83b7f97be618291da - mirror 2 checksum: 48d9870966417328b297263a20f950e1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 9 07:51:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 9 07:51:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 48d9870966417328b297263a20f950e1 - mirror 1 checksum: 8d192c4f3ddf8bd83b7f97be618291da - mirror 2 checksum: 8d192c4f3ddf8bd83b7f97be618291da - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 9 07:51:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 9 07:51:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:51:41 (1746791501) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948769 s, 111 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:51:43 (1746791503) Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server 07:51:49 (1746791509) shut down facet: mds1 facet_host: oleg307-server facet_failover_host: oleg307-server Failover mds1 to oleg307-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 07:52:02 (1746791522) targets are mounted 07:52:02 (1746791522) facet_failover done oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0070956 s, 148 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (26s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:52:09 (1746791529) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00713954 s, 2.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00405628 s, 5.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:52:11 (1746791531) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00883352 s, 2.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00261442 s, 7.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:52:13 (1746791533) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00180638 s, 9.1 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:52:15 (1746791535) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00701004 s, 2.9 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:52:16 (1746791536) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00141743 s, 2.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00685787 s, 597 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:52:18 (1746791538) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:52:21 (1746791541) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:52:38 (1746791558) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:52:55 (1746791575) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:53:12 (1746791592) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 07:53:13 (1746791593) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26305 mirror split pid 26306 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26305 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26306 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 07:54:15 (1746791655) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1655 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read 000002 1746791655.594325 mapwrite 0x01c735 thru 0x021a93 (0x535f bytes) 000003 1746791655.609752 mapread 0x0017d1 thru 0x00949a (0x7cca bytes) 000004 1746791655.612998 mapread 0x009ef0 thru 0x016870 (0xc981 bytes) 000005 1746791655.614744 mapread 0x020901 thru 0x021a93 (0x1193 bytes) 000010 1746791655.616338 read 0x01e14b thru 0x021a93 (0x3949 bytes) 000012 1746791655.616780 mapread 0x00a627 thru 0x00e711 (0x40eb bytes) 000013 1746791655.617121 mapread 0x00aa9b thru 0x0106b0 (0x5c16 bytes) 000014 1746791655.617474 mapread 0x0207e6 thru 0x021a93 (0x12ae bytes) 000016 1746791655.617995 write 0x016dc9 thru 0x01941b (0x2653 bytes) 000019 1746791655.618964 punch from 0x010a0b to 0x01ce5f (0xc455 bytes) fallocating to largest ever: 0x135aa 000020 1746791655.621657 fallocate 0x00b375 thru 0x0135a9 (0x8235 bytes) 000022 1746791655.623225 punch from 0x009297 to 0x00a6e8 (0x1452 bytes) 000023 1746791655.625436 read 0x01aac2 thru 0x01bb88 (0x10c7 bytes) truncating to largest ever: 0x3c5cc 000024 1746791655.628521 trunc from 0x021a94 to 0x03c5cb (0x1ab38 bytes) 000026 1746791655.631871 read 0x01bede thru 0x02658a (0xa6ad bytes) 000028 1746791655.636831 MIRROR_EXTEND to 2 mirrors 000029 1746791655.731634 mapread 0x024564 thru 0x02db7d (0x961a bytes) 000030 1746791655.738813 write 0x02232c thru 0x030671 (0xe346 bytes) 000034 1746791655.750151 mapread 0x00f02f thru 0x015a4f (0x6a21 bytes) 000035 1746791655.753484 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000037 1746791655.785785 punch from 0x00504f to 0x0127ec (0xd79e bytes) 000040 1746791655.788250 mapwrite 0x0398bc thru 0x03a475 (0xbba bytes) 000043 1746791655.795484 mapread 0x01d72d thru 0x023dc5 (0x6699 bytes) 000046 1746791655.799115 write 0x01b574 thru 0x028d24 (0xd7b1 bytes) 000047 1746791655.802359 mapwrite 0x02a367 thru 0x02a7bc (0x456 bytes) 000049 1746791655.807692 write 0x010eb3 thru 0x01c5ea (0xb738 bytes) 000050 1746791655.810645 read 0x03a044 thru 0x03c5cb (0x2588 bytes) 000051 1746791655.813131 MIRROR_EXTEND to 2 mirrors 000052 1746791655.917572 trunc from 0x03c5cc to 0x00aa03 (0x31bc8 bytes) 000053 1746791655.932015 MIRROR_RESYNC 2 mirrors 000055 1746791656.007430 read 0x007985 thru 0x00aa03 (0x307f bytes) 000056 1746791656.008864 punch from 0x000bed to 0x0072fb (0x670f bytes) 000057 1746791656.015674 read 0x001777 thru 0x00aa03 (0x928d bytes) 000058 1746791656.017891 write 0x02fbc9 thru 0x036d00 (0x7138 bytes) 000059 1746791656.018767 mapwrite 0x011aad thru 0x017413 (0x5967 bytes) 000061 1746791656.025899 mapwrite 0x00f6af thru 0x01b4e9 (0xbe3b bytes) 000062 1746791656.030654 read 0x031e8c thru 0x036d00 (0x4e75 bytes) 000063 1746791656.030960 mapread 0x01a0a0 thru 0x026d23 (0xcc84 bytes) 000064 1746791656.031516 read 0x01b7ef thru 0x02066d (0x4e7f bytes) 000065 1746791656.031809 mapread 0x02f8f4 thru 0x032e5b (0x3568 bytes) 000068 1746791656.033747 MIRROR_EXTEND to 3 mirrors 000069 1746791656.133547 mapread 0x03436a thru 0x036d00 (0x2997 bytes) 000070 1746791656.138985 trunc from 0x036d01 to 0x012d9e (0x23f62 bytes) 000071 1746791656.150938 punch from 0x0051c3 to 0x011d9b (0xcbd9 bytes) 000072 1746791656.152344 mapwrite 0x005e5c thru 0x01295f (0xcb04 bytes) 000074 1746791656.162718 write 0x02f957 thru 0x031a62 (0x210c bytes) 000075 1746791656.163404 fallocate 0x00c3db thru 0x015eeb (0x9b11 bytes) 000076 1746791656.166045 MIRROR_EXTEND to 4 mirrors 000077 1746791656.253980 punch from 0x0263b4 to 0x02a2f7 (0x3f44 bytes) 000078 1746791656.261304 trunc from 0x031a63 to 0x01d7cd (0x14295 bytes) 000079 1746791656.269581 punch from 0x00b942 to 0x0151f8 (0x98b7 bytes) 000080 1746791656.270899 mapread 0x016867 thru 0x01d7cd (0x6f67 bytes) 000081 1746791656.274738 punch from 0x000b22 to 0x00e741 (0xdc20 bytes) 000082 1746791656.275999 fallocate 0x01853e thru 0x01dc75 (0x5738 bytes) 000084 1746791656.277753 punch from 0x012c70 to 0x01d7cd (0xab5e bytes) 000085 1746791656.281540 MIRROR_EXTEND to 5 mirrors 000086 1746791656.362389 mapwrite 0x006578 thru 0x00a4eb (0x3f74 bytes) 000087 1746791656.391864 mapread 0x01ccf8 thru 0x01d7cd (0xad6 bytes) 000088 1746791656.394159 write 0x01d5d4 thru 0x0269b8 (0x93e5 bytes) 000091 1746791656.396797 mapread 0x0269af thru 0x0269b8 (0xa bytes) 000092 1746791656.398653 MIRROR_RESYNC 5 mirrors 000093 1746791656.561169 mapwrite 0x031824 thru 0x032238 (0xa15 bytes) 000094 1746791656.577688 write 0x0033b8 thru 0x011284 (0xdecd bytes) 000096 1746791656.581430 mapwrite 0x01821a thru 0x020f15 (0x8cfc bytes) 000098 1746791656.589175 fallocate 0x02996e thru 0x02f7a2 (0x5e35 bytes) 000099 1746791656.590605 punch from 0x0287a2 to 0x02be0b (0x366a bytes) 000100 1746791656.591877 trunc from 0x032239 to 0x02e339 (0x3eff bytes) 000101 1746791656.596570 mapwrite 0x01ae44 thru 0x01b7e6 (0x9a3 bytes) 000102 1746791656.600793 punch from 0x02c0c1 to 0x02e339 (0x2279 bytes) 000103 1746791656.601893 read 0x01f8b0 thru 0x020161 (0x8b2 bytes) 000104 1746791656.602292 trunc from 0x02e33a to 0x01054c (0x1dded bytes) 000105 1746791656.605623 write 0x01a7b4 thru 0x0263ac (0xbbf9 bytes) 000108 1746791656.607065 read 0x006a36 thru 0x013231 (0xc7fc bytes) 000110 1746791656.609572 fallocate 0x02a815 thru 0x02cddd (0x25c9 bytes) 000111 1746791656.611928 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000112 1746791656.657985 mapwrite 0x018a91 thru 0x01e661 (0x5bd1 bytes) 000114 1746791656.666521 write 0x02facc thru 0x035d55 (0x628a bytes) 000115 1746791656.669182 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000116 1746791656.712503 fallocate 0x025638 thru 0x02ced1 (0x789a bytes) 000117 1746791656.714720 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000118 1746791656.755754 mapwrite 0x0282fe thru 0x02b14c (0x2e4f bytes) 000119 1746791656.765396 MIRROR_EXTEND to 3 mirrors 000120 1746791656.866607 write 0x009aa3 thru 0x00cf09 (0x3467 bytes) 000121 1746791656.875252 write 0x025634 thru 0x025dbd (0x78a bytes) 000122 1746791656.877180 trunc from 0x035d56 to 0x0078b4 (0x2e4a1 bytes) 000124 1746791656.884982 trunc from 0x0078b5 to 0x0262db (0x1ea27 bytes) 000125 1746791656.888411 read 0x0137ed thru 0x022ece (0xf6e2 bytes) 000128 1746791656.890944 mapwrite 0x00d0af thru 0x0109e8 (0x393a bytes) 000129 1746791656.897425 read 0x011326 thru 0x0171b0 (0x5e8b bytes) 000130 1746791656.897836 fallocate 0x012b84 thru 0x01d05a (0xa4d7 bytes) 000131 1746791656.900364 MIRROR_RESYNC 3 mirrors 000133 1746791657.020792 MIRROR_RESYNC 3 mirrors 000134 1746791657.052508 punch from 0x020fe4 to 0x0262db (0x52f8 bytes) 000136 1746791657.059993 trunc from 0x0262dc to 0x036651 (0x10376 bytes) 000137 1746791657.068062 read 0x02cfe7 thru 0x032171 (0x518b bytes) 000138 1746791657.070106 read 0x025781 thru 0x025f98 (0x818 bytes) 000140 1746791657.073396 write 0x03edc7 thru 0x03ffff (0x1239 bytes) 000141 1746791657.073878 read 0x024f54 thru 0x0293aa (0x4457 bytes) 000142 1746791657.074143 write 0x002816 thru 0x0028eb (0xd6 bytes) 000143 1746791657.074586 write 0x030c6b thru 0x034939 (0x3ccf bytes) 000144 1746791657.075135 mapread 0x01567a thru 0x02537a (0xfd01 bytes) 000146 1746791657.076017 read 0x007a8c thru 0x017317 (0xf88c bytes) 000147 1746791657.076322 mapread 0x021136 thru 0x02fb0d (0xe9d8 bytes) 000148 1746791657.078505 MIRROR_RESYNC 3 mirrors 000149 1746791657.257014 read 0x03341d thru 0x03ffff (0xcbe3 bytes) 000150 1746791657.259660 trunc from 0x040000 to 0x02c66e (0x13991 bytes) 000151 1746791657.276376 read 0x02b07f thru 0x02c245 (0x11c7 bytes) 000152 1746791657.281957 read 0x005644 thru 0x00a91c (0x52d9 bytes) 000153 1746791657.282659 punch from 0x01db0a to 0x029922 (0xbe19 bytes) 000154 1746791657.284853 mapwrite 0x01fe6c thru 0x021e85 (0x201a bytes) 000155 1746791657.293594 mapwrite 0x00fe5b thru 0x01e15e (0xe304 bytes) 000156 1746791657.304941 trunc from 0x02c66f to 0x014b4f (0x17b1f bytes) 000157 1746791657.313780 fallocate 0x02842a thru 0x028f0f (0xae6 bytes) 000158 1746791657.316346 read 0x00d401 thru 0x00e374 (0xf74 bytes) 000159 1746791657.320387 fallocate 0x005ccf thru 0x00edef (0x9121 bytes) 000160 1746791657.322535 read 0x01287e thru 0x014b4f (0x22d2 bytes) 000162 1746791657.323595 mapread 0x00d2b5 thru 0x010c6e (0x39ba bytes) 000163 1746791657.326070 MIRROR_EXTEND to 4 mirrors 000164 1746791657.433798 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000165 1746791657.499152 read 0x00257e thru 0x008778 (0x61fb bytes) 000166 1746791657.504861 punch from 0x00ca08 to 0x014b4f (0x8148 bytes) 000167 1746791657.514445 trunc from 0x014b50 to 0x00399b (0x111b4 bytes) 000168 1746791657.523977 mapwrite 0x03aef3 thru 0x03ffff (0x510d bytes) 000169 1746791657.538732 mapread 0x038fb1 thru 0x03ffff (0x704f bytes) 000170 1746791657.541088 MIRROR_EXTEND to 4 mirrors 000171 1746791657.662390 mapread 0x012e92 thru 0x01899c (0x5b0b bytes) 000172 1746791657.669706 read 0x02bf36 thru 0x034763 (0x882e bytes) 000173 1746791657.673224 fallocate 0x00ddea thru 0x017f21 (0xa138 bytes) fallocating to largest ever: 0x40000 000174 1746791657.680935 fallocate 0x038f2c thru 0x03ffff (0x70d4 bytes) 000176 1746791657.682941 punch from 0x01550a to 0x01a64d (0x5144 bytes) 000177 1746791657.684447 punch from 0x009605 to 0x013564 (0x9f60 bytes) 000178 1746791657.685671 punch from 0x00c960 to 0x015779 (0x8e1a bytes) 000179 1746791657.686925 trunc from 0x040000 to 0x0031d3 (0x3ce2c bytes) 000180 1746791657.697603 MIRROR_RESYNC 4 mirrors 000183 1746791657.800924 trunc from 0x0031d4 to 0x0234b6 (0x202e3 bytes) 000185 1746791657.812535 read 0x00cc39 thru 0x017e6a (0xb232 bytes) 000186 1746791657.814988 punch from 0x018375 to 0x02001a (0x7ca6 bytes) 000187 1746791657.817443 MIRROR_EXTEND to 5 mirrors 000190 1746791657.922149 mapwrite 0x0304db thru 0x034715 (0x423b bytes) 000191 1746791657.946930 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000192 1746791657.995438 trunc from 0x034716 to 0x00a44b (0x2a2ca bytes) 000193 1746791658.001486 mapwrite 0x00d7f3 thru 0x0181df (0xa9ed bytes) 000194 1746791658.017865 mapread 0x004b2f thru 0x00844b (0x391d bytes) 000196 1746791658.021443 MIRROR_EXTEND to 5 mirrors 000197 1746791658.122179 write 0x0227e9 thru 0x030583 (0xdd9b bytes) 000198 1746791658.131160 trunc from 0x030584 to 0x00de1d (0x22766 bytes) 000199 1746791658.139358 mapread 0x00a7f5 thru 0x00de1d (0x3629 bytes) 000200 1746791658.143833 mapread 0x00090f thru 0x009753 (0x8e45 bytes) 000201 1746791658.146700 mapread 0x003a06 thru 0x004a57 (0x1052 bytes) 000202 1746791658.147285 mapread 0x006c05 thru 0x0083bc (0x17b8 bytes) 000205 1746791658.148150 fallocate 0x016e6c thru 0x0233e0 (0xc575 bytes) 000206 1746791658.149949 fallocate 0x03c4a4 thru 0x03ffff (0x3b5c bytes) 000207 1746791658.151379 read 0x008b73 thru 0x012a62 (0x9ef0 bytes) 000208 1746791658.153627 fallocate 0x00877e thru 0x0091aa (0xa2d bytes) 000209 1746791658.155267 mapread 0x00ae41 thru 0x016e94 (0xc054 bytes) 000210 1746791658.156843 fallocate 0x000213 thru 0x004cad (0x4a9b bytes) 000211 1746791658.158200 mapwrite 0x013207 thru 0x01b398 (0x8192 bytes) 000212 1746791658.166548 fallocate 0x00c6be thru 0x0150f7 (0x8a3a bytes) 000213 1746791658.168122 punch from 0x01c503 to 0x0233e0 (0x6ede bytes) 000215 1746791658.169981 punch from 0x008b52 to 0x00c593 (0x3a42 bytes) 000216 1746791658.172308 punch from 0x01860a to 0x01aa4f (0x2446 bytes) 000217 1746791658.173858 trunc from 0x0233e1 to 0x002fe5 (0x203fb bytes) 000218 1746791658.177980 read 0x0013a0 thru 0x002fe5 (0x1c46 bytes) 000219 1746791658.178450 punch from 0x0022ab to 0x002fe5 (0xd3b bytes) 000220 1746791658.180010 trunc from 0x002fe6 to 0x008ba6 (0x5bc1 bytes) 000221 1746791658.183765 trunc from 0x008ba7 to 0x01fc72 (0x170cc bytes) 000222 1746791658.187344 mapread 0x01c9d3 thru 0x01fc72 (0x32a0 bytes) 000223 1746791658.190513 trunc from 0x01fc73 to 0x02f547 (0xf8d5 bytes) 000225 1746791658.194865 trunc from 0x02f548 to 0x02a1a4 (0x53a3 bytes) 000226 1746791658.198577 fallocate 0x015918 thru 0x0231a0 (0xd889 bytes) 000227 1746791658.200181 mapwrite 0x02683c thru 0x02fedb (0x96a0 bytes) 000228 1746791658.211765 read 0x01c1e7 thru 0x01ee67 (0x2c81 bytes) 000229 1746791658.212052 write 0x00ce09 thru 0x018efc (0xc0f4 bytes) 000232 1746791658.213466 write 0x030730 thru 0x03ffff (0xf8d0 bytes) 000233 1746791658.214235 mapread 0x00d963 thru 0x013a1c (0x60ba bytes) 000234 1746791658.214624 write 0x0071d7 thru 0x00ac0f (0x3a39 bytes) 000238 1746791658.215718 read 0x008d62 thru 0x01239c (0x963b bytes) 000239 1746791658.215988 trunc from 0x040000 to 0x031629 (0xe9d6 bytes) 000240 1746791658.219741 mapwrite 0x025cad thru 0x02a47c (0x47d0 bytes) 000241 1746791658.225482 mapread 0x014532 thru 0x021ee7 (0xd9b6 bytes) 000242 1746791658.226116 trunc from 0x03162a to 0x0010cf (0x3055a bytes) 000244 1746791658.230120 fallocate 0x03c172 thru 0x03ffff (0x3e8e bytes) 000245 1746791658.231541 trunc from 0x0010d0 to 0x002fd8 (0x1f09 bytes) 000246 1746791658.234979 read 0x0015a7 thru 0x002fd8 (0x1a32 bytes) 000247 1746791658.236357 mapwrite 0x00536e thru 0x009aae (0x4741 bytes) 000248 1746791658.245996 trunc from 0x009aaf to 0x013c9a (0xa1ec bytes) 000249 1746791658.249060 fallocate 0x004482 thru 0x00645d (0x1fdc bytes) 000250 1746791658.250373 mapwrite 0x00fe56 thru 0x01d7b5 (0xd960 bytes) 000252 1746791658.261231 fallocate 0x01dc83 thru 0x025c81 (0x7fff bytes) 000253 1746791658.262589 punch from 0x020906 to 0x025c81 (0x537c bytes) 000254 1746791658.263714 write 0x0245e9 thru 0x02f9c1 (0xb3d9 bytes) 000255 1746791658.265930 write 0x02e8ad thru 0x034816 (0x5f6a bytes) 000256 1746791658.266480 mapread 0x02a085 thru 0x02de9b (0x3e17 bytes) 000257 1746791658.267002 fallocate 0x0303e5 thru 0x0312ed (0xf09 bytes) 000258 1746791658.268211 punch from 0x0207b5 to 0x0272b6 (0x6b02 bytes) 000259 1746791658.271425 mapread 0x029f9a thru 0x02c1fe (0x2265 bytes) 000260 1746791658.272066 punch from 0x012fad to 0x01a867 (0x78bb bytes) 000261 1746791658.273406 mapread 0x00ee8b thru 0x01709c (0x8212 bytes) 000262 1746791658.279970 mapread 0x024a2b thru 0x030b6d (0xc143 bytes) 000265 1746791658.283623 mapread 0x03452d thru 0x034816 (0x2ea bytes) 000267 1746791658.284392 punch from 0x02feb7 to 0x034816 (0x4960 bytes) 000268 1746791658.285736 mapwrite 0x03b9b5 thru 0x03ffff (0x464b bytes) 000270 1746791658.296211 mapread 0x0076c1 thru 0x00a0e3 (0x2a23 bytes) 000271 1746791658.297555 MIRROR_EXTEND to 6 mirrors 000272 1746791658.415197 punch from 0x0024e7 to 0x00aa25 (0x853f bytes) 000273 1746791658.424592 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000274 1746791658.465323 mapwrite 0x01e316 thru 0x02dca0 (0xf98b bytes) 000275 1746791658.477594 fallocate 0x02cd05 thru 0x036d2c (0xa028 bytes) 000276 1746791658.478988 read 0x01824a thru 0x01a6c8 (0x247f bytes) 000278 1746791658.480596 trunc from 0x040000 to 0x015f11 (0x2a0ee bytes) 000279 1746791658.488181 trunc from 0x015f12 to 0x02b5bf (0x156ae bytes) 000281 1746791658.491252 mapread 0x029e7c thru 0x02b5bf (0x1744 bytes) 000282 1746791658.493446 fallocate 0x014056 thru 0x016b94 (0x2b3f bytes) 000283 1746791658.494876 fallocate 0x030840 thru 0x03624b (0x5a0c bytes) 000284 1746791658.496147 fallocate 0x035b9c thru 0x03648a (0x8ef bytes) 000285 1746791658.497338 read 0x02c490 thru 0x02e7ce (0x233f bytes) 000286 1746791658.500941 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000287 1746791658.541148 punch from 0x017945 to 0x01f067 (0x7723 bytes) 000289 1746791658.543662 mapwrite 0x02e706 thru 0x03d2d3 (0xebce bytes) 000290 1746791658.560170 trunc from 0x03d2d4 to 0x03e857 (0x1584 bytes) 000291 1746791658.562922 mapread 0x02c885 thru 0x03a92a (0xe0a6 bytes) 000292 1746791658.563548 trunc from 0x03e858 to 0x0240bd (0x1a79a bytes) 000293 1746791658.566771 mapwrite 0x00c60e thru 0x01965c (0xd04f bytes) 000294 1746791658.575900 mapread 0x006796 thru 0x009877 (0x30e2 bytes) 000295 1746791658.576314 write 0x01147b thru 0x017dc6 (0x694c bytes) 000296 1746791658.576886 read 0x017fc7 thru 0x023f46 (0xbf80 bytes) 000297 1746791658.579581 MIRROR_EXTEND to 5 mirrors 000298 1746791658.670315 trunc from 0x0240be to 0x028385 (0x42c8 bytes) 000299 1746791658.681990 trunc from 0x028386 to 0x00a690 (0x1dcf5 bytes) 000300 1746791658.687111 MIRROR_RESYNC 5 mirrors 000301 1746791658.807863 mapwrite 0x01873e thru 0x018dad (0x670 bytes) 000302 1746791658.824417 trunc from 0x018dae to 0x02c789 (0x139dc bytes) 000303 1746791658.828762 trunc from 0x02c78a to 0x01f895 (0xcef4 bytes) 000304 1746791658.832797 mapread 0x019f5b thru 0x01f895 (0x593b bytes) 000305 1746791658.835774 mapread 0x00b268 thru 0x00e7c7 (0x3560 bytes) 000306 1746791658.837878 fallocate 0x01706d thru 0x0262fb (0xf28f bytes) 000307 1746791658.839312 mapread 0x0042f8 thru 0x013e66 (0xfb6f bytes) 000308 1746791658.841369 punch from 0x0032c3 to 0x00f262 (0xbfa0 bytes) 000309 1746791658.842721 trunc from 0x01f896 to 0x035635 (0x15da0 bytes) 000310 1746791658.846084 trunc from 0x035636 to 0x00d03c (0x285f9 bytes) 000311 1746791658.849378 punch from 0x006cb4 to 0x00d03c (0x6389 bytes) 000312 1746791658.850551 fallocate 0x00004f thru 0x003d1c (0x3cce bytes) 000313 1746791658.851946 write 0x02cf0c thru 0x02f5f9 (0x26ee bytes) 000314 1746791658.853516 MIRROR_RESYNC 5 mirrors 000315 1746791658.973470 MIRROR_EXTEND to 6 mirrors 000318 1746791659.067171 write 0x01f747 thru 0x0232de (0x3b98 bytes) 000319 1746791659.076633 mapwrite 0x00daac thru 0x0102f7 (0x284c bytes) 000320 1746791659.083287 fallocate 0x01ac39 thru 0x021b8e (0x6f56 bytes) 000321 1746791659.084598 mapwrite 0x00bebe thru 0x019bf1 (0xdd34 bytes) 000322 1746791659.091223 punch from 0x01d134 to 0x023955 (0x6822 bytes) 000323 1746791659.092543 punch from 0x005dd4 to 0x012449 (0xc676 bytes) 000324 1746791659.093969 read 0x015e4e thru 0x024a08 (0xebbb bytes) 000325 1746791659.096162 mapwrite 0x00eda0 thru 0x01320d (0x446e bytes) 000327 1746791659.102374 punch from 0x006736 to 0x00cf60 (0x682b bytes) 000328 1746791659.103746 trunc from 0x02f5fa to 0x021796 (0xde63 bytes) 000330 1746791659.111229 fallocate 0x0328c7 thru 0x03a336 (0x7a70 bytes) 000331 1746791659.112588 trunc from 0x021797 to 0x02e19c (0xca06 bytes) 000332 1746791659.115625 trunc from 0x02e19d to 0x019909 (0x14893 bytes) 000334 1746791659.119624 fallocate 0x03ca6e thru 0x03ffff (0x3592 bytes) 000335 1746791659.122015 mapwrite 0x01a60f thru 0x01fd38 (0x572a bytes) 000336 1746791659.133266 read 0x002318 thru 0x006e98 (0x4b81 bytes) 000337 1746791659.135723 read 0x0060d5 thru 0x0116dc (0xb608 bytes) 000338 1746791659.136185 trunc from 0x01fd39 to 0x03e2dc (0x1e5a4 bytes) 000339 1746791659.140086 punch from 0x03a224 to 0x03e2dc (0x40b9 bytes) 000340 1746791659.141733 mapwrite 0x00a7a9 thru 0x0116e9 (0x6f41 bytes) 000341 1746791659.148331 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000342 1746791659.196290 trunc from 0x03e2dd to 0x003cf6 (0x3a5e6 bytes) 000343 1746791659.201230 mapread 0x0011a0 thru 0x003cf6 (0x2b57 bytes) 000344 1746791659.205453 trunc from 0x003cf7 to 0x02e74d (0x2aa57 bytes) 000347 1746791659.211652 MIRROR_RESYNC 5 mirrors 000348 1746791659.362171 mapread 0x00ef1f thru 0x01a895 (0xb977 bytes) 000349 1746791659.367796 fallocate 0x00d70f thru 0x0176dc (0x9fce bytes) 000350 1746791659.375315 mapwrite 0x02545a thru 0x034879 (0xf420 bytes) 000351 1746791659.394943 mapwrite 0x02ff6d thru 0x035133 (0x51c7 bytes) 000354 1746791659.405384 trunc from 0x035134 to 0x03c225 (0x70f2 bytes) 000355 1746791659.408807 trunc from 0x03c226 to 0x01f5e4 (0x1cc41 bytes) 000356 1746791659.412078 mapread 0x01d96b thru 0x01f5e4 (0x1c7a bytes) 000357 1746791659.414103 mapwrite 0x020ff4 thru 0x02645b (0x5468 bytes) 000358 1746791659.423955 punch from 0x009868 to 0x0115fb (0x7d94 bytes) 000360 1746791659.426738 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000361 1746791659.467703 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000362 1746791659.507617 mapwrite 0x0339d4 thru 0x0389e2 (0x500f bytes) 000364 1746791659.521007 mapread 0x00f5a3 thru 0x01b854 (0xc2b2 bytes) 000365 1746791659.524962 fallocate 0x039d62 thru 0x03d00c (0x32ab bytes) 000366 1746791659.526619 punch from 0x02cd12 to 0x02f0e5 (0x23d4 bytes) 000368 1746791659.528355 trunc from 0x0389e3 to 0x020a59 (0x17f89 bytes) 000369 1746791659.531884 mapwrite 0x0086e3 thru 0x00b2e7 (0x2c05 bytes) 000370 1746791659.538022 fallocate 0x02997c thru 0x02b730 (0x1db5 bytes) 000371 1746791659.539672 trunc from 0x020a5a to 0x03e0b9 (0x1d660 bytes) 000372 1746791659.542859 mapread 0x0243e3 thru 0x02da24 (0x9642 bytes) 000373 1746791659.546772 mapwrite 0x01a934 thru 0x020836 (0x5f03 bytes) 000374 1746791659.552023 punch from 0x027c7a to 0x02cde4 (0x516b bytes) 000375 1746791659.553357 write 0x01e0f1 thru 0x027372 (0x9282 bytes) 000377 1746791659.555625 write 0x03a47e thru 0x03f488 (0x500b bytes) 000378 1746791659.558004 trunc from 0x03f489 to 0x031125 (0xe363 bytes) 000379 1746791659.561457 fallocate 0x001e3b thru 0x003b6b (0x1d31 bytes) 000381 1746791659.563219 write 0x036530 thru 0x03bbd3 (0x56a4 bytes) 000384 1746791659.564709 punch from 0x03b250 to 0x03bbd3 (0x984 bytes) 000385 1746791659.567847 mapread 0x010698 thru 0x01ed83 (0xe6ec bytes) 000387 1746791659.568744 trunc from 0x03bbd4 to 0x03625d (0x5976 bytes) 000388 1746791659.572134 trunc from 0x03625e to 0x03f604 (0x93a7 bytes) 000389 1746791659.575521 read 0x011f34 thru 0x014a31 (0x2afe bytes) 000390 1746791659.575801 punch from 0x013750 to 0x01b983 (0x8234 bytes) 000391 1746791659.578600 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000392 1746791659.619248 trunc from 0x03f605 to 0x00c8c9 (0x32d3b bytes) 000393 1746791659.623954 read 0x0095c7 thru 0x00c8c9 (0x3303 bytes) 000394 1746791659.625941 trunc from 0x00c8ca to 0x033e19 (0x27550 bytes) 000395 1746791659.629144 mapread 0x025788 thru 0x02b673 (0x5eec bytes) 000397 1746791659.633513 mapread 0x0077f2 thru 0x017405 (0xfc14 bytes) 000399 1746791659.639100 mapwrite 0x0304ff thru 0x038a5f (0x8561 bytes) 000400 1746791659.649453 fallocate 0x007135 thru 0x01558d (0xe459 bytes) 000401 1746791659.651956 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000402 1746791659.678208 read 0x020a28 thru 0x022273 (0x184c bytes) 000403 1746791659.679730 trunc from 0x038a60 to 0x017df7 (0x20c68 bytes) 000406 1746791659.685313 fallocate 0x003488 thru 0x003494 (0xd bytes) 000407 1746791659.687872 MIRROR_EXTEND to 2 mirrors 000409 1746791659.779347 read 0x003065 thru 0x00efa8 (0xbf44 bytes) 000411 1746791659.784736 mapread 0x012c7b thru 0x017df7 (0x517d bytes) 000413 1746791659.793512 fallocate 0x00b03b thru 0x01a1fc (0xf1c2 bytes) 000414 1746791659.801783 trunc from 0x01a1fd to 0x007ec4 (0x12338 bytes) 000415 1746791659.810543 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000416 1746791659.838884 MIRROR_EXTEND to 2 mirrors 000418 1746791659.931258 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000420 1746791659.956825 mapread 0x0021c8 thru 0x00268c (0x4c5 bytes) 000421 1746791659.958213 fallocate 0x0364cb thru 0x03791a (0x1450 bytes) 000423 1746791659.961166 trunc from 0x007ec5 to 0x02f87b (0x279b7 bytes) 000424 1746791659.967793 fallocate 0x02081e thru 0x02be73 (0xb656 bytes) 000425 1746791659.969198 write 0x01ad38 thru 0x020f0a (0x61d3 bytes) 000428 1746791659.972532 trunc from 0x02f87c to 0x031a29 (0x21ae bytes) 000429 1746791659.975656 write 0x00240b thru 0x010046 (0xdc3c bytes) 000430 1746791659.978878 trunc from 0x031a2a to 0x017095 (0x1a994 bytes) 000431 1746791659.982382 trunc from 0x017096 to 0x020d6a (0x9cd5 bytes) 000432 1746791659.985975 mapread 0x00eb60 thru 0x013ad9 (0x4f7a bytes) 000433 1746791659.988357 mapread 0x01b8b6 thru 0x020d6a (0x54b5 bytes) 000434 1746791659.990578 fallocate 0x03db01 thru 0x03ffff (0x24ff bytes) 000435 1746791659.991879 write 0x01f503 thru 0x01ff5f (0xa5d bytes) 000436 1746791659.993464 MIRROR_EXTEND to 2 mirrors 000437 1746791660.093546 trunc from 0x020d6b to 0x027b66 (0x6dfc bytes) 000438 1746791660.101894 trunc from 0x027b67 to 0x019e65 (0xdd01 bytes) 000439 1746791660.105511 mapwrite 0x024536 thru 0x0278a6 (0x3371 bytes) 000440 1746791660.115890 mapread 0x012266 thru 0x013538 (0x12d3 bytes) 000441 1746791660.118500 fallocate 0x02cd44 thru 0x0353b2 (0x866f bytes) 000442 1746791660.120366 mapread 0x0210b9 thru 0x02e644 (0xd58c bytes) 000443 1746791660.123230 read 0x00b0e1 thru 0x019bf9 (0xeb19 bytes) 000444 1746791660.126027 MIRROR_RESYNC 2 mirrors 000446 1746791660.210447 write 0x036730 thru 0x03bf17 (0x57e8 bytes) 000447 1746791660.218042 write 0x03dc3b thru 0x03ffff (0x23c5 bytes) 000448 1746791660.219836 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000449 1746791660.248379 read 0x02f666 thru 0x030379 (0xd14 bytes) 000450 1746791660.249853 fallocate 0x00f1c5 thru 0x014876 (0x56b2 bytes) 000451 1746791660.251365 trunc from 0x040000 to 0x02616e (0x19e91 bytes) 000452 1746791660.259060 punch from 0x00fb3d to 0x01ecd4 (0xf198 bytes) 000453 1746791660.260400 read 0x022519 thru 0x02616e (0x3c56 bytes) 000454 1746791660.263714 mapread 0x013b27 thru 0x02120e (0xd6e8 bytes) 000455 1746791660.264165 mapwrite 0x00e168 thru 0x00f115 (0xfae bytes) 000457 1746791660.268587 read 0x00c4fa thru 0x014c50 (0x8757 bytes) 000458 1746791660.268903 write 0x03a84f thru 0x03e7a2 (0x3f54 bytes) 000459 1746791660.269459 mapread 0x037932 thru 0x03b6bb (0x3d8a bytes) 000460 1746791660.271558 punch from 0x013b60 to 0x018c41 (0x50e2 bytes) 000462 1746791660.272971 fallocate 0x01f1b0 thru 0x023561 (0x43b2 bytes) 000463 1746791660.274335 write 0x00a3bd thru 0x019bc6 (0xf80a bytes) 000464 1746791660.275309 read 0x00ef61 thru 0x01b888 (0xc928 bytes) 000465 1746791660.275756 fallocate 0x0186bf thru 0x0255a8 (0xceea bytes) 000466 1746791660.277054 write 0x03cad5 thru 0x03ffff (0x352b bytes) 000467 1746791660.278518 read 0x01b0f6 thru 0x022f8b (0x7e96 bytes) 000469 1746791660.279011 trunc from 0x040000 to 0x00631a (0x39ce5 bytes) 000471 1746791660.282738 write 0x0121e6 thru 0x0160b1 (0x3ecc bytes) 000474 1746791660.283844 fallocate 0x03378c thru 0x03ffff (0xc874 bytes) 000475 1746791660.285047 punch from 0x02bff1 to 0x0328f1 (0x6901 bytes) 000479 1746791660.286602 punch from 0x02d24a to 0x032a37 (0x57ee bytes) 000480 1746791660.287583 mapread 0x01738d thru 0x01b743 (0x43b7 bytes) 000481 1746791660.290385 fallocate 0x02a56d thru 0x02d6b9 (0x314d bytes) 000482 1746791660.291698 trunc from 0x040000 to 0x003d2b (0x3c2d4 bytes) 000483 1746791660.295147 punch from 0x00154c to 0x003d2b (0x27e0 bytes) 000484 1746791660.296357 mapwrite 0x000284 thru 0x00c951 (0xc6ce bytes) 000485 1746791660.315330 punch from 0x00c237 to 0x00c951 (0x71b bytes) 000486 1746791660.316680 fallocate 0x0050c3 thru 0x00f99f (0xa8dd bytes) 000487 1746791660.317924 fallocate 0x037d39 thru 0x038d2b (0xff3 bytes) 000488 1746791660.319217 read 0x00234b thru 0x007ff2 (0x5ca8 bytes) 000489 1746791660.320563 mapwrite 0x01b08e thru 0x024363 (0x92d6 bytes) 000491 1746791660.327263 mapwrite 0x02e267 thru 0x03dc25 (0xf9bf bytes) 000493 1746791660.339535 trunc from 0x03dc26 to 0x0146b0 (0x29575 bytes) 000495 1746791660.342841 read 0x00eb6b thru 0x010c30 (0x20c6 bytes) 000496 1746791660.343201 read 0x004733 thru 0x0096c3 (0x4f91 bytes) 000497 1746791660.343530 write 0x0041d7 thru 0x00793b (0x3765 bytes) 000498 1746791660.344023 punch from 0x00012c to 0x004f84 (0x4e59 bytes) 000501 1746791660.347440 mapread 0x00e6f4 thru 0x0146b0 (0x5fbd bytes) 000504 1746791660.348748 trunc from 0x0146b1 to 0x013fa9 (0x707 bytes) 000505 1746791660.352096 fallocate 0x009744 thru 0x017022 (0xd8df bytes) 000507 1746791660.353660 punch from 0x00376d to 0x005264 (0x1af8 bytes) 000508 1746791660.354857 read 0x00416c thru 0x011884 (0xd719 bytes) 000510 1746791660.357126 trunc from 0x013faa to 0x038547 (0x2459e bytes) 000511 1746791660.360068 punch from 0x01f2f8 to 0x02deb7 (0xebc0 bytes) 000512 1746791660.361254 mapwrite 0x03ad94 thru 0x03ffff (0x526c bytes) 000513 1746791660.372428 fallocate 0x0316a4 thru 0x033584 (0x1ee1 bytes) 000515 1746791660.374331 fallocate 0x00834b thru 0x00a970 (0x2626 bytes) 000516 1746791660.375596 trunc from 0x040000 to 0x0192b4 (0x26d4b bytes) 000518 1746791660.379944 fallocate 0x036aa4 thru 0x03759f (0xafc bytes) 000520 1746791660.382521 fallocate 0x00521f thru 0x012aff (0xd8e1 bytes) 000522 1746791660.384229 mapread 0x01723a thru 0x0192b4 (0x207b bytes) 000523 1746791660.384667 mapwrite 0x0028b1 thru 0x0108dc (0xe02c bytes) 000525 1746791660.393154 MIRROR_EXTEND to 2 mirrors 000527 1746791660.484336 trunc from 0x0192b5 to 0x03e81d (0x25569 bytes) 000529 1746791660.496688 fallocate 0x033569 thru 0x03656d (0x3005 bytes) 000530 1746791660.497985 read 0x00e380 thru 0x01baef (0xd770 bytes) 000535 1746791660.500869 trunc from 0x03e81e to 0x03f73c (0xf1f bytes) 000536 1746791660.504834 mapread 0x0290a2 thru 0x036709 (0xd668 bytes) 000537 1746791660.511345 mapwrite 0x01d5ea thru 0x0225f8 (0x500f bytes) 000538 1746791660.517844 fallocate 0x024dda thru 0x034019 (0xf240 bytes) 000542 1746791660.520169 trunc from 0x03f73d to 0x02d5a3 (0x12199 bytes) 000543 1746791660.523960 punch from 0x02a24a to 0x02d5a3 (0x335a bytes) 000545 1746791660.525404 read 0x000a06 thru 0x0084cd (0x7ac8 bytes) 000546 1746791660.527643 mapwrite 0x0236f3 thru 0x029b05 (0x6413 bytes) 000547 1746791660.532684 mapwrite 0x007105 thru 0x00afbc (0x3eb8 bytes) 000548 1746791660.535764 mapwrite 0x0057ce thru 0x0145fa (0xee2d bytes) 000549 1746791660.541405 mapread 0x001fd0 thru 0x008fa4 (0x6fd5 bytes) 000550 1746791660.542920 MIRROR_EXTEND to 3 mirrors 000551 1746791660.636881 read 0x016e82 thru 0x01a2e9 (0x3468 bytes) 000552 1746791660.640792 write 0x00c77b thru 0x01b961 (0xf1e7 bytes) 000553 1746791660.653254 MIRROR_EXTEND to 4 mirrors 000555 1746791660.765980 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000556 1746791660.841826 MIRROR_RESYNC 3 mirrors 000557 1746791660.958574 MIRROR_RESYNC 3 mirrors 000558 1746791660.997678 punch from 0x00d10b to 0x010260 (0x3156 bytes) 000559 1746791661.005545 write 0x02e38b thru 0x03ca67 (0xe6dd bytes) 000560 1746791661.006837 read 0x00ef10 thru 0x014bfe (0x5cef bytes) 000561 1746791661.009252 mapwrite 0x02d14d thru 0x02ff2c (0x2de0 bytes) 000562 1746791661.017146 trunc from 0x03ca68 to 0x018ec0 (0x23ba7 bytes) 000563 1746791661.024808 read 0x01293c thru 0x018ec0 (0x6585 bytes) 000565 1746791661.027299 read 0x006f3e thru 0x01490b (0xd9ce bytes) 000568 1746791661.028083 trunc from 0x018ec1 to 0x014654 (0x486c bytes) 000569 1746791661.032978 MIRROR_RESYNC 3 mirrors 000570 1746791661.153218 write 0x03440c thru 0x03e48f (0xa084 bytes) 000571 1746791661.161965 mapread 0x00523d thru 0x007679 (0x243d bytes) 000572 1746791661.166997 mapread 0x00ef6b thru 0x0134c7 (0x455d bytes) 000573 1746791661.169409 punch from 0x01c1c2 to 0x02926b (0xd0aa bytes) 000574 1746791661.170846 punch from 0x01de4e to 0x02d813 (0xf9c6 bytes) 000575 1746791661.173037 fallocate 0x02e760 thru 0x03cfa2 (0xe843 bytes) 000577 1746791661.176259 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000578 1746791661.224533 mapwrite 0x006a91 thru 0x014d0a (0xe27a bytes) 000579 1746791661.241091 punch from 0x00444c to 0x0058ca (0x147f bytes) 000580 1746791661.242725 mapwrite 0x0300a0 thru 0x036655 (0x65b6 bytes) 000581 1746791661.250081 trunc from 0x03e490 to 0x01fa5c (0x1ea33 bytes) 000582 1746791661.260583 MIRROR_EXTEND to 3 mirrors 000584 1746791661.368368 read 0x0092c6 thru 0x01164c (0x8387 bytes) 000585 1746791661.372080 mapwrite 0x022e06 thru 0x02b9fc (0x8bf7 bytes) 000586 1746791661.394543 trunc from 0x02b9fd to 0x03dbba (0x121be bytes) 000587 1746791661.398852 write 0x03ab3a thru 0x03ffff (0x54c6 bytes) 000588 1746791661.402507 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000589 1746791661.450766 MIRROR_RESYNC 2 mirrors 000591 1746791661.550531 read 0x02d664 thru 0x0398ff (0xc29c bytes) 000592 1746791661.552602 trunc from 0x040000 to 0x00826f (0x37d90 bytes) 000593 1746791661.564588 mapread 0x005f93 thru 0x00826f (0x22dd bytes) 000594 1746791661.569103 mapwrite 0x0300e2 thru 0x03b5ac (0xb4cb bytes) 000596 1746791661.582976 mapread 0x026891 thru 0x02cc7e (0x63ee bytes) 000597 1746791661.585303 punch from 0x033e07 to 0x038528 (0x4722 bytes) 000599 1746791661.587252 trunc from 0x03b5ad to 0x038bca (0x29e2 bytes) 000600 1746791661.590698 fallocate 0x01f1c5 thru 0x01fd4a (0xb86 bytes) 000601 1746791661.591922 write 0x03a143 thru 0x03ffff (0x5ebd bytes) 000602 1746791661.592460 write 0x003009 thru 0x00354c (0x544 bytes) 000603 1746791661.592841 read 0x0128b7 thru 0x01f745 (0xce8f bytes) 000604 1746791661.595372 punch from 0x000a98 to 0x006b34 (0x609d bytes) 000605 1746791661.599601 trunc from 0x040000 to 0x00962b (0x369d4 bytes) 000606 1746791661.602986 mapread 0x00702c thru 0x00962b (0x2600 bytes) 000608 1746791661.603760 fallocate 0x0300d2 thru 0x0304e6 (0x415 bytes) 000609 1746791661.604979 read 0x004a6e thru 0x004e3e (0x3d1 bytes) 000610 1746791661.606905 punch from 0x005f9d to 0x00962b (0x368f bytes) 000611 1746791661.608258 fallocate 0x0261b3 thru 0x026338 (0x186 bytes) 000612 1746791661.609615 fallocate 0x00f10d thru 0x01cdad (0xdca1 bytes) 000614 1746791661.612373 read 0x010be2 thru 0x01cdad (0xc1cc bytes) 000616 1746791661.615207 trunc from 0x01cdae to 0x034e91 (0x180e4 bytes) 000617 1746791661.618892 write 0x00272b thru 0x00c008 (0x98de bytes) 000618 1746791661.619586 mapread 0x02d38a thru 0x033b3c (0x67b3 bytes) 000619 1746791661.621958 mapwrite 0x0297ac thru 0x033943 (0xa198 bytes) 000620 1746791661.628011 read 0x005b14 thru 0x00afba (0x54a7 bytes) 000621 1746791661.628485 mapwrite 0x004608 thru 0x0070f9 (0x2af2 bytes) 000622 1746791661.631884 mapread 0x01c476 thru 0x029981 (0xd50c bytes) 000623 1746791661.633640 MIRROR_EXTEND to 3 mirrors 000624 1746791661.738780 fallocate 0x015b12 thru 0x01811f (0x260e bytes) 000625 1746791661.746417 read 0x02df94 thru 0x034e91 (0x6efe bytes) 000626 1746791661.748479 read 0x02bc3d thru 0x02f32d (0x36f1 bytes) 000627 1746791661.751332 mapwrite 0x00c783 thru 0x00ca81 (0x2ff bytes) 000628 1746791661.755590 mapread 0x017948 thru 0x026937 (0xeff0 bytes) 000630 1746791661.756434 read 0x0134b3 thru 0x0151ef (0x1d3d bytes) 000632 1746791661.756910 mapread 0x02f64b thru 0x034e91 (0x5847 bytes) 000634 1746791661.758822 MIRROR_RESYNC 3 mirrors 000635 1746791661.897175 write 0x031ae5 thru 0x039a09 (0x7f25 bytes) 000636 1746791661.905513 trunc from 0x039a0a to 0x02341d (0x165ec bytes) 000637 1746791661.913682 fallocate 0x02070a thru 0x0291b4 (0x8aab bytes) 000639 1746791661.915518 mapread 0x019d76 thru 0x02341d (0x96a8 bytes) 000640 1746791661.922011 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000642 1746791661.966772 fallocate 0x007839 thru 0x00b7fd (0x3fc5 bytes) 000643 1746791661.968311 mapwrite 0x016f05 thru 0x019631 (0x272d bytes) 000644 1746791661.977772 mapread 0x0021f2 thru 0x00bfeb (0x9dfa bytes) 000645 1746791661.981077 trunc from 0x02341e to 0x03f9e6 (0x1c5c9 bytes) 000646 1746791661.985453 write 0x01d903 thru 0x02c06d (0xe76b bytes) 000648 1746791661.987592 trunc from 0x03f9e7 to 0x0064ff (0x394e7 bytes) 000650 1746791661.991569 mapwrite 0x029314 thru 0x033170 (0x9e5d bytes) 000651 1746791662.003579 mapread 0x032b54 thru 0x033170 (0x61d bytes) 000652 1746791662.004041 read 0x01a03f thru 0x01a97c (0x93e bytes) 000654 1746791662.005679 punch from 0x00ae34 to 0x00e3b9 (0x3586 bytes) 000655 1746791662.007091 fallocate 0x030fe6 thru 0x0324b0 (0x14cb bytes) 000657 1746791662.009036 read 0x011bf8 thru 0x018c1f (0x7028 bytes) 000658 1746791662.011733 fallocate 0x02298f thru 0x027e39 (0x54ab bytes) 000659 1746791662.013213 punch from 0x02ae6b to 0x02cb00 (0x1c96 bytes) 000660 1746791662.015980 write 0x02a44c thru 0x0329db (0x8590 bytes) 000662 1746791662.019553 punch from 0x008501 to 0x011902 (0x9402 bytes) 000663 1746791662.021310 mapread 0x012747 thru 0x01818d (0x5a47 bytes) 000664 1746791662.022071 write 0x01e32f thru 0x02502d (0x6cff bytes) 000666 1746791662.023428 punch from 0x0027ba to 0x009a60 (0x72a7 bytes) 000667 1746791662.025348 punch from 0x028c1a to 0x033170 (0xa557 bytes) 000668 1746791662.029057 punch from 0x02161a to 0x02e2f0 (0xccd7 bytes) 000669 1746791662.030561 mapread 0x0274a9 thru 0x030282 (0x8dda bytes) 000670 1746791662.044194 MIRROR_RESYNC 2 mirrors 000672 1746791662.148826 punch from 0x0059f8 to 0x008ea8 (0x34b1 bytes) 000674 1746791662.156871 trunc from 0x033171 to 0x03d5c9 (0xa459 bytes) 000675 1746791662.165439 trunc from 0x03d5ca to 0x016f59 (0x26670 bytes) 000676 1746791662.169024 fallocate 0x037b9c thru 0x03ffff (0x8464 bytes) 000677 1746791662.170723 write 0x0344da thru 0x037fd1 (0x3af8 bytes) 000678 1746791662.173898 write 0x036b47 thru 0x03ffff (0x94b9 bytes) 000679 1746791662.174936 write 0x01af49 thru 0x02a8dc (0xf994 bytes) 000680 1746791662.178514 mapread 0x03c0d3 thru 0x03ffff (0x3f2d bytes) 000682 1746791662.179242 trunc from 0x040000 to 0x00cef0 (0x3310f bytes) 000683 1746791662.182868 trunc from 0x00cef1 to 0x020bcc (0x13cdc bytes) 000684 1746791662.186203 fallocate 0x0106fa thru 0x014c3f (0x4546 bytes) 000685 1746791662.187746 fallocate 0x009d88 thru 0x009eed (0x166 bytes) 000686 1746791662.189034 mapread 0x01d258 thru 0x020bcc (0x3975 bytes) 000687 1746791662.192391 mapwrite 0x037c10 thru 0x03d0a7 (0x5498 bytes) 000689 1746791662.204175 trunc from 0x03d0a8 to 0x0191b4 (0x23ef3 bytes) 000690 1746791662.207607 write 0x031a1e thru 0x034c3a (0x321d bytes) 000693 1746791662.208781 punch from 0x029bcb to 0x0349cb (0xae01 bytes) 000695 1746791662.211617 fallocate 0x018dce thru 0x0272c3 (0xe4f6 bytes) 000696 1746791662.213043 write 0x00801b thru 0x012fb6 (0xaf9c bytes) 000697 1746791662.216396 mapread 0x025330 thru 0x02ddd4 (0x8aa5 bytes) 000698 1746791662.220109 mapread 0x008e54 thru 0x00a3ca (0x1577 bytes) 000699 1746791662.220529 mapread 0x031b9a thru 0x034c3a (0x30a1 bytes) 000702 1746791662.221355 fallocate 0x038dd3 thru 0x03fa9e (0x6ccc bytes) 000704 1746791662.223988 MIRROR_EXTEND to 3 mirrors 000705 1746791662.317669 punch from 0x00534d to 0x009772 (0x4426 bytes) 000706 1746791662.325240 trunc from 0x034c3b to 0x03b4ae (0x6874 bytes) 000709 1746791662.333160 write 0x021f86 thru 0x02cba5 (0xac20 bytes) 000710 1746791662.335634 read 0x0163ef thru 0x01afa3 (0x4bb5 bytes) 000711 1746791662.337558 mapread 0x004b4e thru 0x0059c0 (0xe73 bytes) 000713 1746791662.340144 punch from 0x033505 to 0x03b4ae (0x7faa bytes) 000714 1746791662.341385 trunc from 0x03b4af to 0x0379e9 (0x3ac5 bytes) 000715 1746791662.345261 fallocate 0x023ae3 thru 0x0279c4 (0x3ee2 bytes) 000716 1746791662.346346 write 0x031544 thru 0x034ea9 (0x3966 bytes) 000717 1746791662.348379 punch from 0x036433 to 0x0379e9 (0x15b7 bytes) 000718 1746791662.349478 trunc from 0x0379ea to 0x0161e0 (0x21809 bytes) 000719 1746791662.352714 mapread 0x009c3e thru 0x011d19 (0x80dc bytes) 000722 1746791662.355618 write 0x031a31 thru 0x035110 (0x36e0 bytes) 000723 1746791662.356147 write 0x0024a2 thru 0x002898 (0x3f7 bytes) 000724 1746791662.357869 MIRROR_RESYNC 3 mirrors 000725 1746791662.482831 mapwrite 0x033ab8 thru 0x03ffff (0xc548 bytes) 000726 1746791662.505856 MIRROR_EXTEND to 4 mirrors 000727 1746791662.608712 read 0x01e17f thru 0x020ff7 (0x2e79 bytes) 000729 1746791662.614021 write 0x01f739 thru 0x023f8b (0x4853 bytes) 000730 1746791662.623631 mapread 0x00eba4 thru 0x01a302 (0xb75f bytes) 000731 1746791662.627428 write 0x000d2c thru 0x003a76 (0x2d4b bytes) 000733 1746791662.629994 fallocate 0x009dd1 thru 0x00c9fc (0x2c2c bytes) 000734 1746791662.631482 fallocate 0x009836 thru 0x014fc1 (0xb78c bytes) 000735 1746791662.632935 mapwrite 0x002445 thru 0x0053b2 (0x2f6e bytes) 000736 1746791662.637944 punch from 0x018bf3 to 0x01c1d3 (0x35e1 bytes) 000737 1746791662.639226 mapread 0x02f06e thru 0x033894 (0x4827 bytes) 000738 1746791662.641536 punch from 0x03ad25 to 0x03bbcd (0xea9 bytes) 000739 1746791662.642759 mapread 0x016042 thru 0x01c3a1 (0x6360 bytes) 000741 1746791662.645074 trunc from 0x040000 to 0x02c9ca (0x13635 bytes) 000742 1746791662.652107 read 0x02194e thru 0x028ec1 (0x7574 bytes) 000743 1746791662.654105 fallocate 0x03eef0 thru 0x03ffff (0x1110 bytes) 000745 1746791662.655581 mapwrite 0x02855a thru 0x02f10e (0x6bb5 bytes) 000746 1746791662.664437 punch from 0x005158 to 0x007a43 (0x28ec bytes) 000747 1746791662.665605 trunc from 0x040000 to 0x039ed5 (0x612a bytes) 000748 1746791662.669909 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000750 1746791662.713399 write 0x005a85 thru 0x00b8e8 (0x5e64 bytes) 000751 1746791662.716750 punch from 0x007d66 to 0x014278 (0xc513 bytes) 000752 1746791662.720585 write 0x020bb5 thru 0x022e26 (0x2272 bytes) 000753 1746791662.723771 trunc from 0x039ed6 to 0x0001e0 (0x39cf5 bytes) 000754 1746791662.727258 write 0x0265a6 thru 0x02cf8b (0x69e6 bytes) 000755 1746791662.728192 mapwrite 0x0215b4 thru 0x024b2a (0x3577 bytes) 000756 1746791662.736634 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000757 1746791662.777889 trunc from 0x02cf8c to 0x00dcdc (0x1f2af bytes) 000762 1746791662.783183 punch from 0x0097ab to 0x00dcdc (0x4532 bytes) 000763 1746791662.785725 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000766 1746791662.812575 punch from 0x00c82a to 0x00dcdc (0x14b3 bytes) 000768 1746791662.814134 read 0x00c988 thru 0x00dcdc (0x1355 bytes) 000769 1746791662.815686 trunc from 0x00dcdd to 0x01add9 (0xd0fd bytes) 000770 1746791662.820648 write 0x01b8b8 thru 0x02a102 (0xe84b bytes) 000771 1746791662.821543 write 0x01c7f5 thru 0x01cfdf (0x7eb bytes) 000772 1746791662.821829 trunc from 0x02a103 to 0x02f6b4 (0x55b2 bytes) 000773 1746791662.825136 mapwrite 0x02fb8a thru 0x030ded (0x1264 bytes) 000774 1746791662.835728 trunc from 0x030dee to 0x01267c (0x1e771 bytes) 000775 1746791662.839228 fallocate 0x03f725 thru 0x03ffff (0x8db bytes) 000778 1746791662.841410 fallocate 0x028dca thru 0x031e4d (0x9084 bytes) 000779 1746791662.842723 fallocate 0x009858 thru 0x00f487 (0x5c30 bytes) 000780 1746791662.844046 mapwrite 0x037d2b thru 0x038c1f (0xef5 bytes) 000781 1746791662.850230 punch from 0x0228c0 to 0x02c42c (0x9b6d bytes) 000782 1746791662.852784 write 0x03f7ee thru 0x03ffff (0x812 bytes) 000784 1746791662.853829 mapread 0x01e596 thru 0x021348 (0x2db3 bytes) 000786 1746791662.856740 trunc from 0x040000 to 0x0136a3 (0x2c95c bytes) 000787 1746791662.860288 mapwrite 0x03765c thru 0x039ecb (0x2870 bytes) 000788 1746791662.869380 read 0x01abd7 thru 0x02748f (0xc8b9 bytes) 000789 1746791662.872525 read 0x02591a thru 0x0286dc (0x2dc3 bytes) 000791 1746791662.873022 punch from 0x011e24 to 0x01bf75 (0xa152 bytes) 000792 1746791662.874527 mapread 0x01f88a thru 0x0241b0 (0x4927 bytes) 000793 1746791662.875182 mapread 0x00c3a2 thru 0x013de5 (0x7a44 bytes) 000801 1746791662.879804 write 0x007411 thru 0x00b2e4 (0x3ed4 bytes) 000802 1746791662.880368 mapwrite 0x03680e thru 0x03ab16 (0x4309 bytes) 000804 1746791662.890864 fallocate 0x01f719 thru 0x02c928 (0xd210 bytes) 000805 1746791662.892124 mapwrite 0x0278f9 thru 0x03639c (0xeaa4 bytes) 000806 1746791662.897888 fallocate 0x03952c thru 0x03ffff (0x6ad4 bytes) 000808 1746791662.899554 read 0x02c119 thru 0x033463 (0x734b bytes) 000810 1746791662.900490 mapwrite 0x008e27 thru 0x011089 (0x8263 bytes) 000811 1746791662.905327 punch from 0x01247d to 0x01454c (0x20d0 bytes) 000812 1746791662.906517 mapread 0x01ef02 thru 0x02eadb (0xfbda bytes) 000814 1746791662.907455 mapwrite 0x010084 thru 0x010ba1 (0xb1e bytes) 000816 1746791662.911868 fallocate 0x003b4e thru 0x00eef2 (0xb3a5 bytes) 000820 1746791662.914050 write 0x01a92e thru 0x02a673 (0xfd46 bytes) 000821 1746791662.914946 write 0x02266f thru 0x029f32 (0x78c4 bytes) 000822 1746791662.915511 mapread 0x01d285 thru 0x01d413 (0x18f bytes) 000823 1746791662.916120 mapread 0x0351e5 thru 0x037bcd (0x29e9 bytes) 000825 1746791662.916816 fallocate 0x022cb2 thru 0x0256cc (0x2a1b bytes) 000826 1746791662.918357 punch from 0x0064dc to 0x008906 (0x242b bytes) 000828 1746791662.919925 fallocate 0x02477c thru 0x02e113 (0x9998 bytes) 000831 1746791662.921810 punch from 0x017cf4 to 0x01f0f0 (0x73fd bytes) 000832 1746791662.924971 fallocate 0x00e4fb thru 0x0113f4 (0x2efa bytes) 000833 1746791662.926332 mapwrite 0x00ff79 thru 0x0151f0 (0x5278 bytes) 000834 1746791662.932505 mapread 0x00be45 thru 0x0198bc (0xda78 bytes) 000836 1746791662.933535 trunc from 0x040000 to 0x009b3c (0x364c3 bytes) 000837 1746791662.937961 mapwrite 0x00a250 thru 0x00efba (0x4d6b bytes) 000838 1746791662.952018 fallocate 0x027b8d thru 0x0323fe (0xa872 bytes) 000841 1746791662.953983 mapwrite 0x023e81 thru 0x03033f (0xc4bf bytes) 000842 1746791662.976549 punch from 0x012aca to 0x01d269 (0xa7a0 bytes) 000844 1746791662.978223 mapread 0x01b3c5 thru 0x0277af (0xc3eb bytes) 000845 1746791662.986748 trunc from 0x030340 to 0x030e85 (0xb46 bytes) 000846 1746791662.990726 punch from 0x02daf9 to 0x030e85 (0x338d bytes) 000849 1746791662.993552 trunc from 0x030e86 to 0x00bcfb (0x2518a bytes) 000850 1746791662.997253 punch from 0x004e7e to 0x006ff4 (0x2177 bytes) 000851 1746791662.998626 mapwrite 0x012372 thru 0x01de29 (0xbab8 bytes) 000852 1746791663.020514 trunc from 0x01de2a to 0x031ca7 (0x13e7e bytes) 000853 1746791663.024116 fallocate 0x00d271 thru 0x00e41e (0x11ae bytes) 000854 1746791663.025584 fallocate 0x016fd9 thru 0x01a80e (0x3836 bytes) 000855 1746791663.026934 mapwrite 0x032a7e thru 0x03ee65 (0xc3e8 bytes) 000856 1746791663.050452 trunc from 0x03ee66 to 0x005aac (0x393b9 bytes) 000857 1746791663.054230 trunc from 0x005aad to 0x019f54 (0x144a8 bytes) 000859 1746791663.057939 fallocate 0x010b6e thru 0x014735 (0x3bc8 bytes) 000860 1746791663.060361 read 0x018ed4 thru 0x019f54 (0x1081 bytes) 000863 1746791663.063564 read 0x0166d7 thru 0x019f54 (0x387e bytes) 000864 1746791663.063873 mapwrite 0x0036c4 thru 0x005274 (0x1bb1 bytes) 000866 1746791663.069339 fallocate 0x0000de thru 0x00f12a (0xf04d bytes) 000867 1746791663.070983 mapwrite 0x028ecc thru 0x0295dc (0x711 bytes) 000868 1746791663.081895 fallocate 0x003803 thru 0x00ef09 (0xb707 bytes) 000869 1746791663.084387 fallocate 0x025984 thru 0x02f48f (0x9b0c bytes) 000870 1746791663.085818 write 0x02f38f thru 0x03410c (0x4d7e bytes) 000871 1746791663.088141 read 0x0116bf thru 0x0177f7 (0x6139 bytes) 000874 1746791663.089063 read 0x0330cf thru 0x03410c (0x103e bytes) 000875 1746791663.089472 mapwrite 0x0014f8 thru 0x00bbed (0xa6f6 bytes) 000877 1746791663.095922 mapread 0x014527 thru 0x01fc9e (0xb778 bytes) 000880 1746791663.097445 read 0x018b0c thru 0x02339e (0xa893 bytes) 000882 1746791663.099376 MIRROR_EXTEND to 2 mirrors 000883 1746791663.206257 trunc from 0x03410d to 0x0081cc (0x2bf40 bytes) 000884 1746791663.220695 mapread 0x0013d2 thru 0x0081cc (0x6dfb bytes) 000885 1746791663.227150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000886 1746791663.260412 mapread 0x005c96 thru 0x0081cc (0x2537 bytes) 000887 1746791663.265738 fallocate 0x03ae57 thru 0x03ffff (0x51a9 bytes) 000889 1746791663.268094 trunc from 0x0081cd to 0x0334e3 (0x2b317 bytes) 000891 1746791663.273211 read 0x023387 thru 0x02be79 (0x8af3 bytes) 000892 1746791663.275478 read 0x004aa2 thru 0x0108ae (0xbe0d bytes) 000893 1746791663.278064 mapread 0x009454 thru 0x00baa6 (0x2653 bytes) 000894 1746791663.278479 read 0x0064da thru 0x011457 (0xaf7e bytes) 000895 1746791663.278763 mapwrite 0x03e5b5 thru 0x03ffff (0x1a4b bytes) 000896 1746791663.288551 read 0x01d480 thru 0x027285 (0x9e06 bytes) 000897 1746791663.288897 read 0x02a8cd thru 0x03a683 (0xfdb7 bytes) 000898 1746791663.289402 write 0x022316 thru 0x02ae09 (0x8af4 bytes) 000901 1746791663.290830 mapwrite 0x008fe0 thru 0x00915d (0x17e bytes) 000902 1746791663.296353 mapread 0x032190 thru 0x039289 (0x70fa bytes) 000906 1746791663.297644 fallocate 0x036566 thru 0x03ffff (0x9a9a bytes) 000907 1746791663.299067 trunc from 0x040000 to 0x00d232 (0x32dcd bytes) 000909 1746791663.303600 write 0x022173 thru 0x02ba55 (0x98e3 bytes) 000910 1746791663.304380 fallocate 0x02c739 thru 0x033157 (0x6a1f bytes) 000911 1746791663.306137 fallocate 0x029a37 thru 0x02c195 (0x275f bytes) 000912 1746791663.307890 write 0x02261a thru 0x0295a8 (0x6f8f bytes) 000913 1746791663.308373 read 0x022128 thru 0x025a2d (0x3906 bytes) 000916 1746791663.309288 punch from 0x00def8 to 0x01cfaf (0xf0b8 bytes) 000917 1746791663.310582 punch from 0x029bc8 to 0x02aac1 (0xefa bytes) 000919 1746791663.315509 read 0x00befd thru 0x012b09 (0x6c0d bytes) 000920 1746791663.318352 trunc from 0x02ba56 to 0x01883a (0x1321b bytes) 000922 1746791663.322791 write 0x00e79a thru 0x0198a6 (0xb10d bytes) 000923 1746791663.323529 mapread 0x00dd1a thru 0x0198a6 (0xbb8d bytes) 000924 1746791663.324023 mapread 0x00d3a7 thru 0x01072f (0x3389 bytes) 000925 1746791663.324391 write 0x0249df thru 0x028c79 (0x429b bytes) 000926 1746791663.324882 fallocate 0x006a21 thru 0x00f847 (0x8e27 bytes) 000927 1746791663.326336 mapread 0x0142e0 thru 0x01f206 (0xaf27 bytes) 000928 1746791663.332128 write 0x02295c thru 0x026a3e (0x40e3 bytes) 000929 1746791663.334851 punch from 0x0179f1 to 0x01e22c (0x683c bytes) 000930 1746791663.339085 punch from 0x00d77b to 0x01753d (0x9dc3 bytes) 000931 1746791663.340484 fallocate 0x02351f thru 0x02cbea (0x96cc bytes) 000932 1746791663.341752 punch from 0x00bec9 to 0x014235 (0x836d bytes) 000933 1746791663.343183 punch from 0x02386e to 0x02cbea (0x937d bytes) 000934 1746791663.345617 fallocate 0x03056b thru 0x03192f (0x13c5 bytes) 000935 1746791663.346995 fallocate 0x03b70a thru 0x03ffff (0x48f6 bytes) 000936 1746791663.349753 punch from 0x02b441 to 0x03301d (0x7bdd bytes) 000937 1746791663.351985 mapwrite 0x02a35e thru 0x034934 (0xa5d7 bytes) 000939 1746791663.362417 mapwrite 0x0324d3 thru 0x036f86 (0x4ab4 bytes) 000942 1746791663.367211 fallocate 0x0299ae thru 0x037b47 (0xe19a bytes) 000943 1746791663.368588 mapwrite 0x0380c9 thru 0x03ffff (0x7f37 bytes) 000944 1746791663.373482 fallocate 0x015634 thru 0x01f550 (0x9f1d bytes) 000945 1746791663.375061 read 0x00b527 thru 0x0139e3 (0x84bd bytes) 000946 1746791663.378032 write 0x006bc9 thru 0x00ceba (0x62f2 bytes) 000947 1746791663.378620 fallocate 0x01babc thru 0x022e81 (0x73c6 bytes) 000948 1746791663.379932 mapread 0x022fb5 thru 0x028334 (0x5380 bytes) 000949 1746791663.380400 mapread 0x03d1db thru 0x03ffff (0x2e25 bytes) 000950 1746791663.380763 mapread 0x009e06 thru 0x00a666 (0x861 bytes) 000951 1746791663.381098 mapread 0x016e6c thru 0x021d86 (0xaf1b bytes) 000952 1746791663.381537 punch from 0x00a3c8 to 0x01293e (0x8577 bytes) 000953 1746791663.384732 write 0x02e0ea thru 0x039139 (0xb050 bytes) 000954 1746791663.385355 read 0x02d28f thru 0x03bce0 (0xea52 bytes) 000955 1746791663.385690 write 0x00a07b thru 0x00b4a3 (0x1429 bytes) 000956 1746791663.387760 trunc from 0x040000 to 0x028fff (0x17000 bytes) 000957 1746791663.391631 punch from 0x0089ff to 0x00cda8 (0x43aa bytes) 000958 1746791663.395431 MIRROR_EXTEND to 2 mirrors 000960 1746791663.501039 read 0x013a43 thru 0x015460 (0x1a1e bytes) 000961 1746791663.504871 fallocate 0x03b42a thru 0x03ffff (0x4bd6 bytes) 000962 1746791663.512869 read 0x01cff2 thru 0x01dacb (0xada bytes) 000963 1746791663.516051 trunc from 0x040000 to 0x03c758 (0x38a7 bytes) 000964 1746791663.524443 punch from 0x0307aa to 0x037c07 (0x745e bytes) 000966 1746791663.527226 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000967 1746791663.557161 write 0x004290 thru 0x00e52b (0xa29c bytes) 000968 1746791663.562127 punch from 0x0250a3 to 0x02e7b8 (0x9716 bytes) 000969 1746791663.563616 fallocate 0x03b61f thru 0x03ffff (0x49e1 bytes) 000971 1746791663.565132 mapread 0x026ef7 thru 0x02df0c (0x7016 bytes) 000972 1746791663.567636 trunc from 0x040000 to 0x01c48a (0x23b75 bytes) 000973 1746791663.570818 trunc from 0x01c48b to 0x0302af (0x13e25 bytes) 000974 1746791663.573833 trunc from 0x0302b0 to 0x0111f3 (0x1f0bc bytes) 000975 1746791663.576672 write 0x025f52 thru 0x02f578 (0x9627 bytes) 000976 1746791663.577404 mapread 0x00b6c1 thru 0x018ddb (0xd71b bytes) 000978 1746791663.581170 punch from 0x014912 to 0x015a2c (0x111b bytes) 000979 1746791663.582384 punch from 0x00f717 to 0x012d89 (0x3673 bytes) 000981 1746791663.586209 MIRROR_EXTEND to 2 mirrors 000982 1746791663.683396 fallocate 0x00c32f thru 0x0184f0 (0xc1c2 bytes) 000983 1746791663.692451 MIRROR_RESYNC 2 mirrors 000984 1746791663.767415 read 0x00831b thru 0x00e271 (0x5f57 bytes) 000985 1746791663.769361 fallocate 0x023528 thru 0x029cde (0x67b7 bytes) 000987 1746791663.776683 mapwrite 0x00ebc8 thru 0x01a7f3 (0xbc2c bytes) 000988 1746791663.789374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb48:0x0]: Invalid argument (22) 000989 1746791663.816613 trunc from 0x02f579 to 0x00eb74 (0x20a04 bytes) 000990 1746791663.824140 fallocate 0x01a1fc thru 0x021ded (0x7bf2 bytes) 000991 1746791663.825810 fallocate 0x03028b thru 0x03614d (0x5ec3 bytes) 000993 1746791663.827519 trunc from 0x03614e to 0x03b6da (0x558d bytes) 000994 1746791663.830619 punch from 0x031a32 to 0x03870a (0x6cd9 bytes) 000995 1746791663.831983 write 0x00e1db thru 0x015647 (0x746d bytes) 000996 1746791663.834869 read 0x029acb thru 0x029d07 (0x23d bytes) 000997 1746791663.836161 write 0x0137b3 thru 0x013d05 (0x553 bytes) 000998 1746791663.836432 trunc from 0x03b6db to 0x02cf8c (0xe74e bytes) 000999 1746791663.839919 read 0x027188 thru 0x027df5 (0xc6e bytes) All operations completed A-OK! PASS 70a (9s) == sanity-flr test 71: check mirror extend parent fid ==== 07:54:24 (1746791664) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0xb80:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 07:54:27 (1746791667) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4640160 .. Extending file size to 6651264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5854016 .. Extending file size to 7543840 .. Extending file size to 9201152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9322016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9233184 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5297 5298 5299 5301 5304 done 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:55:30 (1746791730) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1000288 .. Extending file size to 8689536 .. Extending file size to 5935776 .. done Extending file size to 6398304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6861120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7172608 .. Extending file size to 7446496 .. Extending file size to 7904672 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7965568 .. Extending file size to 8756768 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9138496 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8909184 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9016384 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9071584 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 9832 9833 9835 9836 9839 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:56:33 (1746791793) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2800 1284888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3468 1284220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14700 3592320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22684 3584336 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37384 7176656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2800 1284888 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3468 1284220 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14700 3592320 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22684 3584336 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37384 7176656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0328076 s, 63.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0498963 s, 42.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:56:38 (1746791798) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0251301 s, 83.5 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 07:56:39 (1746791799) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:56:41 (1746791801) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0223539 s, 93.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x109c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:56:43 (1746791803) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219643 s, 95.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:56:44 (1746791804) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0236268 s, 88.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:57:07 (1746791827) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126575 s, 82.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x109d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x109d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x109e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x109e:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:57:42 (1746791862) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0200349 s, 105 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:58:17 (1746791897) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg307-server: Pool lustre.test_204f created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.016175 s, 130 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg307-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:59:05 (1746791945) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:59:06 (1746791946) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:59:08 (1746791948) oleg307-server: Pool lustre.test_206 created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg307-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:59:21 (1746791961) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203765 s, 51.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00925516 s, 113 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xdab801b9 Mirror 3: 0xdab801b9 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:59:23 (1746791963) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:59:23 (1746791963) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:59:24 (1746791964) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00860981 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2640000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2640000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2640000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfb0000 Read this (4 bytes): BBBB 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:59:37 (1746791977) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00232049 s, 3.5 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:59:38 (1746791978) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894133 s, 117 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-09 07:59:39.000000000 -0400 Modify: 2025-05-09 07:59:39.000000000 -0400 Change: 2025-05-09 07:59:39.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:59:40 (1746791980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00928126 s, 113 MB/s oleg307-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg307-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg307-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg307-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg307-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:59:54 (1746791994) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.019688 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00361374 s, 1.1 MB/s size after second write -rw-r--r-- 1 root root 4096 May 9 07:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 9 07:59 /mnt/lustre/f211.sanity-flr PASS 211 (1s) == sanity-flr test complete, duration 687 sec ============ 07:59:55 (1746791995) === sanity-flr: start cleanup 07:59:56 (1746791996) === === sanity-flr: finish cleanup 07:59:57 (1746791997) ===