-----============= acceptance-small: sanity-flr ============----- Wed May 7 09:45:58 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg324-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:46:03 (1746625563) === oleg324-client.virtnet: executing check_config_client /mnt/lustre oleg324-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg324-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6f73000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6f73000.idle_timeout=debug disable quota as required oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:46:12 (1746625572) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7642] == sanity-flr test 0a: lfs mirror create with -N option == 09:46:12 (1746625572) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:46:15 (1746625575) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:46:16 (1746625576) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:46:17 (1746625577) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:46:21 (1746625581) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:46:22 (1746625582) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:46:23 (1746625583) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6f73000.stats=clear osc.lustre-OST0001-osc-ffff8800b6f73000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:46:25 (1746625585) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:46:27 (1746625587) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:46:28 (1746625588) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:46:29 (1746625589) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 09:46:31 (1746625591) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:46:33 (1746625593) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:46:34 (1746625594) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:46:36 (1746625596) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:46:38 (1746625598) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:46:40 (1746625600) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.28897 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022533 s, 46.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:46:42 (1746625602) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0233085 s, 135 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:46:44 (1746625604) Stopping /mnt/lustre-ost1 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg324-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg324-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:47:08 (1746625628) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.157878 s, 59.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg324-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:47:18 (1746625638) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg324-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg324-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:47:41 (1746625661) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.715575 s, 73.3 MB/s -rw-r--r-- 1 root root 52428800 May 7 09:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.188s user 0m0.001s sys 0m0.104s Stopping /mnt/lustre-ost1 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.205s user 0m0.000s sys 0m0.100s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg324-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg324-server oleg324-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.560s user 0m0.001s sys 0m0.143s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg324-server: oleg324-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg324-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f73000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:48:09 (1746625689) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:48:10 (1746625690) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:48:11 (1746625691) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:48:12 (1746625692) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 09:48:14 (1746625694) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.143105 s, 87.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:48:17 (1746625697) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016245 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122338 s, 85.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:48:19 (1746625699) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:48:22 (1746625702) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00100555 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00058514 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000497967 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000655971 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 09:48:24 (1746625704) create test files with size 15747782 .. 18f5fe34eff57f2fe34e9562d5200d56 - 2751f5313aff98a4fcc193d7f90628ed - 73a102300fbf61f428a6a0c91766da9e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 09:48:30 (1746625710) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:48:33 (1746625713) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:48:35 (1746625715) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227663 s, 46.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00961204 s, 109 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 09:48:37 (1746625717) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0378223 s, 55.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.127886 s, 32.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0539741 s, 38.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 09:48:41 (1746625721) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:48:42 (1746625722) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:48:44 (1746625724) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166526 s, 63.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 09:48:47 (1746625727) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.109452 s, 28.7 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:48:50 (1746625730) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x43:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:48:53 (1746625733) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.110128 s, 95.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:49:10 (1746625750) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88013027f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0779152 s, 135 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012e01d800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:49:14 (1746625754) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:49:21 (1746625761) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:49:28 (1746625768) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:49:29 (1746625769) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0781442 s, 40.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0514508 s, 61.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b4d0c89fe23afc53b5cb2f8f8321d9e2 - mirror 1 checksum: 3cc27f9253f2f10c9faf6373ea9613bf - mirror 2 checksum: b4d0c89fe23afc53b5cb2f8f8321d9e2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 7 09:49:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 7 09:49:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b4d0c89fe23afc53b5cb2f8f8321d9e2 - mirror 1 checksum: 3cc27f9253f2f10c9faf6373ea9613bf - mirror 2 checksum: 3cc27f9253f2f10c9faf6373ea9613bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 7 09:49:31 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 7 09:49:31 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:49:32 (1746625772) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00923713 s, 114 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 529408.. 530431: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:49:34 (1746625774) Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:49:41 (1746625781) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:49:55 (1746625795) targets are mounted 09:49:55 (1746625795) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924865 s, 113 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:50:03 (1746625803) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0165613 s, 1.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00663927 s, 3.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:50:05 (1746625805) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0153899 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00529193 s, 3.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:50:08 (1746625808) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00414552 s, 4.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:50:10 (1746625810) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.012385 s, 1.7 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:50:13 (1746625813) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00311476 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0107786 s, 380 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:50:15 (1746625815) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:50:20 (1746625820) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:50:39 (1746625839) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:50:56 (1746625856) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:51:13 (1746625873) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:51:15 (1746625875) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26320 mirror split pid 26321 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26320 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26321 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 09:52:17 (1746625937) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5938 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1746625938.224209 MIRROR_EXTEND to 2 mirrors 000004 1746625938.295453 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) skipping zero length punch hole skipping zero size read 000009 1746625938.340190 write 0x017023 thru 0x0216d8 (0xa6b6 bytes) truncating to largest ever: 0xf814 000010 1746625938.346624 trunc from 0x0216d9 to 0x00f813 (0x11ec5 bytes) 000011 1746625938.357869 read 0x0029c4 thru 0x003f6a (0x15a7 bytes) 000012 1746625938.360449 mapwrite 0x021ee9 thru 0x0230a6 (0x11be bytes) truncating to largest ever: 0x25d25 000013 1746625938.375636 trunc from 0x0230a7 to 0x025d24 (0x2c7e bytes) 000014 1746625938.379483 trunc from 0x025d25 to 0x01f071 (0x6cb3 bytes) 000015 1746625938.384102 mapwrite 0x03e883 thru 0x03ffff (0x177d bytes) 000017 1746625938.396998 read 0x03acc8 thru 0x03f043 (0x437c bytes) fallocating to largest ever: 0x3993d 000018 1746625938.397435 fallocate 0x0329ec thru 0x03993c (0x6f51 bytes) 000020 1746625938.401497 read 0x03ab7f thru 0x03ceb3 (0x2335 bytes) 000021 1746625938.404370 MIRROR_EXTEND to 2 mirrors 000023 1746625938.544097 mapread 0x0357c1 thru 0x03e3f6 (0x8c36 bytes) 000025 1746625938.550446 trunc from 0x040000 to 0x008ca1 (0x3735e bytes) 000026 1746625938.570294 mapwrite 0x022ce5 thru 0x02f6e4 (0xca00 bytes) 000027 1746625938.588068 fallocate 0x00b25f thru 0x011ee2 (0x6c84 bytes) 000028 1746625938.589685 fallocate 0x016e3b thru 0x024ee1 (0xe0a7 bytes) 000029 1746625938.591370 read 0x00c551 thru 0x00c781 (0x231 bytes) 000030 1746625938.592920 punch from 0x029bbd to 0x02bd27 (0x216b bytes) 000031 1746625938.595867 read 0x01ead8 thru 0x027b19 (0x9042 bytes) 000032 1746625938.596322 punch from 0x00a59d to 0x00b592 (0xff6 bytes) 000034 1746625938.600420 MIRROR_EXTEND to 3 mirrors 000035 1746625938.747406 punch from 0x02ea20 to 0x02f6e4 (0xcc5 bytes) 000036 1746625938.756721 mapread 0x00c92c thru 0x0129cb (0x60a0 bytes) 000037 1746625938.761967 write 0x006cff thru 0x013c5e (0xcf60 bytes) 000038 1746625938.766721 mapwrite 0x00452b thru 0x00b1f2 (0x6cc8 bytes) 000039 1746625938.775934 MIRROR_EXTEND to 4 mirrors 000041 1746625938.907294 write 0x0245c9 thru 0x028106 (0x3b3e bytes) 000042 1746625938.924844 trunc from 0x02f6e5 to 0x007911 (0x27dd3 bytes) 000043 1746625938.940782 MIRROR_EXTEND to 5 mirrors 000044 1746625939.035002 fallocate 0x0151e8 thru 0x01cca5 (0x7abe bytes) 000045 1746625939.045668 mapwrite 0x03911b thru 0x03ffff (0x6ee5 bytes) fallocating to largest ever: 0x3ceae 000046 1746625939.071160 fallocate 0x039111 thru 0x03cead (0x3d9d bytes) truncating to largest ever: 0x3de50 000047 1746625939.072994 trunc from 0x040000 to 0x03de4f (0x21b0 bytes) 000048 1746625939.077175 fallocate 0x01453f thru 0x01b315 (0x6dd7 bytes) 000049 1746625939.079044 punch from 0x037527 to 0x03de4f (0x6929 bytes) 000050 1746625939.080939 mapwrite 0x0149f9 thru 0x016508 (0x1b10 bytes) 000051 1746625939.089172 mapwrite 0x0103af thru 0x017a87 (0x76d9 bytes) 000053 1746625939.096374 write 0x014c2a thru 0x01bfe0 (0x73b7 bytes) 000054 1746625939.098222 punch from 0x00234e to 0x0063ff (0x40b2 bytes) 000055 1746625939.101499 MIRROR_EXTEND to 6 mirrors 000056 1746625939.277775 read 0x030ebf thru 0x03417f (0x32c1 bytes) 000057 1746625939.282994 mapwrite 0x0004c6 thru 0x00fad2 (0xf60d bytes) 000058 1746625939.329474 punch from 0x018d7c to 0x021751 (0x89d6 bytes) 000060 1746625939.332994 trunc from 0x03de50 to 0x02cf42 (0x10f0d bytes) 000061 1746625939.346853 write 0x00dae7 thru 0x01360a (0x5b24 bytes) 000063 1746625939.351584 trunc from 0x02cf43 to 0x009aa1 (0x234a1 bytes) 000065 1746625939.357716 mapwrite 0x03341c thru 0x03adb1 (0x7996 bytes) 000067 1746625939.378140 write 0x0106fb thru 0x01d992 (0xd298 bytes) 000068 1746625939.384189 read 0x01376e thru 0x01e92e (0xb1c1 bytes) 000069 1746625939.387378 read 0x0046d5 thru 0x00e7e6 (0xa112 bytes) 000070 1746625939.387793 trunc from 0x03adb2 to 0x009768 (0x31649 bytes) 000071 1746625939.394087 trunc from 0x009769 to 0x03a04b (0x308e3 bytes) 000072 1746625939.399863 mapwrite 0x02591a thru 0x02d843 (0x7f2a bytes) 000074 1746625939.414961 MIRROR_EXTEND to 7 mirrors 000075 1746625939.581112 write 0x033a25 thru 0x036570 (0x2b4c bytes) 000076 1746625939.596651 punch from 0x02202d to 0x0260db (0x40af bytes) 000077 1746625939.598753 trunc from 0x03a04c to 0x032c42 (0x7409 bytes) 000078 1746625939.611897 fallocate 0x003103 thru 0x003547 (0x445 bytes) 000079 1746625939.613677 read 0x023310 thru 0x0241a5 (0xe96 bytes) 000080 1746625939.616028 mapread 0x00f251 thru 0x012c7c (0x3a2c bytes) 000082 1746625939.619954 fallocate 0x0180dd thru 0x021428 (0x934c bytes) 000083 1746625939.622036 punch from 0x0005a4 to 0x0078cb (0x7328 bytes) 000085 1746625939.624592 mapread 0x017749 thru 0x01d4e8 (0x5da0 bytes) 000086 1746625939.628333 fallocate 0x029ad2 thru 0x02ae25 (0x1354 bytes) 000088 1746625939.630613 mapwrite 0x03b276 thru 0x03feb0 (0x4c3b bytes) 000089 1746625939.648249 fallocate 0x011037 thru 0x0146fd (0x36c7 bytes) 000090 1746625939.650468 punch from 0x02c2eb to 0x032eb2 (0x6bc8 bytes) 000091 1746625939.652097 mapwrite 0x02be94 thru 0x02e18a (0x22f7 bytes) 000092 1746625939.659661 read 0x03de77 thru 0x03feb0 (0x203a bytes) 000093 1746625939.660044 fallocate 0x007e95 thru 0x010c50 (0x8dbc bytes) 000094 1746625939.662119 read 0x00f3d8 thru 0x011163 (0x1d8c bytes) 000096 1746625939.662764 read 0x02c265 thru 0x0393dc (0xd178 bytes) 000097 1746625939.666871 mapread 0x009131 thru 0x00f9b0 (0x6880 bytes) 000098 1746625939.667534 trunc from 0x03feb1 to 0x02a042 (0x15e6e bytes) 000099 1746625939.672782 read 0x020e7d thru 0x024402 (0x3586 bytes) 000100 1746625939.673216 mapread 0x0120ca thru 0x01e1d4 (0xc10b bytes) 000101 1746625939.673896 mapread 0x00a6b8 thru 0x011523 (0x6e6c bytes) 000103 1746625939.674796 mapwrite 0x0176c0 thru 0x02759d (0xfede bytes) 000105 1746625939.685425 trunc from 0x02a043 to 0x022cbd (0x7385 bytes) 000106 1746625939.693342 MIRROR_EXTEND to 8 mirrors 000109 1746625939.882756 fallocate 0x0374a8 thru 0x03c5c2 (0x511b bytes) 000112 1746625939.895397 mapwrite 0x032646 thru 0x03782d (0x51e8 bytes) 000114 1746625939.918845 fallocate 0x020c6f thru 0x02fd49 (0xf0db bytes) 000115 1746625939.920406 trunc from 0x03782e to 0x0059de (0x31e4f bytes) 000117 1746625939.925019 trunc from 0x0059df to 0x001611 (0x43cd bytes) 000118 1746625939.928621 trunc from 0x001612 to 0x00ef91 (0xd980 bytes) 000120 1746625939.933569 fallocate 0x020f34 thru 0x029094 (0x8161 bytes) 000121 1746625939.935183 trunc from 0x00ef92 to 0x02cdaa (0x1de19 bytes) 000122 1746625939.939319 fallocate 0x02fda5 thru 0x030400 (0x65c bytes) 000123 1746625939.941346 write 0x019bdf thru 0x0229c4 (0x8de6 bytes) 000124 1746625939.946193 mapread 0x01aa09 thru 0x024e88 (0xa480 bytes) 000125 1746625939.949595 mapread 0x02594d thru 0x02971b (0x3dcf bytes) 000126 1746625939.950169 mapread 0x022280 thru 0x02741a (0x519b bytes) 000127 1746625939.950804 mapread 0x0113b0 thru 0x01abaf (0x9800 bytes) 000128 1746625939.956130 MIRROR_RESYNC 8 mirrors 000130 1746625940.354979 trunc from 0x02cdab to 0x03c81b (0xfa71 bytes) 000131 1746625940.371420 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000132 1746625940.441756 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000133 1746625940.502689 mapwrite 0x03b63e thru 0x03fc27 (0x45ea bytes) 000134 1746625940.520855 write 0x031b65 thru 0x03aecd (0x9369 bytes) 000135 1746625940.523938 punch from 0x013286 to 0x0153da (0x2155 bytes) 000137 1746625940.526534 punch from 0x030fc7 to 0x034185 (0x31bf bytes) 000138 1746625940.533667 MIRROR_RESYNC 6 mirrors fallocating to largest ever: 0x40000 000139 1746625940.940017 fallocate 0x03ff92 thru 0x03ffff (0x6e bytes) 000140 1746625940.950589 fallocate 0x00304e thru 0x0085fb (0x55ae bytes) 000141 1746625940.953116 fallocate 0x025af6 thru 0x025f1e (0x429 bytes) 000142 1746625940.956736 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000143 1746625941.011240 fallocate 0x0380aa thru 0x03ffff (0x7f56 bytes) 000144 1746625941.013442 read 0x0141c2 thru 0x016054 (0x1e93 bytes) 000146 1746625941.018755 trunc from 0x03fc28 to 0x015a38 (0x2a1ef bytes) 000147 1746625941.028567 read 0x002d15 thru 0x00507b (0x2367 bytes) 000148 1746625941.033122 MIRROR_RESYNC 5 mirrors 000149 1746625941.219445 fallocate 0x034130 thru 0x03f622 (0xb4f3 bytes) 000150 1746625941.233794 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000151 1746625941.305795 punch from 0x024d6e to 0x0259db (0xc6e bytes) 000154 1746625941.307992 punch from 0x02c8d5 to 0x03819c (0xb8c8 bytes) 000155 1746625941.311160 MIRROR_RESYNC 4 mirrors 000156 1746625941.540507 punch from 0x037377 to 0x03f622 (0x82ac bytes) 000157 1746625941.555283 trunc from 0x03f623 to 0x0190e4 (0x2653e bytes) 000158 1746625941.569460 write 0x010520 thru 0x015a12 (0x54f3 bytes) 000159 1746625941.573698 mapwrite 0x031e04 thru 0x039027 (0x7224 bytes) 000161 1746625941.595679 mapwrite 0x01e61c thru 0x021450 (0x2e35 bytes) 000162 1746625941.608391 fallocate 0x0062ee thru 0x00c90a (0x661d bytes) 000163 1746625941.611333 write 0x015b49 thru 0x01f694 (0x9b4c bytes) 000164 1746625941.612987 read 0x03708c thru 0x0375d4 (0x549 bytes) 000165 1746625941.613394 mapwrite 0x004326 thru 0x00adb5 (0x6a90 bytes) 000166 1746625941.628340 fallocate 0x035760 thru 0x038bc8 (0x3469 bytes) 000167 1746625941.630723 trunc from 0x039028 to 0x0271a4 (0x11e83 bytes) 000168 1746625941.637145 read 0x000c66 thru 0x005b8d (0x4f28 bytes) 000169 1746625941.641278 write 0x038f5b thru 0x03d105 (0x41ab bytes) 000170 1746625941.642093 mapread 0x03a18f thru 0x03d105 (0x2f77 bytes) 000171 1746625941.642792 mapwrite 0x019d23 thru 0x026fbd (0xd29b bytes) 000172 1746625941.654437 mapread 0x00e505 thru 0x01c473 (0xdf6f bytes) 000173 1746625941.655275 read 0x0327f8 thru 0x03d105 (0xa90e bytes) 000174 1746625941.659378 mapread 0x004861 thru 0x006914 (0x20b4 bytes) 000175 1746625941.660069 mapwrite 0x005320 thru 0x00c444 (0x7125 bytes) 000176 1746625941.668249 read 0x03bed9 thru 0x03d105 (0x122d bytes) 000177 1746625941.668834 read 0x006ccd thru 0x015821 (0xeb55 bytes) 000179 1746625941.669523 trunc from 0x03d106 to 0x032a09 (0xa6fc bytes) 000180 1746625941.677954 read 0x00f61a thru 0x0152a4 (0x5c8b bytes) 000182 1746625941.678733 mapwrite 0x033b1e thru 0x03be36 (0x8319 bytes) 000183 1746625941.699726 trunc from 0x03be37 to 0x005cf8 (0x3613e bytes) 000186 1746625941.708168 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000187 1746625941.782731 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000188 1746625941.849947 write 0x0293da thru 0x03484c (0xb473 bytes) 000189 1746625941.852141 fallocate 0x017f52 thru 0x0180ba (0x169 bytes) 000190 1746625941.854629 write 0x015842 thru 0x01b249 (0x5a08 bytes) 000195 1746625941.859193 punch from 0x01defe to 0x028802 (0xa905 bytes) 000196 1746625941.861624 punch from 0x006cdf to 0x00eb54 (0x7e76 bytes) 000197 1746625941.864455 read 0x0007ea thru 0x00d0a4 (0xc8bb bytes) 000199 1746625941.867854 punch from 0x014d3e to 0x016ce5 (0x1fa8 bytes) 000201 1746625941.872868 trunc from 0x03484d to 0x02e88d (0x5fbf bytes) 000202 1746625941.878035 mapwrite 0x00d6d6 thru 0x00df8f (0x8ba bytes) 000203 1746625941.885257 punch from 0x02120b to 0x028ca3 (0x7a99 bytes) 000204 1746625941.887165 write 0x020ccd thru 0x02fa5b (0xed8f bytes) 000205 1746625941.891806 trunc from 0x02fa5c to 0x015c4f (0x19e0c bytes) 000206 1746625941.898531 mapread 0x00a482 thru 0x0157e5 (0xb364 bytes) 000208 1746625941.902428 punch from 0x01512e to 0x015c4f (0xb22 bytes) 000209 1746625941.904703 mapread 0x005bd5 thru 0x0109b9 (0xade5 bytes) 000210 1746625941.905249 read 0x001f50 thru 0x00af8a (0x903b bytes) 000212 1746625941.907036 punch from 0x014240 to 0x015c4f (0x1a10 bytes) 000213 1746625941.910739 mapread 0x00963f thru 0x00f497 (0x5e59 bytes) 000214 1746625941.911391 trunc from 0x015c50 to 0x0050cf (0x10b80 bytes) 000215 1746625941.920836 read 0x002f92 thru 0x003a21 (0xa90 bytes) 000216 1746625941.921245 punch from 0x00029d to 0x0050cf (0x4e33 bytes) 000217 1746625941.923497 fallocate 0x03e231 thru 0x03ffff (0x1dcf bytes) 000218 1746625941.925965 write 0x01346a thru 0x018c99 (0x5830 bytes) 000219 1746625941.927810 fallocate 0x017dce thru 0x026360 (0xe593 bytes) 000220 1746625941.930470 write 0x01ba87 thru 0x01c3d0 (0x94a bytes) 000221 1746625941.939469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000222 1746625942.007229 write 0x0107a8 thru 0x012b08 (0x2361 bytes) 000223 1746625942.016147 write 0x00af63 thru 0x012523 (0x75c1 bytes) 000225 1746625942.024273 trunc from 0x026361 to 0x03864c (0x122ec bytes) 000226 1746625942.034702 punch from 0x031acd to 0x0351b6 (0x36ea bytes) 000228 1746625942.037374 mapread 0x016e54 thru 0x025f91 (0xf13e bytes) 000232 1746625942.049675 punch from 0x022d3d to 0x0270dd (0x43a1 bytes) 000233 1746625942.052677 trunc from 0x03864d to 0x007031 (0x3161b bytes) 000235 1746625942.061087 punch from 0x000bc9 to 0x007031 (0x6469 bytes) 000236 1746625942.064906 read 0x004975 thru 0x005861 (0xeed bytes) 000237 1746625942.067722 write 0x03ec73 thru 0x03ffff (0x138d bytes) 000239 1746625942.068480 fallocate 0x035ce0 thru 0x039808 (0x3b29 bytes) 000240 1746625942.071433 trunc from 0x040000 to 0x016036 (0x29fc9 bytes) 000241 1746625942.079176 mapread 0x00bc4b thru 0x014b2e (0x8ee4 bytes) 000242 1746625942.089381 punch from 0x013c83 to 0x016036 (0x23b4 bytes) 000243 1746625942.092349 read 0x009743 thru 0x015f17 (0xc7d5 bytes) 000245 1746625942.095927 read 0x00e443 thru 0x015475 (0x7033 bytes) 000246 1746625942.096337 read 0x00d7e8 thru 0x016036 (0x884f bytes) 000247 1746625942.096659 trunc from 0x016037 to 0x035c07 (0x1fbd1 bytes) 000249 1746625942.102491 fallocate 0x019f49 thru 0x024291 (0xa349 bytes) 000251 1746625942.105126 mapwrite 0x03c423 thru 0x03ffff (0x3bdd bytes) 000252 1746625942.126159 punch from 0x029f32 to 0x031fa7 (0x8076 bytes) 000254 1746625942.128910 fallocate 0x010b2b thru 0x019101 (0x85d7 bytes) 000255 1746625942.131153 punch from 0x014359 to 0x017955 (0x35fd bytes) 000256 1746625942.132866 fallocate 0x034d04 thru 0x036d24 (0x2021 bytes) 000258 1746625942.135876 trunc from 0x040000 to 0x0202b4 (0x1fd4b bytes) 000260 1746625942.141946 read 0x01fd4b thru 0x0202b4 (0x56a bytes) 000261 1746625942.142258 read 0x00ecff thru 0x00ee13 (0x115 bytes) 000263 1746625942.143068 read 0x011fc9 thru 0x0156d0 (0x3708 bytes) 000264 1746625942.147040 read 0x005f28 thru 0x010655 (0xa72e bytes) 000265 1746625942.147480 mapwrite 0x02679f thru 0x029efc (0x375e bytes) 000266 1746625942.164137 fallocate 0x005a2c thru 0x015058 (0xf62d bytes) 000268 1746625942.166440 fallocate 0x0035c8 thru 0x00bf75 (0x89ae bytes) 000269 1746625942.170355 punch from 0x01f558 to 0x029bc7 (0xa670 bytes) 000270 1746625942.172690 trunc from 0x029efd to 0x03920a (0xf30e bytes) 000272 1746625942.178266 punch from 0x023c6c to 0x0283e7 (0x477c bytes) 000273 1746625942.182345 fallocate 0x018b31 thru 0x01b751 (0x2c21 bytes) 000274 1746625942.189001 MIRROR_EXTEND to 2 mirrors 000275 1746625942.419691 fallocate 0x0392fb thru 0x03ffff (0x6d05 bytes) 000276 1746625942.438181 trunc from 0x040000 to 0x009b19 (0x364e6 bytes) 000277 1746625942.456078 fallocate 0x000c4c thru 0x006bba (0x5f6f bytes) 000279 1746625942.458290 write 0x01405d thru 0x015ad5 (0x1a79 bytes) 000282 1746625942.459171 write 0x038e6c thru 0x03a44a (0x15df bytes) 000283 1746625942.459892 mapwrite 0x01d2a2 thru 0x02a9cc (0xd72b bytes) 000284 1746625942.478724 MIRROR_EXTEND to 3 mirrors 000287 1746625942.725633 punch from 0x01175e to 0x019e5f (0x8702 bytes) 000288 1746625942.739142 punch from 0x016d98 to 0x020455 (0x96be bytes) 000290 1746625942.741389 write 0x032d36 thru 0x03f58a (0xc855 bytes) 000291 1746625942.745769 punch from 0x020c8c to 0x023aa4 (0x2e19 bytes) 000292 1746625942.749031 mapread 0x038744 thru 0x03f58a (0x6e47 bytes) 000293 1746625942.749866 punch from 0x029287 to 0x02f021 (0x5d9b bytes) 000294 1746625942.752637 write 0x03311f thru 0x0390c2 (0x5fa4 bytes) 000295 1746625942.753217 fallocate 0x0006a4 thru 0x00daa4 (0xd401 bytes) 000296 1746625942.755382 mapread 0x0054cc thru 0x01139e (0xbed3 bytes) 000297 1746625942.763585 read 0x0030c0 thru 0x012b2a (0xfa6b bytes) 000298 1746625942.763916 write 0x02004a thru 0x0211cd (0x1184 bytes) 000299 1746625942.772581 MIRROR_RESYNC 3 mirrors 000301 1746625943.082250 mapread 0x001a47 thru 0x007083 (0x563d bytes) 000302 1746625943.088920 mapread 0x022fbb thru 0x02c23d (0x9283 bytes) 000304 1746625943.095380 write 0x03e76d thru 0x03ffff (0x1893 bytes) 000305 1746625943.112629 write 0x02cdb2 thru 0x03745d (0xa6ac bytes) 000306 1746625943.117362 mapwrite 0x03a984 thru 0x03ffff (0x567c bytes) 000307 1746625943.127727 mapread 0x028d3e thru 0x036b0d (0xddd0 bytes) 000308 1746625943.128366 write 0x03efd5 thru 0x03ffff (0x102b bytes) 000311 1746625943.129740 write 0x025663 thru 0x0292a0 (0x3c3e bytes) 000313 1746625943.130709 punch from 0x032a55 to 0x03a718 (0x7cc4 bytes) 000314 1746625943.136919 mapread 0x008f13 thru 0x00de55 (0x4f43 bytes) 000316 1746625943.143070 trunc from 0x040000 to 0x023b91 (0x1c46e bytes) 000317 1746625943.154063 write 0x02e8f0 thru 0x0347a8 (0x5eb9 bytes) 000318 1746625943.155742 punch from 0x027cc9 to 0x032c4c (0xaf84 bytes) 000319 1746625943.161594 punch from 0x02b2c2 to 0x0347a8 (0x94e7 bytes) 000320 1746625943.164464 write 0x02a640 thru 0x0330f7 (0x8ab8 bytes) 000321 1746625943.173217 trunc from 0x0347a9 to 0x013325 (0x21483 bytes) 000324 1746625943.181617 trunc from 0x013326 to 0x02d451 (0x1a12c bytes) 000325 1746625943.192356 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000326 1746625943.268019 MIRROR_EXTEND to 3 mirrors 000327 1746625943.452206 fallocate 0x0230c0 thru 0x02bf41 (0x8e82 bytes) 000329 1746625943.466917 trunc from 0x02d452 to 0x0260dc (0x7375 bytes) 000330 1746625943.475318 MIRROR_RESYNC 3 mirrors 000333 1746625943.700017 read 0x021c28 thru 0x0260dc (0x44b5 bytes) 000334 1746625943.704103 trunc from 0x0260dd to 0x02310d (0x2fcf bytes) 000335 1746625943.722000 mapwrite 0x005d57 thru 0x01238a (0xc634 bytes) 000337 1746625943.750706 punch from 0x011eda to 0x0196c7 (0x77ee bytes) 000338 1746625943.754282 trunc from 0x02310e to 0x003f77 (0x1f196 bytes) 000340 1746625943.762929 fallocate 0x02ad85 thru 0x0339cb (0x8c47 bytes) 000341 1746625943.765669 read 0x01c41c thru 0x01cb4e (0x733 bytes) 000342 1746625943.771194 mapwrite 0x020a24 thru 0x02d0c5 (0xc6a2 bytes) 000343 1746625943.788289 MIRROR_RESYNC 3 mirrors 000344 1746625943.979480 trunc from 0x0339cc to 0x02cf52 (0x6a79 bytes) 000345 1746625944.005372 mapwrite 0x0146e2 thru 0x0243ed (0xfd0c bytes) 000346 1746625944.032725 mapwrite 0x000e05 thru 0x008880 (0x7a7c bytes) 000347 1746625944.054221 fallocate 0x00ef8b thru 0x00f89f (0x915 bytes) 000348 1746625944.057469 trunc from 0x02cf53 to 0x03693d (0x99eb bytes) 000349 1746625944.069302 mapwrite 0x010b76 thru 0x016596 (0x5a21 bytes) 000350 1746625944.084200 punch from 0x026869 to 0x02d2a7 (0x6a3f bytes) 000351 1746625944.089165 mapwrite 0x02d7f2 thru 0x038a96 (0xb2a5 bytes) 000352 1746625944.113132 punch from 0x019fcc to 0x02918d (0xf1c2 bytes) 000354 1746625944.117280 punch from 0x0342bd to 0x035b9d (0x18e1 bytes) 000355 1746625944.122512 trunc from 0x038a97 to 0x01143c (0x2765a bytes) 000356 1746625944.134520 mapwrite 0x037ee2 thru 0x03f8a1 (0x79c0 bytes) 000357 1746625944.156850 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000358 1746625944.236317 read 0x01cf4d thru 0x01cf6e (0x22 bytes) 000361 1746625944.239699 mapread 0x01b212 thru 0x0218fe (0x66ed bytes) 000362 1746625944.248275 mapwrite 0x000e1a thru 0x004287 (0x346e bytes) 000363 1746625944.271223 MIRROR_RESYNC 2 mirrors 000364 1746625944.475268 mapread 0x029998 thru 0x02a86e (0xed7 bytes) 000365 1746625944.481073 trunc from 0x03f8a2 to 0x0325d2 (0xd2cf bytes) 000366 1746625944.514195 trunc from 0x0325d3 to 0x018298 (0x1a33a bytes) 000367 1746625944.521488 mapread 0x01758c thru 0x018298 (0xd0d bytes) 000368 1746625944.525373 write 0x02c51b thru 0x02f11b (0x2c01 bytes) 000369 1746625944.526518 read 0x009d98 thru 0x00a115 (0x37e bytes) 000370 1746625944.530418 punch from 0x00e034 to 0x01b511 (0xd4de bytes) 000371 1746625944.533982 write 0x01ef98 thru 0x01fcae (0xd17 bytes) 000372 1746625944.539196 mapread 0x018b90 thru 0x019fbf (0x1430 bytes) 000373 1746625944.539832 write 0x03fd46 thru 0x03ffff (0x2ba bytes) 000374 1746625944.544120 MIRROR_RESYNC 2 mirrors 000375 1746625944.781930 read 0x02a09c thru 0x02bc14 (0x1b79 bytes) 000376 1746625944.784169 fallocate 0x00abbc thru 0x010f54 (0x6399 bytes) 000377 1746625944.796809 fallocate 0x02c178 thru 0x0324ac (0x6335 bytes) 000378 1746625944.799125 mapread 0x000027 thru 0x00dcfe (0xdcd8 bytes) 000379 1746625944.815738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000380 1746625944.886731 mapwrite 0x0334d7 thru 0x03ffff (0xcb29 bytes) 000381 1746625944.910129 MIRROR_EXTEND to 2 mirrors 000387 1746625945.146208 punch from 0x00616d to 0x01069f (0xa533 bytes) 000388 1746625945.163594 read 0x03a4b8 thru 0x03ab85 (0x6ce bytes) 000389 1746625945.168631 trunc from 0x040000 to 0x00dac8 (0x32537 bytes) 000392 1746625945.187804 mapwrite 0x0322f1 thru 0x0343a8 (0x20b8 bytes) 000393 1746625945.213092 mapread 0x022fc7 thru 0x0277e2 (0x481c bytes) 000394 1746625945.220051 MIRROR_EXTEND to 3 mirrors 000395 1746625945.408722 mapread 0x0236d4 thru 0x031525 (0xde52 bytes) 000396 1746625945.432744 fallocate 0x00bccf thru 0x00d885 (0x1bb7 bytes) 000397 1746625945.453153 fallocate 0x01d0fe thru 0x026fe1 (0x9ee4 bytes) 000399 1746625945.457918 trunc from 0x0343a9 to 0x004598 (0x2fe10 bytes) 000400 1746625945.476411 fallocate 0x029ad7 thru 0x02d6ae (0x3bd8 bytes) 000402 1746625945.479139 punch from 0x000062 to 0x004598 (0x4537 bytes) 000403 1746625945.483403 mapread 0x0025e9 thru 0x004598 (0x1fb0 bytes) 000404 1746625945.487051 punch from 0x0022c9 to 0x004598 (0x22d0 bytes) 000408 1746625945.492193 mapwrite 0x0165ea thru 0x017b0e (0x1525 bytes) 000409 1746625945.515798 trunc from 0x017b0f to 0x027496 (0xf988 bytes) 000410 1746625945.522811 mapread 0x01a60e thru 0x023ea3 (0x9896 bytes) 000411 1746625945.530268 MIRROR_EXTEND to 4 mirrors 000412 1746625945.717382 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000413 1746625945.829859 write 0x03dfbd thru 0x03ffff (0x2043 bytes) 000414 1746625945.845622 mapread 0x03dc77 thru 0x03ffff (0x2389 bytes) 000415 1746625945.846022 trunc from 0x040000 to 0x03e15b (0x1ea4 bytes) 000416 1746625945.861805 mapread 0x007071 thru 0x01144f (0xa3df bytes) 000418 1746625945.872772 mapwrite 0x0286ea thru 0x02e49a (0x5db1 bytes) 000419 1746625945.885853 mapread 0x0248ae thru 0x02697c (0x20cf bytes) 000420 1746625945.886437 mapread 0x030d5e thru 0x03a3e2 (0x9685 bytes) 000421 1746625945.893064 read 0x036f4a thru 0x03e15b (0x7212 bytes) 000422 1746625945.893502 fallocate 0x02057b thru 0x02b1a4 (0xac2a bytes) 000423 1746625945.896196 write 0x030518 thru 0x03336c (0x2e55 bytes) 000424 1746625945.897938 write 0x00b49a thru 0x0156f3 (0xa25a bytes) 000425 1746625945.900927 mapwrite 0x01abce thru 0x0217b6 (0x6be9 bytes) 000426 1746625945.912950 mapread 0x01e5ce thru 0x02aa80 (0xc4b3 bytes) 000427 1746625945.913500 write 0x027a71 thru 0x031595 (0x9b25 bytes) 000428 1746625945.916214 MIRROR_RESYNC 3 mirrors 000429 1746625946.179045 punch from 0x00d15b to 0x00dd63 (0xc09 bytes) 000430 1746625946.191063 read 0x03bb63 thru 0x03e15b (0x25f9 bytes) 000431 1746625946.193406 fallocate 0x00d572 thru 0x01b527 (0xdfb6 bytes) 000432 1746625946.195157 fallocate 0x013547 thru 0x01aeb3 (0x796d bytes) 000433 1746625946.196973 punch from 0x01c926 to 0x0287d6 (0xbeb1 bytes) 000434 1746625946.198873 fallocate 0x038212 thru 0x03f0df (0x6ece bytes) 000436 1746625946.201858 read 0x0044bd thru 0x007142 (0x2c86 bytes) 000437 1746625946.206718 fallocate 0x02be25 thru 0x03a1fb (0xe3d7 bytes) 000438 1746625946.208730 mapwrite 0x00204f thru 0x009200 (0x71b2 bytes) 000439 1746625946.219268 read 0x0212fc thru 0x02b7b7 (0xa4bc bytes) 000441 1746625946.220113 trunc from 0x03e15c to 0x03c95e (0x17fd bytes) 000442 1746625946.227026 read 0x020662 thru 0x02ecb2 (0xe651 bytes) 000443 1746625946.227525 write 0x023c47 thru 0x0333ea (0xf7a4 bytes) 000446 1746625946.229303 fallocate 0x00c785 thru 0x00d761 (0xfdd bytes) 000447 1746625946.231680 mapread 0x008acd thru 0x01615e (0xd692 bytes) 000448 1746625946.232328 write 0x011d7a thru 0x020efd (0xf184 bytes) 000449 1746625946.234429 mapread 0x0384dc thru 0x03b739 (0x325e bytes) 000450 1746625946.235013 trunc from 0x03c95f to 0x011040 (0x2b91e bytes) 000454 1746625946.242191 fallocate 0x01183c thru 0x01a020 (0x87e5 bytes) 000455 1746625946.244425 read 0x009bbd thru 0x0140d1 (0xa515 bytes) 000456 1746625946.249157 write 0x00c67d thru 0x0128b3 (0x6237 bytes) 000457 1746625946.250246 fallocate 0x00db43 thru 0x014ec1 (0x737f bytes) 000458 1746625946.255324 write 0x03614b thru 0x03d0d6 (0x6f8c bytes) 000459 1746625946.257928 write 0x0263ef thru 0x026e0b (0xa1d bytes) 000461 1746625946.263392 trunc from 0x03d0d7 to 0x032f94 (0xa142 bytes) 000462 1746625946.271352 fallocate 0x00c6b2 thru 0x01aeac (0xe7fb bytes) 000463 1746625946.273153 mapread 0x02a760 thru 0x032f94 (0x8835 bytes) 000465 1746625946.274192 mapread 0x012e28 thru 0x01f790 (0xc969 bytes) 000468 1746625946.276791 write 0x02e9f5 thru 0x033327 (0x4933 bytes) 000469 1746625946.277638 read 0x02fde1 thru 0x033327 (0x3547 bytes) 000470 1746625946.277994 read 0x00a6bb thru 0x01993e (0xf284 bytes) 000471 1746625946.278356 trunc from 0x033328 to 0x01de1b (0x1550c bytes) 000472 1746625946.289087 read 0x007905 thru 0x00ad3e (0x343a bytes) 000473 1746625946.289470 punch from 0x0163e9 to 0x01de1b (0x7a33 bytes) 000474 1746625946.291821 mapwrite 0x01b830 thru 0x02ab0f (0xf2e0 bytes) 000475 1746625946.317899 mapwrite 0x0286dd thru 0x034cc5 (0xc5e9 bytes) 000476 1746625946.339202 fallocate 0x025e95 thru 0x03105c (0xb1c8 bytes) 000478 1746625946.341885 write 0x01f333 thru 0x01ffa6 (0xc74 bytes) 000479 1746625946.342651 mapwrite 0x012d5a thru 0x01f7f0 (0xca97 bytes) 000481 1746625946.353212 MIRROR_RESYNC 3 mirrors 000483 1746625946.570331 mapread 0x028da2 thru 0x034cc5 (0xbf24 bytes) 000484 1746625946.580855 write 0x01a81e thru 0x028599 (0xdd7c bytes) 000485 1746625946.596457 fallocate 0x003b5b thru 0x00609d (0x2543 bytes) 000486 1746625946.600680 MIRROR_RESYNC 3 mirrors 000487 1746625946.803470 read 0x01ad43 thru 0x026d2a (0xbfe8 bytes) 000488 1746625946.808243 read 0x00b4d2 thru 0x0107ea (0x5319 bytes) 000489 1746625946.812131 fallocate 0x001de5 thru 0x00b712 (0x992e bytes) 000490 1746625946.828363 write 0x03d3f0 thru 0x03ffff (0x2c10 bytes) 000491 1746625946.829149 mapread 0x029460 thru 0x039115 (0xfcb6 bytes) 000492 1746625946.852445 mapwrite 0x031975 thru 0x034aef (0x317b bytes) 000493 1746625946.862134 mapread 0x017956 thru 0x017ebc (0x567 bytes) 000495 1746625946.865130 fallocate 0x0255c2 thru 0x02ccb9 (0x76f8 bytes) 000496 1746625946.867228 read 0x018a7b thru 0x020aa7 (0x802d bytes) 000497 1746625946.871872 punch from 0x004387 to 0x006bce (0x2848 bytes) 000498 1746625946.874284 write 0x01a34e thru 0x023920 (0x95d3 bytes) 000499 1746625946.875218 mapwrite 0x005eac thru 0x0060ae (0x203 bytes) 000500 1746625946.890503 mapread 0x029c8d thru 0x036b7f (0xcef3 bytes) 000501 1746625946.891162 fallocate 0x0189bf thru 0x019933 (0xf75 bytes) 000502 1746625946.893515 write 0x02fe2d thru 0x03f86e (0xfa42 bytes) 000503 1746625946.894743 mapwrite 0x0331b6 thru 0x03f571 (0xc3bc bytes) 000504 1746625946.905491 write 0x03a8b5 thru 0x03ffff (0x574b bytes) 000505 1746625946.906273 trunc from 0x040000 to 0x0278e1 (0x1871e bytes) 000506 1746625946.924701 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000508 1746625946.987954 mapwrite 0x00dd48 thru 0x01347c (0x5735 bytes) 000509 1746625947.002756 write 0x00c85f thru 0x00fa3d (0x31df bytes) 000510 1746625947.004659 mapwrite 0x016258 thru 0x022174 (0xbf1d bytes) 000511 1746625947.019665 mapread 0x01ca07 thru 0x01ff09 (0x3503 bytes) 000512 1746625947.020138 read 0x01eee1 thru 0x0278e1 (0x8a01 bytes) 000513 1746625947.020606 mapwrite 0x00f45b thru 0x019efe (0xaaa4 bytes) 000514 1746625947.029113 fallocate 0x00b9fe thru 0x01ad64 (0xf367 bytes) 000515 1746625947.031641 write 0x0380e0 thru 0x038505 (0x426 bytes) 000516 1746625947.032239 punch from 0x009b48 to 0x01772f (0xdbe8 bytes) 000517 1746625947.034719 read 0x037a1b thru 0x038505 (0xaeb bytes) 000519 1746625947.039730 mapwrite 0x03185b thru 0x037349 (0x5aef bytes) 000520 1746625947.046326 mapwrite 0x036c9b thru 0x03ebc8 (0x7f2e bytes) 000521 1746625947.065494 read 0x01dfc2 thru 0x02012b (0x216a bytes) 000523 1746625947.066193 write 0x03b546 thru 0x03ffff (0x4aba bytes) 000524 1746625947.066913 trunc from 0x040000 to 0x01884c (0x277b3 bytes) 000525 1746625947.074365 trunc from 0x01884d to 0x01307e (0x57ce bytes) 000526 1746625947.079768 read 0x002264 thru 0x00d4b0 (0xb24d bytes) 000527 1746625947.080203 fallocate 0x01a1da thru 0x025cb0 (0xbad7 bytes) 000529 1746625947.082739 read 0x00977b thru 0x01307e (0x9904 bytes) 000530 1746625947.083116 fallocate 0x036df1 thru 0x03e994 (0x7ba4 bytes) 000531 1746625947.084752 punch from 0x02765e to 0x0318f0 (0xa293 bytes) 000532 1746625947.088033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000533 1746625947.134201 punch from 0x02d5bc to 0x0382c5 (0xad0a bytes) 000534 1746625947.135944 trunc from 0x03e995 to 0x035cbc (0x8cd8 bytes) 000535 1746625947.142130 mapwrite 0x018ce5 thru 0x025698 (0xc9b4 bytes) 000536 1746625947.162288 mapwrite 0x02f805 thru 0x0342bd (0x4ab9 bytes) 000537 1746625947.173252 mapwrite 0x0314b2 thru 0x035802 (0x4351 bytes) 000538 1746625947.180832 mapread 0x00d1c1 thru 0x00d618 (0x458 bytes) 000540 1746625947.184453 fallocate 0x0023be thru 0x006e93 (0x4ad6 bytes) 000542 1746625947.187336 trunc from 0x035cbd to 0x03a8b8 (0x4bfc bytes) 000543 1746625947.192810 write 0x02063f thru 0x02a96d (0xa32f bytes) 000546 1746625947.196375 MIRROR_EXTEND to 2 mirrors 000548 1746625947.349336 mapread 0x008a42 thru 0x009ee9 (0x14a8 bytes) 000549 1746625947.353123 read 0x037e6d thru 0x039484 (0x1618 bytes) 000551 1746625947.357585 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000553 1746625947.397770 mapread 0x01e4ce thru 0x0205e3 (0x2116 bytes) 000554 1746625947.403450 mapwrite 0x0274ad thru 0x028e96 (0x19ea bytes) 000555 1746625947.418705 punch from 0x013c46 to 0x013f48 (0x303 bytes) 000556 1746625947.420813 fallocate 0x036661 thru 0x03d47f (0x6e1f bytes) 000559 1746625947.423173 read 0x012ad5 thru 0x013259 (0x785 bytes) 000560 1746625947.425700 punch from 0x03b09e to 0x03d47f (0x23e2 bytes) 000561 1746625947.427458 fallocate 0x010f47 thru 0x01ed5d (0xde17 bytes) 000562 1746625947.431258 MIRROR_EXTEND to 2 mirrors 000563 1746625947.587495 mapread 0x025c71 thru 0x02fdca (0xa15a bytes) 000564 1746625947.593445 mapwrite 0x032c71 thru 0x036cbe (0x404e bytes) 000565 1746625947.630913 trunc from 0x03d480 to 0x007920 (0x35b5f bytes) 000566 1746625947.648392 fallocate 0x015c31 thru 0x01a6ad (0x4a7d bytes) 000567 1746625947.650333 trunc from 0x007921 to 0x01eca0 (0x17380 bytes) 000568 1746625947.660682 trunc from 0x01eca1 to 0x035a36 (0x16d96 bytes) 000569 1746625947.666778 trunc from 0x035a37 to 0x01002d (0x25a09 bytes) 000571 1746625947.672568 mapread 0x006fa8 thru 0x007bf4 (0xc4d bytes) 000572 1746625947.676224 mapwrite 0x03e758 thru 0x03efa2 (0x84b bytes) 000573 1746625947.690703 punch from 0x03b925 to 0x03efa2 (0x367e bytes) 000574 1746625947.693729 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000575 1746625947.748342 trunc from 0x03efa3 to 0x036387 (0x8c1b bytes) 000576 1746625947.760697 mapwrite 0x00f1e9 thru 0x01dc7d (0xea95 bytes) 000577 1746625947.785768 punch from 0x00fca3 to 0x01531f (0x567d bytes) 000578 1746625947.788484 read 0x02530d thru 0x0292da (0x3fce bytes) 000579 1746625947.791657 mapwrite 0x01900c thru 0x02084e (0x7843 bytes) 000580 1746625947.799670 punch from 0x0319a2 to 0x036387 (0x49e6 bytes) 000581 1746625947.802075 trunc from 0x036388 to 0x01598b (0x209fc bytes) 000582 1746625947.811070 trunc from 0x01598c to 0x02173a (0xbdaf bytes) 000583 1746625947.816851 trunc from 0x02173b to 0x036557 (0x14e1d bytes) 000585 1746625947.823725 mapwrite 0x022a0e thru 0x0261cc (0x37bf bytes) 000586 1746625947.837996 MIRROR_EXTEND to 2 mirrors 000587 1746625948.007942 mapwrite 0x012876 thru 0x019ad9 (0x7264 bytes) 000588 1746625948.043808 mapread 0x00f3d5 thru 0x01d5db (0xe207 bytes) 000590 1746625948.047628 fallocate 0x02cc9a thru 0x03720b (0xa572 bytes) 000593 1746625948.053001 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000594 1746625948.099388 MIRROR_EXTEND to 2 mirrors 000595 1746625948.290035 write 0x010f5f thru 0x014ab8 (0x3b5a bytes) 000596 1746625948.301490 trunc from 0x03720c to 0x012e64 (0x243a7 bytes) 000597 1746625948.306950 fallocate 0x025619 thru 0x0303da (0xadc2 bytes) 000598 1746625948.308733 read 0x007df4 thru 0x00f6ad (0x78ba bytes) 000599 1746625948.311374 punch from 0x010ed0 to 0x012e64 (0x1f95 bytes) 000600 1746625948.314607 punch from 0x00e25b to 0x012e64 (0x4c0a bytes) 000601 1746625948.316148 trunc from 0x012e65 to 0x035fc8 (0x23164 bytes) 000602 1746625948.320767 fallocate 0x0386aa thru 0x03e66d (0x5fc4 bytes) 000603 1746625948.322617 mapwrite 0x019015 thru 0x0281f8 (0xf1e4 bytes) 000604 1746625948.336264 fallocate 0x01847f thru 0x02606e (0xdbf0 bytes) 000605 1746625948.338139 punch from 0x011fa4 to 0x017936 (0x5993 bytes) 000608 1746625948.340566 fallocate 0x0016cf thru 0x00b508 (0x9e3a bytes) 000609 1746625948.342320 punch from 0x00ea64 to 0x019abd (0xb05a bytes) 000610 1746625948.344988 write 0x0379e7 thru 0x03aed7 (0x34f1 bytes) 000611 1746625948.347773 trunc from 0x03e66e to 0x039cd1 (0x499c bytes) 000612 1746625948.352133 mapwrite 0x006440 thru 0x00ae69 (0x4a2a bytes) 000613 1746625948.361441 punch from 0x008fa6 to 0x016827 (0xd882 bytes) 000614 1746625948.363013 trunc from 0x039cd2 to 0x035d57 (0x3f7a bytes) 000615 1746625948.367416 trunc from 0x035d58 to 0x024de8 (0x10f6f bytes) 000616 1746625948.371753 fallocate 0x03b63a thru 0x03ffff (0x49c6 bytes) 000617 1746625948.373367 mapwrite 0x02baf0 thru 0x03815b (0xc66c bytes) 000618 1746625948.390711 fallocate 0x023205 thru 0x02b869 (0x8665 bytes) 000619 1746625948.393945 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000620 1746625948.441297 fallocate 0x020cf2 thru 0x02d718 (0xca27 bytes) 000621 1746625948.443569 write 0x020057 thru 0x024c08 (0x4bb2 bytes) 000623 1746625948.451461 mapread 0x01747a thru 0x020e95 (0x9a1c bytes) 000624 1746625948.455175 fallocate 0x01b89a thru 0x01fe77 (0x45de bytes) 000625 1746625948.457794 mapread 0x020791 thru 0x02c41a (0xbc8a bytes) 000626 1746625948.461222 punch from 0x011f59 to 0x016ea2 (0x4f4a bytes) 000627 1746625948.463329 punch from 0x01f2e9 to 0x023db3 (0x4acb bytes) 000628 1746625948.468883 punch from 0x017f43 to 0x021319 (0x93d7 bytes) 000632 1746625948.472244 fallocate 0x01090d thru 0x01da04 (0xd0f8 bytes) 000633 1746625948.474649 write 0x028b37 thru 0x02c043 (0x350d bytes) 000634 1746625948.477083 MIRROR_EXTEND to 2 mirrors 000635 1746625948.621341 mapread 0x03645d thru 0x03815b (0x1cff bytes) 000636 1746625948.624328 mapwrite 0x037f52 thru 0x03ffff (0x80ae bytes) 000637 1746625948.656998 trunc from 0x040000 to 0x027a1e (0x185e1 bytes) 000638 1746625948.663579 mapread 0x01c836 thru 0x023003 (0x67ce bytes) 000639 1746625948.668391 read 0x026791 thru 0x027a1e (0x128e bytes) 000640 1746625948.671524 MIRROR_RESYNC 2 mirrors 000641 1746625948.783941 fallocate 0x000bc4 thru 0x00f1e4 (0xe621 bytes) 000643 1746625948.793192 punch from 0x006705 to 0x0106eb (0x9fe7 bytes) 000644 1746625948.794999 write 0x01a27b thru 0x026199 (0xbf1f bytes) 000645 1746625948.800492 MIRROR_RESYNC 2 mirrors 000647 1746625948.926111 mapwrite 0x02cee2 thru 0x03019a (0x32b9 bytes) 000648 1746625948.954019 trunc from 0x03019b to 0x020c5d (0xf53d bytes) 000649 1746625948.960436 mapread 0x00951d thru 0x0109f4 (0x74d8 bytes) 000650 1746625948.963960 fallocate 0x0124a0 thru 0x012ac6 (0x627 bytes) 000651 1746625948.965820 mapread 0x009627 thru 0x016271 (0xcc4b bytes) 000652 1746625948.966435 trunc from 0x020c5e to 0x01f821 (0x143c bytes) 000653 1746625948.970907 fallocate 0x0186c8 thru 0x0206d7 (0x8010 bytes) 000654 1746625948.974611 MIRROR_EXTEND to 3 mirrors 000656 1746625949.117380 punch from 0x01a979 to 0x01f821 (0x4ea9 bytes) 000657 1746625949.129835 fallocate 0x013dbb thru 0x021670 (0xd8b6 bytes) 000658 1746625949.132082 fallocate 0x00800d thru 0x00d444 (0x5438 bytes) 000659 1746625949.134058 mapwrite 0x023bf7 thru 0x0275cb (0x39d5 bytes) 000660 1746625949.155451 mapread 0x00a8c0 thru 0x013c32 (0x9373 bytes) 000662 1746625949.161634 write 0x03f494 thru 0x03ffff (0xb6c bytes) 000663 1746625949.162270 mapwrite 0x008f48 thru 0x016ddd (0xde96 bytes) 000664 1746625949.171796 mapwrite 0x02713a thru 0x0311bb (0xa082 bytes) 000665 1746625949.185075 mapwrite 0x00f0ed thru 0x01ea3a (0xf94e bytes) 000666 1746625949.194818 fallocate 0x038018 thru 0x03d86b (0x5854 bytes) 000667 1746625949.197173 punch from 0x024664 to 0x031d9a (0xd737 bytes) 000669 1746625949.199290 write 0x025363 thru 0x02a656 (0x52f4 bytes) 000673 1746625949.204755 read 0x0153c0 thru 0x01f5e2 (0xa223 bytes) 000675 1746625949.205505 mapwrite 0x027811 thru 0x02a9e5 (0x31d5 bytes) 000676 1746625949.214078 MIRROR_RESYNC 3 mirrors 000677 1746625949.414669 trunc from 0x040000 to 0x008474 (0x37b8b bytes) 000678 1746625949.431087 read 0x002157 thru 0x008474 (0x631e bytes) 000679 1746625949.433837 fallocate 0x015ee0 thru 0x01f7b7 (0x98d8 bytes) 000680 1746625949.435657 punch from 0x004d37 to 0x011548 (0xc812 bytes) 000681 1746625949.437577 write 0x01a44e thru 0x022e6b (0x8a1e bytes) 000684 1746625949.439948 mapread 0x015cf8 thru 0x01e420 (0x8729 bytes) 000685 1746625949.446905 MIRROR_RESYNC 3 mirrors 000686 1746625949.570942 write 0x02206e thru 0x027207 (0x519a bytes) 000687 1746625949.581868 trunc from 0x027208 to 0x027a2f (0x828 bytes) 000688 1746625949.593108 fallocate 0x0368b7 thru 0x03aacf (0x4219 bytes) 000690 1746625949.595576 write 0x01a9dc thru 0x0228d5 (0x7efa bytes) 000691 1746625949.597437 mapread 0x01139f thru 0x015069 (0x3ccb bytes) 000693 1746625949.600652 read 0x000101 thru 0x00dcc0 (0xdbc0 bytes) 000694 1746625949.603870 trunc from 0x027a30 to 0x03df34 (0x16505 bytes) 000696 1746625949.608255 write 0x03c42a thru 0x03ffff (0x3bd6 bytes) 000698 1746625949.611484 mapwrite 0x012044 thru 0x0182bd (0x627a bytes) 000700 1746625949.620511 punch from 0x02f7b3 to 0x03ee3e (0xf68c bytes) 000701 1746625949.622691 fallocate 0x029ba8 thru 0x02f74f (0x5ba8 bytes) 000702 1746625949.624964 trunc from 0x040000 to 0x02a192 (0x15e6d bytes) 000705 1746625949.630711 trunc from 0x02a193 to 0x0026b6 (0x27adc bytes) 000706 1746625949.635787 read 0x0011d0 thru 0x0026b6 (0x14e7 bytes) 000710 1746625949.637228 fallocate 0x033fa4 thru 0x03ab7a (0x6bd7 bytes) 000711 1746625949.639802 write 0x031225 thru 0x03c556 (0xb332 bytes) 000712 1746625949.644669 trunc from 0x03c557 to 0x021ffc (0x1a55a bytes) 000714 1746625949.651376 trunc from 0x021ffd to 0x02426f (0x2273 bytes) 000715 1746625949.655521 write 0x00d77f thru 0x016f7a (0x97fc bytes) 000716 1746625949.656429 read 0x0193a2 thru 0x022245 (0x8ea4 bytes) 000717 1746625949.659431 punch from 0x0222c6 to 0x02426f (0x1faa bytes) 000718 1746625949.662741 write 0x016798 thru 0x017610 (0xe79 bytes) 000721 1746625949.664003 read 0x0162de thru 0x023553 (0xd276 bytes) 000722 1746625949.666522 fallocate 0x03dfac thru 0x03ffff (0x2054 bytes) 000723 1746625949.668654 mapread 0x01006a thru 0x01b5a7 (0xb53e bytes) 000724 1746625949.671386 MIRROR_RESYNC 3 mirrors 000725 1746625949.816490 write 0x0358fd thru 0x03f424 (0x9b28 bytes) 000726 1746625949.824923 mapread 0x01e10a thru 0x01fe55 (0x1d4c bytes) 000727 1746625949.827829 mapread 0x033fff thru 0x0391e0 (0x51e2 bytes) 000728 1746625949.830408 punch from 0x01d71a to 0x02c64f (0xef36 bytes) 000729 1746625949.835055 MIRROR_RESYNC 3 mirrors 000730 1746625949.976130 read 0x03e44b thru 0x03f424 (0xfda bytes) 000731 1746625949.977768 punch from 0x02be3d to 0x036ecc (0xb090 bytes) 000732 1746625949.988438 write 0x01b7bd thru 0x022971 (0x71b5 bytes) 000733 1746625949.993294 MIRROR_EXTEND to 4 mirrors 000734 1746625950.136346 fallocate 0x0109db thru 0x01fd8a (0xf3b0 bytes) 000735 1746625950.146507 punch from 0x01ee84 to 0x01f941 (0xabe bytes) 000736 1746625950.148248 mapread 0x00caa6 thru 0x00f517 (0x2a72 bytes) 000737 1746625950.154277 punch from 0x007fb6 to 0x01747b (0xf4c6 bytes) 000738 1746625950.156514 punch from 0x02d315 to 0x02e2f4 (0xfe0 bytes) 000739 1746625950.158016 punch from 0x020b74 to 0x0290cb (0x8558 bytes) 000740 1746625950.160014 punch from 0x02b7c6 to 0x02fa5b (0x4296 bytes) 000741 1746625950.161608 fallocate 0x0188c0 thru 0x01ab2f (0x2270 bytes) 000742 1746625950.165105 MIRROR_RESYNC 4 mirrors 000744 1746625950.354140 trunc from 0x03f425 to 0x03b048 (0x43dc bytes) 000745 1746625950.367319 punch from 0x0011ec to 0x0013bd (0x1d2 bytes) 000746 1746625950.369036 mapread 0x024305 thru 0x02aebf (0x6bbb bytes) 000747 1746625950.374694 write 0x0043f9 thru 0x0143eb (0xfff3 bytes) 000748 1746625950.379276 mapwrite 0x004fa0 thru 0x0068a7 (0x1908 bytes) 000749 1746625950.385893 fallocate 0x00114b thru 0x005ab4 (0x496a bytes) 000750 1746625950.388074 punch from 0x00c182 to 0x00c939 (0x7b8 bytes) 000751 1746625950.389851 read 0x023cfd thru 0x024bcc (0xed0 bytes) 000752 1746625950.390199 write 0x017f40 thru 0x01cfed (0x50ae bytes) 000753 1746625950.394763 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000754 1746625950.450878 trunc from 0x03b049 to 0x03fafd (0x4ab5 bytes) 000756 1746625950.457896 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000757 1746625950.516203 mapwrite 0x02c430 thru 0x03c046 (0xfc17 bytes) 000758 1746625950.535591 fallocate 0x03cf02 thru 0x03ffff (0x30fe bytes) 000760 1746625950.538174 mapwrite 0x01af5f thru 0x0239a5 (0x8a47 bytes) 000761 1746625950.552822 mapread 0x0004fc thru 0x009c14 (0x9719 bytes) 000762 1746625950.559897 punch from 0x015680 to 0x01c592 (0x6f13 bytes) 000764 1746625950.564442 MIRROR_RESYNC 2 mirrors 000765 1746625950.734096 mapread 0x00ee9b thru 0x01b27e (0xc3e4 bytes) 000766 1746625950.743070 fallocate 0x0035ee thru 0x00789b (0x42ae bytes) 000767 1746625950.753835 punch from 0x010113 to 0x01aba3 (0xaa91 bytes) 000768 1746625950.755839 mapwrite 0x0232e7 thru 0x02bbc0 (0x88da bytes) 000769 1746625950.769067 write 0x030f21 thru 0x03bd42 (0xae22 bytes) 000770 1746625950.775801 read 0x027365 thru 0x03429e (0xcf3a bytes) 000771 1746625950.776245 fallocate 0x02de2a thru 0x03a0af (0xc286 bytes) 000772 1746625950.779541 punch from 0x016471 to 0x019655 (0x31e5 bytes) 000773 1746625950.781713 mapread 0x00ece9 thru 0x01765f (0x8977 bytes) 000774 1746625950.788168 mapread 0x01300f thru 0x021b37 (0xeb29 bytes) 000775 1746625950.791399 MIRROR_EXTEND to 3 mirrors 000776 1746625950.961924 write 0x01a15b thru 0x0291be (0xf064 bytes) 000777 1746625950.977029 punch from 0x00d21a to 0x01321d (0x6004 bytes) 000778 1746625950.980400 MIRROR_EXTEND to 4 mirrors 000779 1746625951.118246 fallocate 0x03c68e thru 0x03c73d (0xb0 bytes) 000781 1746625951.129291 trunc from 0x040000 to 0x00b3db (0x34c24 bytes) 000782 1746625951.135391 punch from 0x005a8a to 0x00b3db (0x5952 bytes) 000783 1746625951.137180 trunc from 0x00b3dc to 0x019a9f (0xe6c4 bytes) 000784 1746625951.141083 write 0x03bbd0 thru 0x03ffff (0x4430 bytes) 000785 1746625951.141808 write 0x001b42 thru 0x00c375 (0xa834 bytes) 000786 1746625951.146149 read 0x01de97 thru 0x027300 (0x946a bytes) 000787 1746625951.148944 fallocate 0x000942 thru 0x00b73a (0xadf9 bytes) 000788 1746625951.150900 mapwrite 0x01af3f thru 0x0269e1 (0xbaa3 bytes) 000790 1746625951.165766 write 0x028e1c thru 0x02d422 (0x4607 bytes) 000791 1746625951.172020 MIRROR_RESYNC 4 mirrors 000792 1746625951.408410 trunc from 0x040000 to 0x00c730 (0x338cf bytes) 000793 1746625951.429071 read 0x006e3a thru 0x00c730 (0x58f7 bytes) 000794 1746625951.432202 mapwrite 0x03ce31 thru 0x03ffff (0x31cf bytes) 000796 1746625951.449411 read 0x03c4dc thru 0x03ffff (0x3b24 bytes) 000797 1746625951.449790 punch from 0x03a04b to 0x03fa22 (0x59d8 bytes) 000798 1746625951.451675 write 0x00853e thru 0x010fe3 (0x8aa6 bytes) 000799 1746625951.456432 fallocate 0x03f3a7 thru 0x03ffff (0xc59 bytes) 000800 1746625951.458299 mapwrite 0x02049b thru 0x02dc67 (0xd7cd bytes) 000801 1746625951.472465 read 0x03d07a thru 0x03d5de (0x565 bytes) 000803 1746625951.473074 punch from 0x011f52 to 0x015bb4 (0x3c63 bytes) 000804 1746625951.475038 mapread 0x02c423 thru 0x0306fa (0x42d8 bytes) 000805 1746625951.475648 punch from 0x0262d4 to 0x02a310 (0x403d bytes) 000807 1746625951.477939 trunc from 0x040000 to 0x024123 (0x1bedc bytes) 000808 1746625951.483194 write 0x00b68a thru 0x013aa0 (0x8417 bytes) 000809 1746625951.486434 trunc from 0x024124 to 0x000f4c (0x231d7 bytes) 000810 1746625951.491809 read 0x000761 thru 0x000f4c (0x7ec bytes) 000811 1746625951.492185 mapwrite 0x00661a thru 0x00f528 (0x8f0f bytes) 000813 1746625951.510671 fallocate 0x01383a thru 0x0167c8 (0x2f8f bytes) 000814 1746625951.513018 read 0x00d4da thru 0x00f528 (0x204f bytes) 000815 1746625951.513483 mapwrite 0x020c01 thru 0x03054d (0xf94d bytes) 000816 1746625951.535268 write 0x029d26 thru 0x033d9a (0xa075 bytes) 000817 1746625951.536240 write 0x01fc6e thru 0x026324 (0x66b7 bytes) 000818 1746625951.538815 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000819 1746625951.611128 read 0x03376c thru 0x033d9a (0x62f bytes) 000820 1746625951.613260 read 0x0339b4 thru 0x033d9a (0x3e7 bytes) 000821 1746625951.613794 mapread 0x027d77 thru 0x033d9a (0xc024 bytes) 000822 1746625951.617659 punch from 0x00cf19 to 0x015f64 (0x904c bytes) 000825 1746625951.620159 mapwrite 0x016c4c thru 0x017d56 (0x110b bytes) 000826 1746625951.627635 MIRROR_RESYNC 3 mirrors 000827 1746625951.813774 fallocate 0x02387b thru 0x030a71 (0xd1f7 bytes) 000830 1746625951.826529 mapread 0x0092eb thru 0x00f639 (0x634f bytes) 000832 1746625951.833412 fallocate 0x020cea thru 0x02f183 (0xe49a bytes) 000833 1746625951.835288 mapwrite 0x00c72f thru 0x01409a (0x796c bytes) 000834 1746625951.845094 trunc from 0x033d9b to 0x03def4 (0xa15a bytes) 000835 1746625951.857336 mapread 0x019004 thru 0x028c51 (0xfc4e bytes) 000836 1746625951.864272 trunc from 0x03def5 to 0x027612 (0x168e2 bytes) 000837 1746625951.869534 mapread 0x00b002 thru 0x015127 (0xa126 bytes) 000838 1746625951.870144 read 0x001768 thru 0x00a700 (0x8f99 bytes) 000839 1746625951.874185 write 0x02d2a2 thru 0x031bec (0x494b bytes) 000841 1746625951.877176 MIRROR_RESYNC 3 mirrors 000842 1746625952.104909 mapread 0x00e32a thru 0x018821 (0xa4f8 bytes) 000843 1746625952.114855 fallocate 0x02a2dc thru 0x030211 (0x5f36 bytes) 000844 1746625952.130081 MIRROR_RESYNC 3 mirrors 000845 1746625952.334810 write 0x0063a6 thru 0x00a339 (0x3f94 bytes) 000847 1746625952.348086 punch from 0x01ce3d to 0x023bae (0x6d72 bytes) 000848 1746625952.349965 fallocate 0x015e56 thru 0x01e599 (0x8744 bytes) 000849 1746625952.351877 fallocate 0x02bdd2 thru 0x036927 (0xab56 bytes) 000850 1746625952.354154 mapwrite 0x023560 thru 0x026e09 (0x38aa bytes) 000851 1746625952.366103 fallocate 0x034067 thru 0x03be7b (0x7e15 bytes) 000852 1746625952.368026 trunc from 0x031bed to 0x025d2c (0xbec0 bytes) 000854 1746625952.375242 read 0x0194a2 thru 0x021b95 (0x86f4 bytes) 000857 1746625952.378962 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000858 1746625952.431471 write 0x0261e4 thru 0x035a0f (0xf82c bytes) 000859 1746625952.433723 punch from 0x02ef41 to 0x035a0f (0x6acf bytes) 000860 1746625952.438984 read 0x016052 thru 0x021990 (0xb93f bytes) 000861 1746625952.442093 trunc from 0x035a10 to 0x0300ff (0x5910 bytes) 000862 1746625952.448670 mapread 0x01aec9 thru 0x0276a3 (0xc7db bytes) 000863 1746625952.454957 read 0x016bd2 thru 0x01916a (0x2599 bytes) 000864 1746625952.455603 punch from 0x0061ea to 0x01570d (0xf524 bytes) 000865 1746625952.458923 fallocate 0x02bcbc thru 0x030826 (0x4b6b bytes) 000868 1746625952.464782 MIRROR_EXTEND to 3 mirrors 000869 1746625952.635034 trunc from 0x030827 to 0x03135a (0xb34 bytes) 000870 1746625952.656286 mapwrite 0x030d4e thru 0x03ffff (0xf2b2 bytes) 000871 1746625952.680432 punch from 0x03e7f8 to 0x03ffff (0x1808 bytes) 000872 1746625952.683736 trunc from 0x040000 to 0x03b923 (0x46dc bytes) 000873 1746625952.690847 trunc from 0x03b924 to 0x02a20b (0x11718 bytes) 000875 1746625952.697392 fallocate 0x032ab6 thru 0x039723 (0x6c6e bytes) 000876 1746625952.700090 mapread 0x00fe1f thru 0x014a1f (0x4c01 bytes) 000877 1746625952.704584 write 0x0339e1 thru 0x03d835 (0x9e55 bytes) 000878 1746625952.707828 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000880 1746625952.775341 trunc from 0x03d836 to 0x017fd7 (0x2585e bytes) 000881 1746625952.782019 read 0x00aafe thru 0x017fd7 (0xd4da bytes) 000882 1746625952.785476 fallocate 0x01e991 thru 0x01ea0a (0x7a bytes) 000883 1746625952.787974 punch from 0x0082ca to 0x008872 (0x5a9 bytes) 000884 1746625952.791703 MIRROR_RESYNC 2 mirrors 000886 1746625952.913312 MIRROR_EXTEND to 3 mirrors 000887 1746625953.062348 fallocate 0x01dd6b thru 0x02abdc (0xce72 bytes) 000888 1746625953.073365 read 0x00e46e thru 0x00f0b4 (0xc47 bytes) 000893 1746625953.077084 write 0x02fcb0 thru 0x03ca41 (0xcd92 bytes) 000894 1746625953.079955 write 0x02e20c thru 0x03d335 (0xf12a bytes) 000896 1746625953.082591 trunc from 0x03d336 to 0x00a788 (0x32bad bytes) 000898 1746625953.094537 punch from 0x00550e to 0x00a788 (0x527b bytes) 000900 1746625953.096881 read 0x006e65 thru 0x00a788 (0x3924 bytes) 000901 1746625953.100370 trunc from 0x00a789 to 0x014d90 (0xa608 bytes) 000902 1746625953.108521 fallocate 0x006079 thru 0x00f0f2 (0x907a bytes) 000903 1746625953.111038 read 0x01219a thru 0x014d90 (0x2bf7 bytes) 000905 1746625953.114947 mapread 0x00433b thru 0x00b08b (0x6d51 bytes) 000906 1746625953.115671 punch from 0x0016b0 to 0x00fcf9 (0xe64a bytes) 000907 1746625953.118842 mapwrite 0x0018ac thru 0x00685d (0x4fb2 bytes) 000908 1746625953.137282 mapwrite 0x003b0e thru 0x010ccf (0xd1c2 bytes) 000909 1746625953.158752 write 0x008c4c thru 0x00aa21 (0x1dd6 bytes) 000910 1746625953.159330 mapwrite 0x000709 thru 0x000792 (0x8a bytes) 000911 1746625953.165995 mapread 0x0007ad thru 0x00ffb0 (0xf804 bytes) 000913 1746625953.166966 read 0x012ee8 thru 0x014d90 (0x1ea9 bytes) 000915 1746625953.167605 trunc from 0x014d91 to 0x00a803 (0xa58d bytes) 000916 1746625953.172169 trunc from 0x00a804 to 0x002c4c (0x7bb7 bytes) 000918 1746625953.176957 read 0x000de6 thru 0x002c4c (0x1e67 bytes) 000919 1746625953.177392 fallocate 0x01f475 thru 0x02ef06 (0xfa92 bytes) 000922 1746625953.180286 trunc from 0x02ef07 to 0x0213a9 (0xdb5d bytes) 000923 1746625953.185944 read 0x0017d7 thru 0x0095eb (0x7e15 bytes) 000924 1746625953.189765 punch from 0x00b840 to 0x016f09 (0xb6ca bytes) 000926 1746625953.193627 MIRROR_RESYNC 3 mirrors 000927 1746625953.351893 mapread 0x01a51c thru 0x01ef82 (0x4a67 bytes) 000928 1746625953.357771 read 0x00835f thru 0x00f383 (0x7025 bytes) 000929 1746625953.360805 trunc from 0x0213aa to 0x01c604 (0x4da5 bytes) 000930 1746625953.379876 mapwrite 0x0086bd thru 0x012d08 (0xa64c bytes) 000931 1746625953.400927 fallocate 0x01d6c9 thru 0x028a0e (0xb346 bytes) 000932 1746625953.404633 mapread 0x011f57 thru 0x0165fc (0x46a6 bytes) 000933 1746625953.405771 trunc from 0x01c605 to 0x02cf5b (0x10957 bytes) 000934 1746625953.414296 punch from 0x016932 to 0x022296 (0xb965 bytes) 000935 1746625953.416361 write 0x03624d thru 0x0375ef (0x13a3 bytes) 000936 1746625953.419483 MIRROR_EXTEND to 4 mirrors 000937 1746625953.632805 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000939 1746625953.750895 fallocate 0x03c0dc thru 0x03ffff (0x3f24 bytes) 000940 1746625953.769842 punch from 0x018380 to 0x020380 (0x8001 bytes) 000941 1746625953.771938 fallocate 0x01052f thru 0x01516a (0x4c3c bytes) 000942 1746625953.774041 fallocate 0x0052c1 thru 0x00a132 (0x4e72 bytes) 000943 1746625953.776103 trunc from 0x040000 to 0x0078ff (0x38700 bytes) 000944 1746625953.788594 mapwrite 0x02c0ce thru 0x03479b (0x86ce bytes) 000945 1746625953.813750 mapwrite 0x007125 thru 0x01636e (0xf24a bytes) 000946 1746625953.828354 trunc from 0x03479c to 0x029a31 (0xad6a bytes) 000947 1746625953.835982 MIRROR_RESYNC 3 mirrors 000948 1746625954.011667 write 0x0144d8 thru 0x01e1df (0x9d08 bytes) 000949 1746625954.027783 mapwrite 0x002fda thru 0x009225 (0x624c bytes) 000950 1746625954.042261 write 0x0127c3 thru 0x01c44d (0x9c8b bytes) 000951 1746625954.046021 write 0x010f2d thru 0x01a3e6 (0x94ba bytes) 000954 1746625954.047107 read 0x00c9e9 thru 0x00e16d (0x1785 bytes) 000955 1746625954.047541 mapread 0x020545 thru 0x029a31 (0x94ed bytes) 000957 1746625954.055230 mapwrite 0x009bc1 thru 0x00c8c4 (0x2d04 bytes) 000958 1746625954.065485 punch from 0x009e67 to 0x00e519 (0x46b3 bytes) 000959 1746625954.068131 punch from 0x020812 to 0x029a31 (0x9220 bytes) 000963 1746625954.071723 trunc from 0x029a32 to 0x028e6a (0xbc7 bytes) 000964 1746625954.086834 write 0x02dfcb thru 0x034564 (0x659a bytes) 000967 1746625954.088652 punch from 0x025fce to 0x02961d (0x3650 bytes) 000968 1746625954.091236 mapread 0x029c1a thru 0x034564 (0xa94b bytes) 000969 1746625954.095922 punch from 0x02a040 to 0x031d1d (0x7cde bytes) 000970 1746625954.104814 MIRROR_RESYNC 3 mirrors 000971 1746625954.313315 read 0x00827f thru 0x00fa01 (0x7783 bytes) 000972 1746625954.316068 trunc from 0x034565 to 0x0290cf (0xb495 bytes) 000973 1746625954.335825 MIRROR_EXTEND to 4 mirrors 000974 1746625954.505959 fallocate 0x03233c thru 0x038d85 (0x6a4a bytes) 000975 1746625954.519781 punch from 0x01f26f to 0x0290cf (0x9e61 bytes) 000976 1746625954.521737 fallocate 0x031228 thru 0x038881 (0x765a bytes) 000977 1746625954.526211 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x81c:0x0]: Invalid argument (22) 000978 1746625954.597914 trunc from 0x0290d0 to 0x01bec4 (0xd20b bytes) 000979 1746625954.612987 fallocate 0x00acbf thru 0x00ad89 (0xcb bytes) 000980 1746625954.615178 mapread 0x0176c6 thru 0x01bec4 (0x47ff bytes) 000981 1746625954.623836 trunc from 0x01bec5 to 0x002033 (0x19e91 bytes) 000982 1746625954.633650 MIRROR_RESYNC 3 mirrors 000983 1746625954.768365 mapwrite 0x039626 thru 0x03ffff (0x69da bytes) 000984 1746625954.813287 read 0x00e565 thru 0x01c075 (0xdb11 bytes) 000985 1746625954.819396 trunc from 0x040000 to 0x02f1a1 (0x10e5e bytes) 000986 1746625954.827578 read 0x01587c thru 0x0247d1 (0xef56 bytes) 000987 1746625954.831671 mapwrite 0x038f1f thru 0x03fe61 (0x6f43 bytes) 000988 1746625954.853861 punch from 0x039fe3 to 0x03fe61 (0x5e7f bytes) 000989 1746625954.856418 fallocate 0x0280bb thru 0x02850b (0x451 bytes) 000990 1746625954.858879 punch from 0x005e97 to 0x00ad71 (0x4edb bytes) 000992 1746625954.861328 punch from 0x01bb9d to 0x0285a3 (0xca07 bytes) 000993 1746625954.863678 read 0x00d24b thru 0x01540b (0x81c1 bytes) 000994 1746625954.864137 trunc from 0x03fe62 to 0x03c6a0 (0x37c1 bytes) 000995 1746625954.872569 fallocate 0x039d95 thru 0x03f533 (0x579f bytes) 000996 1746625954.875061 trunc from 0x03f534 to 0x005b2e (0x39a05 bytes) 000997 1746625954.881844 mapread 0x004434 thru 0x005b2e (0x16fb bytes) 000998 1746625954.885876 trunc from 0x005b2f to 0x0332dd (0x2d7af bytes) 000999 1746625954.893404 punch from 0x01100b to 0x019f77 (0x8f6d bytes) 001000 1746625954.895185 fallocate 0x03f5bf thru 0x03ffff (0xa41 bytes) All operations completed A-OK! PASS 70a (19s) == sanity-flr test 71: check mirror extend parent fid ==== 09:52:36 (1746625956) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x56:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 09:52:39 (1746625959) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 928992 .. Extending file size to 5902144 .. Extending file size to 3560640 .. Extending file size to 6757888 .. done Extending file size to 7027712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7353216 .. done Extending file size to 7211360 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8781216 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8221472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8284128 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8540416 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8740480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8945536 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8897344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9084352 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9171360 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 1175 1176 1177 1180 1182 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) 192.168.203.124@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:53:43 (1746626023) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8763744 .. Extending file size to 3561568 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9219264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8795072 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9057760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9116256 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 5248 5249 5251 5253 5255 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) 192.168.203.124@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:54:46 (1746626086) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2724 1284964 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2968 1284720 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22664 3584356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14788 3592232 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37452 7176588 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2724 1284964 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2968 1284720 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22664 3584356 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14788 3592232 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37452 7176588 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0411588 s, 51.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0636588 s, 32.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:54:52 (1746626092) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0288922 s, 72.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 09:54:53 (1746626093) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:54:55 (1746626095) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263384 s, 79.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x9b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:54:57 (1746626097) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0269973 s, 77.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:54:59 (1746626099) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0336501 s, 62.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:22 (1746626122) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg324-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169176 s, 62.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x9b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x9b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x9b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x9b5:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:54 (1746626154) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg324-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0265779 s, 78.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:56:29 (1746626189) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg324-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg324-server: Pool lustre.test_204f created Adding targets to pool oleg324-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg324-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0255102 s, 82.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg324-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg324-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg324-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:18 (1746626238) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:20 (1746626240) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:22 (1746626242) oleg324-server: Pool lustre.test_206 created Adding targets to pool oleg324-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg324-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg324-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg324-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg324-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:57:37 (1746626257) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203265 s, 51.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993376 s, 106 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x21af22ea Mirror 3: 0x21af22ea PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:57:40 (1746626260) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:57:41 (1746626261) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:57:42 (1746626262) Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0152085 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xbe0000 Read this (4 bytes): BBBB 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:57:56 (1746626276) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00418374 s, 2.0 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:57:58 (1746626278) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143461 s, 73.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9c:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-07 09:57:58.000000000 -0400 Modify: 2025-05-07 09:57:59.000000000 -0400 Change: 2025-05-07 09:57:59.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:58:00 (1746626280) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00940981 s, 111 MB/s oleg324-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg324-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg324-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg324-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg324-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:58:11 (1746626291) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.041594 s, 985 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0102053 s, 401 kB/s size after second write -rw-r--r-- 1 root root 4096 May 7 09:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 7 09:58 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 734 sec ============ 09:58:13 (1746626293) === sanity-flr: start cleanup 09:58:13 (1746626293) === === sanity-flr: finish cleanup 09:58:16 (1746626296) ===