-----============= acceptance-small: sanity-flr ============----- Sat Sep 14 04:50:19 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 04:50:25 (1726303825) === oleg109-client.virtnet: executing check_config_client /mnt/lustre oleg109-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg109-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5913800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5913800.idle_timeout=debug disable quota as required oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:50:37 (1726303837) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7586] == sanity-flr test 0a: lfs mirror create with -N option == 04:50:37 (1726303837) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:50:43 (1726303843) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:50:44 (1726303844) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:50:45 (1726303845) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:50:53 (1726303853) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:50:54 (1726303854) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 04:50:55 (1726303855) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5913800.stats=clear osc.lustre-OST0001-osc-ffff8800b5913800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:50:58 (1726303858) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:51:01 (1726303861) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:51:03 (1726303863) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:51:04 (1726303864) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 04:51:07 (1726303867) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:51:09 (1726303869) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:51:11 (1726303871) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:51:13 (1726303873) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:51:17 (1726303877) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:51:19 (1726303879) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.478711 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0463506 s, 22.6 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:51:22 (1726303882) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0729386 s, 86.3 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:51:25 (1726303885) Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:51:53 (1726303913) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.161432 s, 52.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:52:08 (1726303928) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:52:32 (1726303952) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.813532 s, 64.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 14 04:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.235s user 0m0.000s sys 0m0.101s Stopping /mnt/lustre-ost1 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.250s user 0m0.000s sys 0m0.111s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg109-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg109-server oleg109-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.270s user 0m0.000s sys 0m0.111s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg109-server: oleg109-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg109-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5913800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:52:59 (1726303979) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:53:00 (1726303980) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:53:01 (1726303981) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 04:53:02 (1726303982) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 04:53:04 (1726303984) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.188935 s, 66.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:53:08 (1726303988) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125763 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0236368 s, 44.4 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:53:10 (1726303990) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:53:13 (1726303993) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00184015 s, 556 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000915723 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000716728 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000771051 s, 1.3 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 04:53:15 (1726303995) create test files with size 15757280 .. 488989368c23ac023a25ea0e1e8a5ee3 - 26c1bbedc5c3f8367ec1874550a80fbc - 8ab1b134dd985d6adcf1b58c0213c279 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 04:53:22 (1726304002) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:53:26 (1726304006) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:53:28 (1726304008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390215 s, 26.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140322 s, 74.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 04:53:31 (1726304011) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0378296 s, 55.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.107727 s, 38.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0612585 s, 34.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 04:53:35 (1726304015) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 04:53:36 (1726304016) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:53:37 (1726304017) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115999 s, 90.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 04:53:40 (1726304020) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.115276 s, 27.3 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:53:43 (1726304023) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x43:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:53:47 (1726304027) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.123568 s, 84.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:53:50 (1726304030) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a7cc5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.109993 s, 95.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg109-client.virtnet /mnt/lustre (opts:) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b3929000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:53:55 (1726304035) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:54:06 (1726304046) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:54:16 (1726304056) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 04:54:17 (1726304057) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.102758 s, 30.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0712468 s, 44.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 33f64c9cb72000592cf4e982ea596550 - mirror 1 checksum: e23a720e79d98387f6527aa9bb4fa27c - mirror 2 checksum: 33f64c9cb72000592cf4e982ea596550 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Sep 14 04:54:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Sep 14 04:54:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 33f64c9cb72000592cf4e982ea596550 - mirror 1 checksum: e23a720e79d98387f6527aa9bb4fa27c - mirror 2 checksum: e23a720e79d98387f6527aa9bb4fa27c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Sep 14 04:54:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Sep 14 04:54:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:54:20 (1726304060) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011993 s, 87.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:54:23 (1726304063) Failing mds1 on oleg109-server Stopping /mnt/lustre-mds1 (opts:) on oleg109-server 04:54:30 (1726304070) shut down Failover mds1 to oleg109-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 04:54:45 (1726304085) targets are mounted 04:54:45 (1726304085) facet_failover done oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00976681 s, 107 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:54:52 (1726304092) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0124743 s, 1.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00634199 s, 3.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:54:55 (1726304095) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0161776 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00371747 s, 5.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:54:57 (1726304097) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00346849 s, 4.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:55:00 (1726304100) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0154606 s, 1.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:55:02 (1726304102) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00282721 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0111687 s, 367 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:55:04 (1726304104) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:55:10 (1726304110) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:55:30 (1726304130) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:55:47 (1726304147) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:56:04 (1726304164) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 04:56:06 (1726304166) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 25463 mirror split pid 25464 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 25463 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 25464 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 04:57:08 (1726304228) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4229 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000002 1726304229.349131 MIRROR_EXTEND to 2 mirrors 000003 1726304229.437188 mapwrite 0x01f0b5 thru 0x020c51 (0x1b9d bytes) 000005 1726304229.471042 write 0x02be24 thru 0x02f858 (0x3a35 bytes) fallocating to largest ever: 0x218e1 000007 1726304229.472023 fallocate 0x0123c1 thru 0x0218e0 (0xf520 bytes) 000009 1726304229.474284 trunc from 0x02f859 to 0x007614 (0x28244 bytes) 000010 1726304229.480880 mapread 0x003887 thru 0x007614 (0x3d8e bytes) 000011 1726304229.483720 mapwrite 0x01b3e2 thru 0x02a9d5 (0xf5f4 bytes) 000013 1726304229.507809 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000015 1726304229.552327 trunc from 0x02a9d6 to 0x00ead7 (0x1befe bytes) 000018 1726304229.558743 write 0x00a84c thru 0x017c26 (0xd3db bytes) 000019 1726304229.561895 punch from 0x00d96b to 0x011c72 (0x4308 bytes) 000020 1726304229.567134 mapwrite 0x020d78 thru 0x02fb97 (0xee20 bytes) fallocating to largest ever: 0x36d34 000021 1726304229.589791 fallocate 0x0331cb thru 0x036d33 (0x3b69 bytes) 000023 1726304229.594900 MIRROR_EXTEND to 2 mirrors 000024 1726304229.784162 write 0x009aea thru 0x00cf01 (0x3418 bytes) 000026 1726304229.801696 mapwrite 0x00ee30 thru 0x01945e (0xa62f bytes) 000028 1726304229.827802 fallocate 0x01ec60 thru 0x023d4b (0x50ec bytes) 000029 1726304229.831458 read 0x02a4b5 thru 0x02c77d (0x22c9 bytes) 000030 1726304229.834240 write 0x03504b thru 0x03ff1a (0xaed0 bytes) 000031 1726304229.840029 read 0x0016c2 thru 0x005fbe (0x48fd bytes) 000033 1726304229.847941 trunc from 0x03ff1b to 0x00315a (0x3cdc0 bytes) 000034 1726304229.866281 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000037 1726304229.914484 mapread 0x0018b1 thru 0x002000 (0x750 bytes) 000038 1726304229.919017 mapread 0x001ac1 thru 0x00315a (0x169a bytes) 000039 1726304229.921594 mapwrite 0x0154ad thru 0x01e27b (0x8dcf bytes) 000040 1726304229.947727 write 0x027251 thru 0x036704 (0xf4b4 bytes) 000044 1726304229.950463 mapread 0x003d7b thru 0x00b77d (0x7a03 bytes) 000046 1726304229.954276 read 0x0202b8 thru 0x023b6f (0x38b8 bytes) fallocating to largest ever: 0x3ee0d 000047 1726304229.956294 fallocate 0x031db0 thru 0x03ee0c (0xd05d bytes) 000048 1726304229.958066 fallocate 0x02c359 thru 0x02f8a3 (0x354b bytes) 000049 1726304229.959605 write 0x0269ae thru 0x029cd8 (0x332b bytes) 000050 1726304229.961910 mapwrite 0x02c1aa thru 0x03b11b (0xef72 bytes) 000051 1726304229.970714 mapread 0x03d1f0 thru 0x03ee0c (0x1c1d bytes) 000052 1726304229.971192 write 0x021ab0 thru 0x02dbe7 (0xc138 bytes) 000053 1726304229.971929 punch from 0x00f50f to 0x01249c (0x2f8e bytes) 000054 1726304229.973512 fallocate 0x02281a thru 0x03206a (0xf851 bytes) 000055 1726304229.975097 read 0x034500 thru 0x03ee0c (0xa90d bytes) 000057 1726304229.976060 punch from 0x034259 to 0x03d881 (0x9629 bytes) 000059 1726304229.977959 trunc from 0x03ee0d to 0x02fb55 (0xf2b7 bytes) 000060 1726304229.981612 write 0x01cbe0 thru 0x02a06f (0xd490 bytes) 000061 1726304229.982396 fallocate 0x02f396 thru 0x03e512 (0xf17d bytes) 000062 1726304229.984044 trunc from 0x02fb56 to 0x00b7a2 (0x243b3 bytes) 000063 1726304229.988027 punch from 0x00abe9 to 0x00b7a2 (0xbba bytes) 000064 1726304229.989336 mapwrite 0x02f218 thru 0x031345 (0x212e bytes) 000067 1726304230.002010 trunc from 0x031346 to 0x03e271 (0xcf2c bytes) 000068 1726304230.006133 fallocate 0x016583 thru 0x01cbcf (0x664d bytes) 000069 1726304230.007954 read 0x028a07 thru 0x02cc1d (0x4217 bytes) 000070 1726304230.008359 write 0x004902 thru 0x00c317 (0x7a16 bytes) 000071 1726304230.009200 trunc from 0x03e272 to 0x019b4f (0x24722 bytes) 000072 1726304230.013732 trunc from 0x019b50 to 0x012e98 (0x6cb7 bytes) 000074 1726304230.018799 read 0x009cc1 thru 0x012e98 (0x91d8 bytes) 000075 1726304230.019239 mapread 0x003c79 thru 0x005724 (0x1aac bytes) 000078 1726304230.020415 mapwrite 0x026eb4 thru 0x029178 (0x22c5 bytes) 000080 1726304230.036696 MIRROR_EXTEND to 2 mirrors 000081 1726304230.158367 mapread 0x00c7a5 thru 0x013b36 (0x7392 bytes) 000082 1726304230.165216 punch from 0x01011d to 0x0185e1 (0x84c5 bytes) 000083 1726304230.179291 MIRROR_RESYNC 2 mirrors 000084 1726304230.283134 mapwrite 0x018196 thru 0x0221dc (0xa047 bytes) 000085 1726304230.316037 mapread 0x015bfb thru 0x022179 (0xc57f bytes) 000086 1726304230.316505 trunc from 0x029179 to 0x021cfc (0x747c bytes) 000087 1726304230.324707 mapread 0x018bf3 thru 0x021cfc (0x910a bytes) 000088 1726304230.325225 read 0x015a91 thru 0x021a8e (0xbffe bytes) 000089 1726304230.325600 trunc from 0x021cfd to 0x011dde (0xff1e bytes) 000091 1726304230.329303 write 0x024a8e thru 0x02be9e (0x7411 bytes) 000092 1726304230.330017 trunc from 0x02be9f to 0x02b3bc (0xae2 bytes) 000095 1726304230.333680 trunc from 0x02b3bd to 0x01ec57 (0xc765 bytes) 000096 1726304230.336587 write 0x00c305 thru 0x015f78 (0x9c74 bytes) 000097 1726304230.339868 mapread 0x007f1a thru 0x011051 (0x9138 bytes) 000098 1726304230.340412 trunc from 0x01ec58 to 0x009f41 (0x14d16 bytes) 000099 1726304230.344268 trunc from 0x009f42 to 0x00a28e (0x34d bytes) 000100 1726304230.347561 trunc from 0x00a28f to 0x016b29 (0xc89b bytes) 000102 1726304230.351171 read 0x009836 thru 0x00f5ce (0x5d99 bytes) 000104 1726304230.354467 trunc from 0x016b2a to 0x03ab68 (0x2403f bytes) 000105 1726304230.359117 punch from 0x02c38d to 0x039e4c (0xdac0 bytes) 000106 1726304230.360890 mapread 0x009d32 thru 0x012aa5 (0x8d74 bytes) 000107 1726304230.361521 read 0x01744a thru 0x018fa8 (0x1b5f bytes) 000108 1726304230.366482 write 0x004f47 thru 0x0064fb (0x15b5 bytes) 000109 1726304230.367166 read 0x0067d8 thru 0x00c0cf (0x58f8 bytes) 000110 1726304230.369275 MIRROR_RESYNC 2 mirrors 000113 1726304230.479070 fallocate 0x02fd55 thru 0x03d33f (0xd5eb bytes) 000114 1726304230.492371 fallocate 0x0178a7 thru 0x01b9c6 (0x4120 bytes) 000115 1726304230.495898 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000116 1726304230.536866 mapwrite 0x028d43 thru 0x02a3f7 (0x16b5 bytes) 000118 1726304230.548843 mapread 0x01187b thru 0x01cee9 (0xb66f bytes) 000119 1726304230.553166 write 0x03e423 thru 0x03f673 (0x1251 bytes) 000121 1726304230.555559 mapwrite 0x038ba1 thru 0x03c823 (0x3c83 bytes) 000122 1726304230.562787 trunc from 0x03f674 to 0x03219c (0xd4d7 bytes) 000123 1726304230.570569 mapwrite 0x01f050 thru 0x02ad38 (0xbce9 bytes) 000124 1726304230.577263 trunc from 0x03219d to 0x027968 (0xa834 bytes) 000125 1726304230.581127 fallocate 0x014ff9 thru 0x019698 (0x46a0 bytes) 000126 1726304230.582590 mapread 0x01dd5d thru 0x01fa84 (0x1d28 bytes) 000128 1726304230.583416 mapwrite 0x008b7b thru 0x00ffdb (0x7461 bytes) 000129 1726304230.592945 read 0x01ec59 thru 0x027968 (0x8d10 bytes) 000130 1726304230.593372 write 0x03d29a thru 0x03ffff (0x2d66 bytes) 000131 1726304230.593986 mapwrite 0x01ec2a thru 0x025900 (0x6cd7 bytes) 000132 1726304230.600425 mapwrite 0x013f88 thru 0x01baf7 (0x7b70 bytes) 000133 1726304230.606735 mapread 0x028b0f thru 0x030f55 (0x8447 bytes) 000134 1726304230.611994 write 0x002328 thru 0x00ee3d (0xcb16 bytes) 000135 1726304230.612989 punch from 0x03e38d to 0x03ffff (0x1c73 bytes) 000139 1726304230.616387 trunc from 0x040000 to 0x02fe6c (0x10193 bytes) 000140 1726304230.622064 write 0x039818 thru 0x03ffff (0x67e8 bytes) 000141 1726304230.622878 trunc from 0x040000 to 0x012003 (0x2dffc bytes) 000142 1726304230.628499 write 0x01553b thru 0x0224f9 (0xcfbf bytes) 000143 1726304230.629544 trunc from 0x0224fa to 0x01022e (0x122cb bytes) 000145 1726304230.634953 trunc from 0x01022f to 0x019a89 (0x985b bytes) 000147 1726304230.640772 mapwrite 0x036dd0 thru 0x03ffff (0x9230 bytes) 000148 1726304230.659362 fallocate 0x02a48a thru 0x032699 (0x8210 bytes) 000149 1726304230.661179 write 0x00b4ac thru 0x00f81e (0x4373 bytes) fallocating to largest ever: 0x40000 000150 1726304230.661852 fallocate 0x03537d thru 0x03ffff (0xac83 bytes) 000151 1726304230.664520 fallocate 0x0059c1 thru 0x00f8ff (0x9f3f bytes) 000152 1726304230.666953 trunc from 0x040000 to 0x008f3b (0x370c4 bytes) 000153 1726304230.678394 fallocate 0x0328e1 thru 0x038030 (0x5750 bytes) 000154 1726304230.680650 punch from 0x0045e5 to 0x011976 (0xd392 bytes) 000156 1726304230.684215 punch from 0x00f087 to 0x01de74 (0xedee bytes) 000157 1726304230.686321 read 0x00c970 thru 0x01aaa2 (0xe133 bytes) 000158 1726304230.690730 write 0x03b535 thru 0x03ffff (0x4acb bytes) 000159 1726304230.691508 punch from 0x010151 to 0x01edc5 (0xec75 bytes) 000160 1726304230.695343 MIRROR_EXTEND to 2 mirrors 000161 1726304230.818039 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000162 1726304230.848356 punch from 0x03dfe8 to 0x03ffff (0x2018 bytes) 000163 1726304230.851928 trunc from 0x040000 to 0x03c64c (0x39b3 bytes) 000166 1726304230.861948 punch from 0x03718f to 0x03c64c (0x54be bytes) 000168 1726304230.864390 read 0x000f18 thru 0x00c54e (0xb637 bytes) 000169 1726304230.867360 mapread 0x026b58 thru 0x033e5c (0xd305 bytes) 000171 1726304230.875869 trunc from 0x03c64d to 0x039908 (0x2d44 bytes) 000173 1726304230.880582 write 0x025e75 thru 0x0294f8 (0x3684 bytes) 000174 1726304230.881333 trunc from 0x039909 to 0x02c3f8 (0xd510 bytes) 000175 1726304230.894302 MIRROR_EXTEND to 2 mirrors 000176 1726304231.078809 write 0x03813c thru 0x03ffd9 (0x7e9e bytes) 000177 1726304231.092540 fallocate 0x022bab thru 0x026706 (0x3b5c bytes) 000178 1726304231.094802 trunc from 0x03ffda to 0x0021a6 (0x3de33 bytes) 000179 1726304231.103212 punch from 0x0003f7 to 0x00179b (0x13a5 bytes) 000180 1726304231.105146 fallocate 0x025032 thru 0x02f696 (0xa665 bytes) 000181 1726304231.107243 mapread 0x014326 thru 0x019c71 (0x594c bytes) 000182 1726304231.114038 MIRROR_EXTEND to 3 mirrors 000183 1726304231.263166 mapwrite 0x00849a thru 0x00fa7c (0x75e3 bytes) 000185 1726304231.312245 mapread 0x01e154 thru 0x0238fc (0x57a9 bytes) 000187 1726304231.316631 trunc from 0x02f697 to 0x01e117 (0x1157f bytes) 000188 1726304231.335509 read 0x00ebb4 thru 0x010728 (0x1b75 bytes) 000189 1726304231.335942 read 0x013926 thru 0x01e117 (0xa7f2 bytes) 000190 1726304231.340929 mapread 0x0011cb thru 0x0093e5 (0x821b bytes) 000191 1726304231.341647 fallocate 0x010238 thru 0x01ad44 (0xab0d bytes) 000192 1726304231.344122 punch from 0x00c0b2 to 0x0196c6 (0xd615 bytes) 000193 1726304231.347429 fallocate 0x024b06 thru 0x02a372 (0x586d bytes) 000194 1726304231.351092 mapread 0x00f29f thru 0x015fd0 (0x6d32 bytes) 000195 1726304231.367231 punch from 0x0031a3 to 0x0049f2 (0x1850 bytes) 000197 1726304231.369170 read 0x005e38 thru 0x005f6e (0x137 bytes) 000198 1726304231.369445 fallocate 0x00a5e1 thru 0x0119b0 (0x73d0 bytes) 000199 1726304231.371455 trunc from 0x02a373 to 0x03e24d (0x13edb bytes) 000200 1726304231.376381 trunc from 0x03e24e to 0x01b6f4 (0x22b59 bytes) 000201 1726304231.380620 mapwrite 0x0075b7 thru 0x013f12 (0xc95c bytes) 000202 1726304231.391529 trunc from 0x01b6f5 to 0x0129ca (0x8d2a bytes) 000203 1726304231.397834 fallocate 0x007923 thru 0x01204f (0xa72d bytes) 000204 1726304231.400833 write 0x016866 thru 0x01952e (0x2cc9 bytes) 000206 1726304231.401888 write 0x02309e thru 0x02dbc9 (0xab2c bytes) 000207 1726304231.402840 trunc from 0x02dbca to 0x01612c (0x17a9d bytes) 000208 1726304231.408833 write 0x02c102 thru 0x02fdbc (0x3cbb bytes) 000209 1726304231.409617 mapwrite 0x030736 thru 0x039718 (0x8fe3 bytes) 000210 1726304231.432829 fallocate 0x000ef1 thru 0x009c61 (0x8d71 bytes) 000211 1726304231.435260 trunc from 0x039719 to 0x0300af (0x9669 bytes) 000212 1726304231.441098 mapread 0x01dabd thru 0x01e27d (0x7c1 bytes) 000213 1726304231.441685 mapread 0x008d25 thru 0x0166f2 (0xd9ce bytes) 000214 1726304231.448620 mapread 0x0213e0 thru 0x02e699 (0xd2ba bytes) 000215 1726304231.449253 write 0x0100af thru 0x01cea2 (0xcdf4 bytes) 000216 1726304231.450221 punch from 0x02e7a5 to 0x0300af (0x190b bytes) 000217 1726304231.453004 punch from 0x01ebb0 to 0x02b6c9 (0xcb1a bytes) 000218 1726304231.455564 write 0x026c69 thru 0x0356aa (0xea42 bytes) 000219 1726304231.459013 fallocate 0x0228a1 thru 0x02985a (0x6fba bytes) 000220 1726304231.462672 MIRROR_RESYNC 3 mirrors 000221 1726304231.648171 mapread 0x004567 thru 0x010564 (0xbffe bytes) 000223 1726304231.656038 mapread 0x0246d8 thru 0x02a06f (0x5998 bytes) 000224 1726304231.659242 punch from 0x02f00b to 0x0356aa (0x66a0 bytes) 000225 1726304231.674066 mapwrite 0x024a57 thru 0x0281bf (0x3769 bytes) 000226 1726304231.685721 mapwrite 0x016961 thru 0x01c91a (0x5fba bytes) 000227 1726304231.695337 punch from 0x01719c to 0x01a94d (0x37b2 bytes) 000228 1726304231.697509 mapwrite 0x00ee68 thru 0x010ce8 (0x1e81 bytes) 000229 1726304231.704534 read 0x007f73 thru 0x0131a5 (0xb233 bytes) 000230 1726304231.706584 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000231 1726304231.772485 trunc from 0x0356ab to 0x01509c (0x2060e bytes) 000232 1726304231.782716 MIRROR_EXTEND to 3 mirrors 000235 1726304231.912890 read 0x00c153 thru 0x00f631 (0x34df bytes) 000237 1726304231.920642 punch from 0x010ad2 to 0x01509c (0x45cb bytes) 000238 1726304231.933270 write 0x031608 thru 0x036f62 (0x595b bytes) 000241 1726304231.936287 write 0x019958 thru 0x025546 (0xbbef bytes) 000242 1726304231.942425 MIRROR_EXTEND to 4 mirrors 000245 1726304232.114841 read 0x033d2b thru 0x036f62 (0x3238 bytes) 000246 1726304232.119320 read 0x0098e8 thru 0x009959 (0x72 bytes) 000247 1726304232.126532 mapwrite 0x02279e thru 0x02e25e (0xbac1 bytes) 000248 1726304232.174768 read 0x00fe49 thru 0x01362e (0x37e6 bytes) 000249 1726304232.178147 fallocate 0x03e772 thru 0x03ffff (0x188e bytes) 000250 1726304232.180136 read 0x0268f3 thru 0x02f32a (0x8a38 bytes) 000251 1726304232.180697 write 0x0183f7 thru 0x023fe9 (0xbbf3 bytes) 000252 1726304232.183119 punch from 0x022d58 to 0x031a7a (0xed23 bytes) 000254 1726304232.188475 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000255 1726304232.243921 write 0x034cce thru 0x037307 (0x263a bytes) 000256 1726304232.250802 MIRROR_EXTEND to 4 mirrors 000257 1726304232.444767 MIRROR_EXTEND to 5 mirrors 000258 1726304232.654762 punch from 0x01fa65 to 0x0299d8 (0x9f74 bytes) 000259 1726304232.675034 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000260 1726304232.748609 punch from 0x0007ef to 0x00ed19 (0xe52b bytes) 000261 1726304232.751016 trunc from 0x040000 to 0x02a6fe (0x15901 bytes) 000262 1726304232.764562 mapwrite 0x01ac9b thru 0x0222d7 (0x763d bytes) 000263 1726304232.786254 punch from 0x01218c to 0x019bb8 (0x7a2d bytes) 000264 1726304232.789044 punch from 0x015d35 to 0x018bcb (0x2e97 bytes) 000265 1726304232.790609 mapread 0x0253d3 thru 0x029614 (0x4242 bytes) 000267 1726304232.797255 MIRROR_EXTEND to 5 mirrors 000268 1726304232.969926 mapread 0x01f985 thru 0x027f7e (0x85fa bytes) 000269 1726304232.978820 read 0x026f3e thru 0x02a6fe (0x37c1 bytes) 000272 1726304232.989529 MIRROR_EXTEND to 6 mirrors 000273 1726304233.146186 trunc from 0x02a6ff to 0x001c45 (0x28ab9 bytes) 000275 1726304233.165800 mapread 0x000c1d thru 0x001c45 (0x1029 bytes) 000276 1726304233.170712 read 0x0019af thru 0x001c45 (0x297 bytes) 000277 1726304233.171037 fallocate 0x028e06 thru 0x02d5f5 (0x47f0 bytes) 000278 1726304233.173581 fallocate 0x03446c thru 0x03bcdb (0x7870 bytes) 000279 1726304233.175630 mapwrite 0x006799 thru 0x00c0a7 (0x590f bytes) 000280 1726304233.193092 write 0x00a046 thru 0x00f7d1 (0x578c bytes) 000281 1726304233.193912 write 0x0229f4 thru 0x028540 (0x5b4d bytes) 000282 1726304233.194465 fallocate 0x004a4e thru 0x00f244 (0xa7f7 bytes) 000283 1726304233.197174 read 0x010f82 thru 0x01ef13 (0xdf92 bytes) 000285 1726304233.201454 punch from 0x000812 to 0x002a43 (0x2232 bytes) 000286 1726304233.205470 MIRROR_RESYNC 6 mirrors 000287 1726304233.550996 punch from 0x02750f to 0x027a53 (0x545 bytes) 000288 1726304233.566525 mapwrite 0x00538e thru 0x00ae85 (0x5af8 bytes) 000290 1726304233.589648 MIRROR_RESYNC 6 mirrors 000291 1726304233.939643 trunc from 0x028541 to 0x002e1e (0x25722 bytes) 000292 1726304233.964997 mapwrite 0x002db1 thru 0x003b8b (0xddb bytes) 000293 1726304233.993584 mapread 0x003148 thru 0x003b8b (0xa44 bytes) 000294 1726304233.994157 mapwrite 0x03ada1 thru 0x03ffff (0x525f bytes) 000296 1726304234.018413 mapwrite 0x0281ff thru 0x0380c7 (0xfec9 bytes) 000297 1726304234.035580 MIRROR_EXTEND to 7 mirrors 000298 1726304234.234080 MIRROR_RESYNC 7 mirrors 000300 1726304234.625568 trunc from 0x040000 to 0x037c1b (0x83e4 bytes) 000301 1726304234.653047 read 0x014052 thru 0x0152b1 (0x1260 bytes) 000302 1726304234.656255 punch from 0x011cb5 to 0x01c4db (0xa827 bytes) 000303 1726304234.661545 MIRROR_RESYNC 7 mirrors 000304 1726304235.058674 mapread 0x02b40a thru 0x02ec1f (0x3816 bytes) 000305 1726304235.065216 punch from 0x009212 to 0x018fce (0xfdbd bytes) 000306 1726304235.080768 punch from 0x02a195 to 0x037c1b (0xda87 bytes) 000307 1726304235.083481 mapwrite 0x03fcbb thru 0x03ffff (0x345 bytes) 000308 1726304235.106376 read 0x03e5b6 thru 0x03ffff (0x1a4a bytes) 000309 1726304235.110938 MIRROR_RESYNC 7 mirrors 000310 1726304235.642738 MIRROR_EXTEND to 8 mirrors 000311 1726304235.878336 read 0x00161c thru 0x00665a (0x503f bytes) 000312 1726304235.884316 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000313 1726304235.994575 mapwrite 0x0211af thru 0x027d1d (0x6b6f bytes) 000315 1726304236.056383 punch from 0x022a4f to 0x02d454 (0xaa06 bytes) 000316 1726304236.059171 mapread 0x01944e thru 0x01e619 (0x51cc bytes) 000317 1726304236.063749 write 0x00b6a5 thru 0x014444 (0x8da0 bytes) 000318 1726304236.068632 fallocate 0x0300b7 thru 0x033368 (0x32b2 bytes) 000320 1726304236.071276 mapwrite 0x00b9b5 thru 0x019f1e (0xe56a bytes) 000321 1726304236.079924 read 0x02ec15 thru 0x0359d5 (0x6dc1 bytes) 000322 1726304236.086211 MIRROR_EXTEND to 8 mirrors 000323 1726304236.269417 mapwrite 0x02e6c5 thru 0x035a11 (0x734d bytes) 000325 1726304236.313724 mapread 0x018455 thru 0x01b19f (0x2d4b bytes) 000326 1726304236.319149 trunc from 0x040000 to 0x0296c8 (0x16937 bytes) 000328 1726304236.337641 mapwrite 0x0365ca thru 0x03ffff (0x9a36 bytes) 000329 1726304236.358595 mapwrite 0x00914f thru 0x013ff7 (0xaea9 bytes) 000330 1726304236.371958 trunc from 0x040000 to 0x01d62b (0x229d4 bytes) 000332 1726304236.379878 mapwrite 0x038ff2 thru 0x03d404 (0x4413 bytes) 000333 1726304236.404429 read 0x00f427 thru 0x016688 (0x7262 bytes) 000335 1726304236.405124 punch from 0x0317bc to 0x037626 (0x5e6b bytes) 000338 1726304236.410820 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000339 1726304236.480822 trunc from 0x03d405 to 0x00dff7 (0x2f40d bytes) 000340 1726304236.488576 write 0x032fe2 thru 0x03bb0e (0x8b2d bytes) 000341 1726304236.489639 read 0x022bb2 thru 0x02e201 (0xb650 bytes) 000342 1726304236.493611 fallocate 0x019156 thru 0x01c62c (0x34d7 bytes) 000344 1726304236.496008 write 0x00a416 thru 0x01706f (0xcc5a bytes) 000345 1726304236.500802 mapread 0x01e064 thru 0x02978c (0xb729 bytes) 000346 1726304236.504493 mapwrite 0x0226c7 thru 0x02f5d8 (0xcf12 bytes) 000347 1726304236.518586 fallocate 0x01e7ec thru 0x021dd0 (0x35e5 bytes) 000348 1726304236.524857 mapread 0x025243 thru 0x02c36b (0x7129 bytes) 000349 1726304236.525597 read 0x02f568 thru 0x0398f3 (0xa38c bytes) 000350 1726304236.526929 mapwrite 0x02e4d0 thru 0x0350c1 (0x6bf2 bytes) 000351 1726304236.539501 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000353 1726304236.611063 read 0x01b8e5 thru 0x01f990 (0x40ac bytes) 000354 1726304236.614527 punch from 0x02a3f0 to 0x02c494 (0x20a5 bytes) 000355 1726304236.616196 mapread 0x03abae thru 0x03bb0e (0xf61 bytes) 000356 1726304236.619623 write 0x0231ad thru 0x02ce06 (0x9c5a bytes) 000357 1726304236.626014 punch from 0x00476e to 0x01442b (0xfcbe bytes) 000358 1726304236.629502 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000359 1726304236.700156 trunc from 0x03bb0f to 0x01df69 (0x1dba5 bytes) 000360 1726304236.708552 trunc from 0x01df6a to 0x006fad (0x16fbc bytes) 000361 1726304236.714860 mapwrite 0x01d906 thru 0x01ed7d (0x1478 bytes) 000362 1726304236.735866 fallocate 0x01ede0 thru 0x02d2d4 (0xe4f5 bytes) 000363 1726304236.738211 read 0x00afde thru 0x00f533 (0x4556 bytes) 000365 1726304236.742186 fallocate 0x018d7a thru 0x01e888 (0x5b0f bytes) 000366 1726304236.747838 MIRROR_EXTEND to 6 mirrors 000367 1726304236.897391 read 0x02ce05 thru 0x02d2d4 (0x4d0 bytes) 000369 1726304236.903841 mapwrite 0x03db62 thru 0x03ffff (0x249e bytes) 000370 1726304236.932717 MIRROR_EXTEND to 7 mirrors 000371 1726304237.089171 trunc from 0x040000 to 0x036108 (0x9ef7 bytes) 000372 1726304237.106666 MIRROR_RESYNC 7 mirrors 000373 1726304237.434462 write 0x006002 thru 0x00ee69 (0x8e68 bytes) 000374 1726304237.446757 mapwrite 0x01f5c2 thru 0x020be5 (0x1624 bytes) 000375 1726304237.456316 mapread 0x0273bc thru 0x03559f (0xe1e4 bytes) 000376 1726304237.461162 punch from 0x02d273 to 0x036108 (0x8e96 bytes) 000377 1726304237.462797 trunc from 0x036109 to 0x012813 (0x238f5 bytes) 000379 1726304237.467771 write 0x008405 thru 0x008de6 (0x9e2 bytes) 000380 1726304237.469632 MIRROR_RESYNC 7 mirrors 000381 1726304237.765383 trunc from 0x012814 to 0x01fcb2 (0xd49f bytes) 000382 1726304237.775945 punch from 0x01d3a0 to 0x01fcb2 (0x2913 bytes) 000383 1726304237.777632 punch from 0x01f3bd to 0x01fcb2 (0x8f6 bytes) 000386 1726304237.779931 punch from 0x002f0b to 0x005645 (0x273b bytes) 000388 1726304237.783167 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000390 1726304237.833030 punch from 0x01235e to 0x01fcb2 (0xd955 bytes) 000393 1726304237.836639 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000394 1726304237.885070 write 0x03b6bb thru 0x03ffff (0x4945 bytes) 000395 1726304237.887098 mapread 0x02c01d thru 0x033bec (0x7bd0 bytes) 000396 1726304237.891690 write 0x0192e9 thru 0x023393 (0xa0ab bytes) 000397 1726304237.896031 fallocate 0x034a3a thru 0x03516e (0x735 bytes) 000399 1726304237.898041 fallocate 0x01ef08 thru 0x02e99d (0xfa96 bytes) 000400 1726304237.899671 punch from 0x0262bc to 0x02a99e (0x46e3 bytes) 000401 1726304237.900817 mapwrite 0x022f70 thru 0x02ef47 (0xbfd8 bytes) 000403 1726304237.912153 MIRROR_RESYNC 5 mirrors 000404 1726304238.209422 read 0x005e4e thru 0x01431c (0xe4cf bytes) 000405 1726304238.213058 trunc from 0x040000 to 0x01ae8b (0x25174 bytes) 000406 1726304238.228681 mapread 0x01810e thru 0x01a692 (0x2585 bytes) 000409 1726304238.236434 read 0x019c41 thru 0x01ae8b (0x124b bytes) 000410 1726304238.236840 mapwrite 0x01753a thru 0x023fe5 (0xcaac bytes) 000411 1726304238.257607 read 0x01f8ed thru 0x023fe5 (0x46f9 bytes) 000412 1726304238.258071 mapread 0x0021d0 thru 0x010324 (0xe155 bytes) 000414 1726304238.258906 trunc from 0x023fe6 to 0x0320c7 (0xe0e2 bytes) 000415 1726304238.263378 fallocate 0x03aa64 thru 0x03ffff (0x559c bytes) 000416 1726304238.266318 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000417 1726304238.313385 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000419 1726304238.366942 punch from 0x015d9b to 0x021eac (0xc112 bytes) 000420 1726304238.369057 write 0x00bab4 thru 0x0165e1 (0xab2e bytes) 000421 1726304238.376550 MIRROR_EXTEND to 4 mirrors 000422 1726304238.547586 mapwrite 0x01348f thru 0x01c692 (0x9204 bytes) 000423 1726304238.588042 MIRROR_EXTEND to 5 mirrors 000424 1726304238.775336 punch from 0x007274 to 0x00dcc7 (0x6a54 bytes) 000426 1726304238.787125 punch from 0x00c7fa to 0x00fdee (0x35f5 bytes) 000427 1726304238.788930 read 0x01cc3f thru 0x0253f3 (0x87b5 bytes) 000428 1726304238.791552 punch from 0x03120e to 0x0320c7 (0xeba bytes) 000429 1726304238.793031 mapwrite 0x00baa9 thru 0x013877 (0x7dcf bytes) 000430 1726304238.804229 read 0x012f74 thru 0x013f84 (0x1011 bytes) 000431 1726304238.804651 mapread 0x011e0e thru 0x01bf39 (0xa12c bytes) 000432 1726304238.807842 mapwrite 0x01cec9 thru 0x0202e6 (0x341e bytes) 000433 1726304238.813599 trunc from 0x0320c8 to 0x024631 (0xda96 bytes) 000434 1726304238.820489 read 0x00061a thru 0x00e0d0 (0xdab7 bytes) 000435 1726304238.820805 trunc from 0x024632 to 0x010cfb (0x13936 bytes) 000436 1726304238.825618 write 0x0243c5 thru 0x0286ac (0x42e8 bytes) 000437 1726304238.826493 mapread 0x01d8bb thru 0x021e03 (0x4549 bytes) 000438 1726304238.832424 MIRROR_EXTEND to 6 mirrors 000439 1726304239.003210 punch from 0x00520f to 0x005ed9 (0xccb bytes) 000440 1726304239.021715 mapread 0x01f665 thru 0x0286ac (0x9048 bytes) 000441 1726304239.030913 write 0x0293bb thru 0x032b32 (0x9778 bytes) 000443 1726304239.033984 mapwrite 0x03c9e1 thru 0x03ffff (0x361f bytes) 000444 1726304239.058934 mapwrite 0x003361 thru 0x00b6f2 (0x8392 bytes) 000445 1726304239.072305 read 0x02ac22 thru 0x02f8af (0x4c8e bytes) 000446 1726304239.072761 mapread 0x031bee thru 0x038e12 (0x7225 bytes) 000448 1726304239.078202 MIRROR_EXTEND to 7 mirrors 000449 1726304239.232290 trunc from 0x040000 to 0x00cd98 (0x33267 bytes) 000450 1726304239.252124 write 0x03cefd thru 0x03ea4e (0x1b52 bytes) 000451 1726304239.252998 mapwrite 0x01ab6c thru 0x01be0d (0x12a2 bytes) 000452 1726304239.269687 mapread 0x0181db thru 0x01844e (0x274 bytes) 000453 1726304239.273044 read 0x00b80d thru 0x010951 (0x5145 bytes) 000454 1726304239.275820 write 0x0163f4 thru 0x01ac30 (0x483d bytes) 000456 1726304239.276787 trunc from 0x03ea4f to 0x030f9e (0xdab0 bytes) 000457 1726304239.284756 read 0x02c230 thru 0x030f9e (0x4d6f bytes) 000458 1726304239.288614 trunc from 0x030f9f to 0x0258b4 (0xb6ea bytes) 000460 1726304239.295328 punch from 0x0249d0 to 0x0258b4 (0xee5 bytes) 000461 1726304239.297378 write 0x00c1f3 thru 0x01749b (0xb2a9 bytes) 000462 1726304239.298262 write 0x01de10 thru 0x02db70 (0xfd61 bytes) 000463 1726304239.299272 read 0x02d769 thru 0x02db70 (0x408 bytes) 000465 1726304239.300017 read 0x01b502 thru 0x021c45 (0x6744 bytes) 000467 1726304239.300837 trunc from 0x02db71 to 0x022d75 (0xadfb bytes) 000468 1726304239.306699 mapwrite 0x02da1c thru 0x03a89e (0xce83 bytes) 000469 1726304239.329466 mapwrite 0x03983c thru 0x03ffff (0x67c4 bytes) 000470 1726304239.346757 read 0x008f87 thru 0x0152ef (0xc369 bytes) 000471 1726304239.347259 trunc from 0x040000 to 0x0117bb (0x2e844 bytes) 000473 1726304239.352512 trunc from 0x0117bc to 0x003768 (0xe053 bytes) 000475 1726304239.358156 write 0x03c5d7 thru 0x03ffff (0x3a29 bytes) 000476 1726304239.358897 read 0x02d0b0 thru 0x0318ee (0x483f bytes) 000477 1726304239.362484 punch from 0x03436c to 0x035118 (0xdad bytes) 000478 1726304239.364576 punch from 0x03879a to 0x03ffff (0x7866 bytes) 000479 1726304239.368916 read 0x00bdb5 thru 0x00d771 (0x19bd bytes) 000480 1726304239.369354 fallocate 0x023d25 thru 0x02875e (0x4a3a bytes) 000481 1726304239.371424 punch from 0x01001d to 0x011543 (0x1527 bytes) 000482 1726304239.373628 mapread 0x00db01 thru 0x018933 (0xae33 bytes) 000484 1726304239.378386 mapread 0x000cfe thru 0x008d61 (0x8064 bytes) 000485 1726304239.378958 mapread 0x028258 thru 0x02ab98 (0x2941 bytes) 000486 1726304239.379440 mapwrite 0x036c95 thru 0x03ffff (0x936b bytes) 000487 1726304239.393232 MIRROR_EXTEND to 8 mirrors 000488 1726304239.577434 fallocate 0x00f1e9 thru 0x01cc2e (0xda46 bytes) 000489 1726304239.588467 fallocate 0x03eff3 thru 0x03ffff (0x100d bytes) 000490 1726304239.590185 trunc from 0x040000 to 0x02d25c (0x12da3 bytes) 000491 1726304239.605117 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000492 1726304239.671125 mapwrite 0x02af2e thru 0x02d693 (0x2766 bytes) 000493 1726304239.699147 mapwrite 0x022fa3 thru 0x02f440 (0xc49e bytes) 000494 1726304239.724352 trunc from 0x02f441 to 0x0256af (0x9d91 bytes) 000495 1726304239.728787 mapread 0x003104 thru 0x00cedf (0x9ddc bytes) 000496 1726304239.734617 fallocate 0x01b10d thru 0x029060 (0xdf54 bytes) 000497 1726304239.736529 mapread 0x016119 thru 0x01fd47 (0x9c2f bytes) 000498 1726304239.741906 MIRROR_EXTEND to 8 mirrors 000499 1726304239.920064 read 0x011870 thru 0x01f3ab (0xdb3c bytes) 000500 1726304239.926171 fallocate 0x016896 thru 0x019dbc (0x3527 bytes) 000501 1726304239.942300 trunc from 0x0256b0 to 0x0159b7 (0xfcf8 bytes) 000502 1726304239.957644 trunc from 0x0159b8 to 0x00acc3 (0xacf4 bytes) 000503 1726304239.963222 fallocate 0x034338 thru 0x034b53 (0x81c bytes) 000505 1726304239.965678 fallocate 0x0194dc thru 0x02521b (0xbd40 bytes) 000506 1726304239.967606 fallocate 0x02af24 thru 0x02c582 (0x165f bytes) 000507 1726304239.969331 mapwrite 0x00a6e6 thru 0x00e6b2 (0x3fcd bytes) 000508 1726304239.982620 mapread 0x004013 thru 0x00aec4 (0x6eb2 bytes) 000509 1726304239.985309 fallocate 0x03e404 thru 0x03ffff (0x1bfc bytes) 000510 1726304239.987493 write 0x033568 thru 0x03ffff (0xca98 bytes) 000511 1726304239.990651 MIRROR_EXTEND to 9 mirrors 000512 1726304240.152220 fallocate 0x025d81 thru 0x0335e0 (0xd860 bytes) 000513 1726304240.163678 mapwrite 0x013172 thru 0x01db2f (0xa9be bytes) 000514 1726304240.178830 trunc from 0x040000 to 0x038bfc (0x7403 bytes) 000518 1726304240.185801 write 0x028578 thru 0x037455 (0xeede bytes) 000519 1726304240.190196 mapread 0x034ea0 thru 0x038bfc (0x3d5d bytes) 000520 1726304240.191923 punch from 0x013e1d to 0x01763c (0x3820 bytes) 000521 1726304240.194067 read 0x026fd4 thru 0x035771 (0xe79e bytes) 000522 1726304240.194856 mapread 0x002ebb thru 0x0045ad (0x16f3 bytes) 000523 1726304240.198432 fallocate 0x026758 thru 0x03605b (0xf904 bytes) 000524 1726304240.200694 mapwrite 0x03329f thru 0x038474 (0x51d6 bytes) 000525 1726304240.208683 mapwrite 0x0030be thru 0x00718e (0x40d1 bytes) 000526 1726304240.215511 mapwrite 0x013981 thru 0x01f986 (0xc006 bytes) 000527 1726304240.226121 mapwrite 0x0175d3 thru 0x0227df (0xb20d bytes) 000528 1726304240.232833 punch from 0x000f0d to 0x005e16 (0x4f0a bytes) 000529 1726304240.234813 mapwrite 0x033cc8 thru 0x03ffff (0xc338 bytes) 000530 1726304240.249587 read 0x001449 thru 0x00e5ac (0xd164 bytes) 000531 1726304240.252198 punch from 0x03fedf to 0x03ffff (0x121 bytes) 000532 1726304240.254298 fallocate 0x02e402 thru 0x037f0d (0x9b0c bytes) 000533 1726304240.256419 fallocate 0x031b05 thru 0x03ffff (0xe4fb bytes) 000534 1726304240.258737 write 0x004e00 thru 0x00e1d1 (0x93d2 bytes) 000535 1726304240.259705 mapread 0x01a2c8 thru 0x01f911 (0x564a bytes) 000536 1726304240.260369 fallocate 0x02a6ea thru 0x039543 (0xee5a bytes) 000537 1726304240.263593 fallocate 0x01848a thru 0x01e371 (0x5ee8 bytes) 000539 1726304240.265871 trunc from 0x040000 to 0x01d851 (0x227ae bytes) 000540 1726304240.271146 trunc from 0x01d852 to 0x00f23e (0xe613 bytes) 000542 1726304240.276072 punch from 0x0008ba to 0x0012e5 (0xa2c bytes) 000543 1726304240.278185 trunc from 0x00f23f to 0x030afc (0x218be bytes) 000544 1726304240.283736 fallocate 0x029b18 thru 0x02f023 (0x550c bytes) 000546 1726304240.286158 write 0x026579 thru 0x0332be (0xcd46 bytes) 000547 1726304240.288819 punch from 0x010b7e to 0x01437f (0x3802 bytes) 000549 1726304240.290810 trunc from 0x0332bf to 0x007dc1 (0x2b4fd bytes) 000550 1726304240.295908 punch from 0x00191e to 0x007dc1 (0x64a4 bytes) 000551 1726304240.300362 read 0x004e5b thru 0x007dc1 (0x2f67 bytes) 000552 1726304240.303403 trunc from 0x007dc2 to 0x03629e (0x2e4dd bytes) 000553 1726304240.310175 MIRROR_EXTEND to 10 mirrors 000554 1726304240.454418 punch from 0x0064b1 to 0x0138c8 (0xd418 bytes) 000555 1726304240.467238 mapread 0x02495d thru 0x02593a (0xfde bytes) 000556 1726304240.470769 read 0x01a0e7 thru 0x020962 (0x687c bytes) 000557 1726304240.475891 trunc from 0x03629f to 0x000407 (0x35e97 bytes) 000558 1726304240.491074 mapread 0x000064 thru 0x000407 (0x3a4 bytes) 000559 1726304240.493817 read 0x0001ad thru 0x000407 (0x25b bytes) 000560 1726304240.494233 mapread 0x0003e4 thru 0x000407 (0x24 bytes) 000561 1726304240.494744 read 0x000267 thru 0x000407 (0x1a1 bytes) 000562 1726304240.497422 MIRROR_RESYNC 10 mirrors 000563 1726304240.689548 punch from 0x000236 to 0x000407 (0x1d2 bytes) 000564 1726304240.700896 punch from 0x0001f6 to 0x000407 (0x212 bytes) 000565 1726304240.704180 MIRROR_EXTEND to 11 mirrors 000566 1726304240.806538 read 0x0001fc thru 0x000407 (0x20c bytes) 000567 1726304240.808165 write 0x02a099 thru 0x0378f0 (0xd858 bytes) 000568 1726304240.822569 punch from 0x024c29 to 0x02ac24 (0x5ffc bytes) 000569 1726304240.830350 MIRROR_RESYNC 11 mirrors 000571 1726304241.246858 fallocate 0x03b5ab thru 0x03ffff (0x4a55 bytes) 000572 1726304241.260929 fallocate 0x03ba98 thru 0x03ffff (0x4568 bytes) 000574 1726304241.263203 punch from 0x01db3c to 0x02d10d (0xf5d2 bytes) 000575 1726304241.265649 write 0x03a8b0 thru 0x03ffff (0x5750 bytes) 000576 1726304241.269429 mapwrite 0x02c29d thru 0x03454f (0x82b3 bytes) 000577 1726304241.287332 punch from 0x036e17 to 0x03f343 (0x852d bytes) 000578 1726304241.290079 mapread 0x01ec40 thru 0x029d91 (0xb152 bytes) 000579 1726304241.294622 trunc from 0x040000 to 0x03aa99 (0x5566 bytes) 000580 1726304241.304033 read 0x024f07 thru 0x02f70a (0xa804 bytes) 000581 1726304241.304510 fallocate 0x0120f4 thru 0x01e1fe (0xc10b bytes) 000582 1726304241.309372 MIRROR_RESYNC 11 mirrors 000583 1726304241.846619 write 0x03873e thru 0x03ffff (0x78c2 bytes) 000584 1726304241.863756 trunc from 0x040000 to 0x01a153 (0x25eac bytes) 000585 1726304241.882105 punch from 0x005501 to 0x006e8c (0x198c bytes) 000586 1726304241.884227 mapwrite 0x00f337 thru 0x01672c (0x73f6 bytes) 000587 1726304241.907280 MIRROR_RESYNC 11 mirrors 000588 1726304242.423438 write 0x02e89f thru 0x0367be (0x7f20 bytes) 000589 1726304242.435527 trunc from 0x0367bf to 0x014e6b (0x21953 bytes) 000592 1726304242.442908 read 0x003eb5 thru 0x0079f8 (0x3b44 bytes) 000594 1726304242.447038 punch from 0x0047b6 to 0x0120fd (0xd948 bytes) 000595 1726304242.449927 trunc from 0x014e6c to 0x02a00f (0x151a4 bytes) 000596 1726304242.456520 write 0x00de02 thru 0x00fda7 (0x1fa6 bytes) 000597 1726304242.463177 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000598 1726304242.552079 mapread 0x00f1ce thru 0x01a10d (0xaf40 bytes) 000599 1726304242.563942 trunc from 0x02a010 to 0x00d7f1 (0x1c81e bytes) 000600 1726304242.574943 write 0x011527 thru 0x01bf71 (0xaa4b bytes) 000601 1726304242.578450 MIRROR_RESYNC 10 mirrors 000602 1726304243.124695 fallocate 0x02e629 thru 0x037fdf (0x99b7 bytes) 000603 1726304243.139132 mapwrite 0x03d214 thru 0x03ffff (0x2dec bytes) 000604 1726304243.161924 mapwrite 0x03c8ea thru 0x03ffff (0x3716 bytes) 000605 1726304243.170262 punch from 0x01488e to 0x01ed19 (0xa48c bytes) 000607 1726304243.172820 trunc from 0x040000 to 0x03a61e (0x59e1 bytes) 000608 1726304243.180881 fallocate 0x028d4b thru 0x0317e2 (0x8a98 bytes) 000609 1726304243.183522 trunc from 0x03a61f to 0x039959 (0xcc5 bytes) 000612 1726304243.192875 MIRROR_RESYNC 10 mirrors 000613 1726304243.806735 trunc from 0x03995a to 0x01ee70 (0x1aae9 bytes) 000614 1726304243.823013 MIRROR_EXTEND to 11 mirrors 000615 1726304244.060175 trunc from 0x01ee71 to 0x00d2a9 (0x11bc7 bytes) 000616 1726304244.080200 MIRROR_RESYNC 11 mirrors 000617 1726304244.593937 MIRROR_RESYNC 11 mirrors 000618 1726304244.685276 punch from 0x0002a5 to 0x00d2a9 (0xd005 bytes) 000619 1726304244.701275 mapwrite 0x038b01 thru 0x03ea48 (0x5f48 bytes) 000620 1726304244.727689 write 0x0372b8 thru 0x03e1bb (0x6f04 bytes) 000621 1726304244.728770 punch from 0x01316e to 0x01b7d7 (0x866a bytes) 000622 1726304244.733424 punch from 0x02412d to 0x028909 (0x47dd bytes) 000623 1726304244.741349 MIRROR_RESYNC 11 mirrors 000624 1726304245.413800 MIRROR_EXTEND to 12 mirrors 000625 1726304245.643930 mapwrite 0x03b884 thru 0x03ffff (0x477c bytes) 000626 1726304245.690493 write 0x022ab1 thru 0x03090a (0xde5a bytes) 000629 1726304245.695802 mapwrite 0x00b45f thru 0x00b8ac (0x44e bytes) 000630 1726304245.704515 mapread 0x0012f8 thru 0x002aec (0x17f5 bytes) 000631 1726304245.707155 mapwrite 0x02bd74 thru 0x038f33 (0xd1c0 bytes) 000633 1726304245.718654 mapwrite 0x035adb thru 0x03ffff (0xa525 bytes) 000634 1726304245.730360 trunc from 0x040000 to 0x03bf2f (0x40d0 bytes) 000635 1726304245.740197 mapread 0x033633 thru 0x037604 (0x3fd2 bytes) 000636 1726304245.740837 mapwrite 0x017bf3 thru 0x026046 (0xe454 bytes) 000637 1726304245.753993 mapwrite 0x030e3c thru 0x033b95 (0x2d5a bytes) 000638 1726304245.762478 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000639 1726304245.819124 fallocate 0x01b2e1 thru 0x0292b5 (0xdfd5 bytes) 000640 1726304245.821356 mapread 0x01c911 thru 0x0272b8 (0xa9a8 bytes) 000641 1726304245.831829 mapwrite 0x0247ac thru 0x02e7a2 (0x9ff7 bytes) 000642 1726304245.842126 fallocate 0x00c9ea thru 0x014c3a (0x8251 bytes) 000643 1726304245.844607 read 0x02e8e2 thru 0x03100d (0x272c bytes) 000645 1726304245.847072 MIRROR_RESYNC 11 mirrors 000646 1726304246.397204 mapread 0x036560 thru 0x03bf2f (0x59d0 bytes) 000647 1726304246.403734 write 0x02127b thru 0x028900 (0x7686 bytes) 000649 1726304246.419108 fallocate 0x03d333 thru 0x03ffff (0x2ccd bytes) 000650 1726304246.422176 punch from 0x026806 to 0x034c76 (0xe471 bytes) 000651 1726304246.427553 mapwrite 0x019e81 thru 0x0226d6 (0x8856 bytes) 000652 1726304246.438594 mapwrite 0x010c11 thru 0x01c213 (0xb603 bytes) 000655 1726304246.448808 fallocate 0x00f1eb thru 0x00fd9f (0xbb5 bytes) 000656 1726304246.450904 write 0x005a36 thru 0x00c56d (0x6b38 bytes) 000657 1726304246.454243 write 0x033002 thru 0x03ecfd (0xbcfc bytes) 000658 1726304246.458875 read 0x003083 thru 0x00ea39 (0xb9b7 bytes) 000660 1726304246.459654 write 0x015d6d thru 0x01ef4e (0x91e2 bytes) 000661 1726304246.460433 trunc from 0x040000 to 0x014d6a (0x2b295 bytes) 000662 1726304246.466265 punch from 0x010e71 to 0x014d6a (0x3efa bytes) 000663 1726304246.467955 mapread 0x008f53 thru 0x014d6a (0xbe18 bytes) 000665 1726304246.471151 mapread 0x004dca thru 0x00c5d6 (0x780d bytes) 000666 1726304246.471675 trunc from 0x014d6b to 0x033039 (0x1e2cf bytes) 000667 1726304246.477525 MIRROR_EXTEND to 12 mirrors 000668 1726304246.643472 punch from 0x0172fd to 0x0271bd (0xfec1 bytes) 000669 1726304246.655326 MIRROR_RESYNC 12 mirrors 000670 1726304247.271111 punch from 0x00bcd4 to 0x01a602 (0xe92f bytes) 000671 1726304247.282063 write 0x03f7b0 thru 0x03ffff (0x850 bytes) 000672 1726304247.282712 fallocate 0x0396f9 thru 0x03dac7 (0x43cf bytes) 000673 1726304247.284722 punch from 0x016087 to 0x0169c8 (0x942 bytes) 000675 1726304247.286708 read 0x025b7f thru 0x02b7fe (0x5c80 bytes) 000676 1726304247.289440 read 0x03cc9d thru 0x03ffff (0x3363 bytes) 000677 1726304247.295390 mapwrite 0x020ccf thru 0x023f23 (0x3255 bytes) 000678 1726304247.305741 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000679 1726304247.362452 fallocate 0x016971 thru 0x01c448 (0x5ad8 bytes) 000680 1726304247.364536 write 0x017704 thru 0x021ab1 (0xa3ae bytes) 000681 1726304247.371315 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000682 1726304247.436247 mapread 0x00b2dd thru 0x01aba5 (0xf8c9 bytes) 000683 1726304247.449832 punch from 0x00881c to 0x018454 (0xfc39 bytes) 000686 1726304247.455460 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000687 1726304247.525474 mapwrite 0x03c802 thru 0x03ffff (0x37fe bytes) 000689 1726304247.533809 punch from 0x00c02f to 0x0124c1 (0x6493 bytes) 000690 1726304247.535311 fallocate 0x03a046 thru 0x03ffff (0x5fba bytes) 000691 1726304247.536878 fallocate 0x02490d thru 0x02d582 (0x8c76 bytes) 000692 1726304247.538607 mapwrite 0x033d1d thru 0x03e509 (0xa7ed bytes) 000693 1726304247.548018 fallocate 0x00dd73 thru 0x00f740 (0x19ce bytes) 000694 1726304247.549635 mapread 0x01834b thru 0x0183cf (0x85 bytes) 000695 1726304247.553007 mapread 0x01b11d thru 0x0243df (0x92c3 bytes) 000696 1726304247.556625 fallocate 0x03d9e4 thru 0x03ffff (0x261c bytes) 000697 1726304247.558924 mapwrite 0x01295c thru 0x01da66 (0xb10b bytes) 000698 1726304247.567750 mapwrite 0x03ae95 thru 0x03d38c (0x24f8 bytes) 000699 1726304247.573632 mapread 0x036473 thru 0x03ffff (0x9b8d bytes) 000700 1726304247.574262 mapwrite 0x0262b7 thru 0x03301a (0xcd64 bytes) 000701 1726304247.582235 trunc from 0x040000 to 0x02d03b (0x12fc4 bytes) 000702 1726304247.588348 write 0x01b58f thru 0x02b15f (0xfbd1 bytes) 000703 1726304247.589374 punch from 0x029bcb to 0x02d03b (0x3471 bytes) 000704 1726304247.594304 mapread 0x01c233 thru 0x02775d (0xb52b bytes) 000707 1726304247.597422 MIRROR_RESYNC 9 mirrors 000708 1726304248.156681 MIRROR_EXTEND to 10 mirrors 000709 1726304248.380276 fallocate 0x0031f4 thru 0x010a76 (0xd883 bytes) 000710 1726304248.393868 read 0x01e6b1 thru 0x020942 (0x2292 bytes) 000711 1726304248.398236 mapwrite 0x0126c3 thru 0x01fbcd (0xd50b bytes) 000712 1726304248.420616 MIRROR_RESYNC 10 mirrors 000714 1726304249.127739 mapread 0x01ff99 thru 0x021973 (0x19db bytes) 000715 1726304249.135659 fallocate 0x030c67 thru 0x03397c (0x2d16 bytes) 000716 1726304249.154444 mapread 0x02bcf7 thru 0x02d03b (0x1345 bytes) 000717 1726304249.158546 write 0x00b64a thru 0x0125eb (0x6fa2 bytes) 000718 1726304249.162582 mapread 0x0191ef thru 0x02495a (0xb76c bytes) 000719 1726304249.173127 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000720 1726304249.245598 trunc from 0x02d03c to 0x015e88 (0x171b3 bytes) 000721 1726304249.256261 MIRROR_RESYNC 9 mirrors 000722 1726304249.620055 read 0x012875 thru 0x015e88 (0x3614 bytes) 000723 1726304249.623736 fallocate 0x03d992 thru 0x03ffff (0x266e bytes) 000725 1726304249.638671 trunc from 0x040000 to 0x022157 (0x1dea8 bytes) 000726 1726304249.646324 punch from 0x00ec24 to 0x018709 (0x9ae6 bytes) 000727 1726304249.650297 mapread 0x006d74 thru 0x009b27 (0x2db4 bytes) 000728 1726304249.660862 mapwrite 0x037cb5 thru 0x03ffff (0x834b bytes) 000729 1726304249.695315 MIRROR_EXTEND to 10 mirrors 000730 1726304249.874851 mapwrite 0x009251 thru 0x00d0fc (0x3eac bytes) 000734 1726304249.936390 MIRROR_EXTEND to 11 mirrors 000736 1726304250.146654 mapread 0x02706c thru 0x02c591 (0x5526 bytes) 000738 1726304250.161838 mapwrite 0x033886 thru 0x0353ac (0x1b27 bytes) 000739 1726304250.220694 write 0x014600 thru 0x020465 (0xbe66 bytes) 000741 1726304250.230572 punch from 0x03d1b6 to 0x03ffff (0x2e4a bytes) 000742 1726304250.233005 write 0x03b4c2 thru 0x03ffff (0x4b3e bytes) 000743 1726304250.234387 write 0x01edc6 thru 0x02a115 (0xb350 bytes) 000744 1726304250.237762 trunc from 0x040000 to 0x0006ec (0x3f913 bytes) 000745 1726304250.247093 read 0x000299 thru 0x0006ec (0x454 bytes) 000747 1726304250.250175 punch from 0x0001ac to 0x0006ec (0x541 bytes) 000748 1726304250.252739 write 0x022ced thru 0x032628 (0xf93c bytes) 000749 1726304250.256302 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000750 1726304250.339421 mapwrite 0x036dce thru 0x03f29e (0x84d1 bytes) 000751 1726304250.358288 write 0x025795 thru 0x0269ed (0x1259 bytes) 000752 1726304250.361066 read 0x000c1b thru 0x00b8a4 (0xac8a bytes) 000753 1726304250.363904 fallocate 0x02ecab thru 0x03a2ed (0xb643 bytes) 000754 1726304250.366337 mapwrite 0x030974 thru 0x03d119 (0xc7a6 bytes) 000755 1726304250.378983 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000756 1726304250.446297 mapwrite 0x01525b thru 0x0235e3 (0xe389 bytes) 000757 1726304250.463401 fallocate 0x0155a9 thru 0x01a8a3 (0x52fb bytes) 000759 1726304250.465323 write 0x021e8e thru 0x025cff (0x3e72 bytes) 000760 1726304250.467530 write 0x023060 thru 0x02bc22 (0x8bc3 bytes) 000761 1726304250.469792 write 0x005388 thru 0x013ae5 (0xe75e bytes) 000762 1726304250.473733 mapwrite 0x00f9d9 thru 0x013e46 (0x446e bytes) 000763 1726304250.481690 mapwrite 0x0383be thru 0x03d85d (0x54a0 bytes) 000764 1726304250.493347 read 0x0397ee thru 0x03e9e8 (0x51fb bytes) 000765 1726304250.493851 fallocate 0x003435 thru 0x00e70b (0xb2d7 bytes) 000766 1726304250.497079 write 0x03e4e8 thru 0x03ffff (0x1b18 bytes) 000767 1726304250.497774 mapread 0x01d684 thru 0x01e292 (0xc0f bytes) 000769 1726304250.498721 mapread 0x00689b thru 0x0135e0 (0xcd46 bytes) 000770 1726304250.499475 punch from 0x036c90 to 0x03fe57 (0x91c8 bytes) 000771 1726304250.506659 MIRROR_RESYNC 9 mirrors 000772 1726304250.982758 mapread 0x032459 thru 0x03c665 (0xa20d bytes) 000774 1726304250.992454 read 0x024915 thru 0x02a621 (0x5d0d bytes) 000775 1726304250.995841 fallocate 0x017238 thru 0x01be7a (0x4c43 bytes) 000776 1726304251.007822 read 0x0276c8 thru 0x02b122 (0x3a5b bytes) 000777 1726304251.012671 punch from 0x02b438 to 0x02d679 (0x2242 bytes) 000778 1726304251.017351 MIRROR_EXTEND to 10 mirrors 000779 1726304251.187688 mapread 0x0289ab thru 0x031db4 (0x940a bytes) 000781 1726304251.198523 read 0x024a15 thru 0x0276c0 (0x2cac bytes) 000782 1726304251.202175 fallocate 0x030044 thru 0x038d3f (0x8cfc bytes) 000783 1726304251.214208 trunc from 0x040000 to 0x03c19a (0x3e65 bytes) 000785 1726304251.222666 read 0x00c446 thru 0x013678 (0x7233 bytes) 000786 1726304251.226843 mapwrite 0x02ca9e thru 0x0366e7 (0x9c4a bytes) 000787 1726304251.236107 mapread 0x01df20 thru 0x024f01 (0x6fe2 bytes) 000788 1726304251.236668 mapwrite 0x03b2d5 thru 0x03ffff (0x4d2b bytes) 000791 1726304251.252122 trunc from 0x040000 to 0x008899 (0x37766 bytes) 000792 1726304251.257637 mapread 0x001b47 thru 0x008899 (0x6d53 bytes) 000793 1726304251.259811 MIRROR_EXTEND to 11 mirrors 000794 1726304251.384204 read 0x0060f3 thru 0x008135 (0x2043 bytes) 000796 1726304251.390951 punch from 0x0038da to 0x00635f (0x2a86 bytes) 000797 1726304251.402617 write 0x0241ec thru 0x02e653 (0xa468 bytes) 000798 1726304251.405033 write 0x03756d thru 0x03ffff (0x8a93 bytes) 000799 1726304251.405874 mapwrite 0x02e425 thru 0x03a2ca (0xbea6 bytes) 000800 1726304251.418053 mapread 0x02faa8 thru 0x039247 (0x97a0 bytes) 000801 1726304251.418613 write 0x03c6ad thru 0x03ffff (0x3953 bytes) 000802 1726304251.419244 punch from 0x03d72d to 0x03ffff (0x28d3 bytes) 000803 1726304251.423007 mapread 0x00a97f thru 0x00e1b2 (0x3834 bytes) 000804 1726304251.425886 punch from 0x03ccd5 to 0x03ee84 (0x21b0 bytes) 000805 1726304251.427899 mapwrite 0x006f67 thru 0x00f4f8 (0x8592 bytes) 000807 1726304251.436258 write 0x03aa59 thru 0x03ffff (0x55a7 bytes) 000808 1726304251.436940 mapwrite 0x02b42b thru 0x030628 (0x51fe bytes) 000809 1726304251.443311 write 0x02744a thru 0x036dd1 (0xf988 bytes) 000810 1726304251.444185 mapread 0x011853 thru 0x01c1cd (0xa97b bytes) 000811 1726304251.449818 trunc from 0x040000 to 0x00af08 (0x350f7 bytes) 000813 1726304251.460042 write 0x005b3e thru 0x011925 (0xbde8 bytes) 000814 1726304251.460870 fallocate 0x029c1e thru 0x029d5e (0x141 bytes) 000815 1726304251.462522 read 0x00d656 thru 0x012a22 (0x53cd bytes) 000817 1726304251.465817 mapwrite 0x010bca thru 0x011e34 (0x126b bytes) 000818 1726304251.473037 trunc from 0x029d5f to 0x017db1 (0x11fad bytes) 000819 1726304251.476925 mapwrite 0x00920c thru 0x018031 (0xee26 bytes) 000820 1726304251.491678 mapwrite 0x009389 thru 0x00e438 (0x50b0 bytes) 000821 1726304251.498906 read 0x007b6b thru 0x0108ca (0x8d60 bytes) 000822 1726304251.500909 MIRROR_RESYNC 11 mirrors 000823 1726304252.025156 mapread 0x002422 thru 0x003fe5 (0x1bc4 bytes) 000825 1726304252.032096 MIRROR_RESYNC 11 mirrors 000826 1726304252.126201 trunc from 0x018032 to 0x018631 (0x600 bytes) 000827 1726304252.146713 MIRROR_RESYNC 11 mirrors 000828 1726304252.726071 punch from 0x00cf49 to 0x0152de (0x8396 bytes) 000829 1726304252.738026 mapwrite 0x00a80b thru 0x018495 (0xdc8b bytes) 000830 1726304252.761409 MIRROR_EXTEND to 12 mirrors 000831 1726304252.927991 fallocate 0x039c99 thru 0x03ffff (0x6367 bytes) 000832 1726304252.945211 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000833 1726304253.007710 read 0x01647c thru 0x018631 (0x21b6 bytes) 000834 1726304253.012565 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000836 1726304253.090576 trunc from 0x018632 to 0x034cf6 (0x1c6c5 bytes) 000837 1726304253.104456 read 0x002de0 thru 0x00adb5 (0x7fd6 bytes) 000838 1726304253.109067 fallocate 0x006473 thru 0x0161a1 (0xfd2f bytes) 000839 1726304253.111585 read 0x00886f thru 0x008e99 (0x62b bytes) 000841 1726304253.112279 write 0x02538a thru 0x032cf7 (0xd96e bytes) 000842 1726304253.117207 mapwrite 0x00f797 thru 0x017dc4 (0x862e bytes) 000843 1726304253.132517 punch from 0x01d79b to 0x0243df (0x6c45 bytes) 000844 1726304253.135394 punch from 0x00621a to 0x006ca5 (0xa8c bytes) 000845 1726304253.138269 fallocate 0x038364 thru 0x03dfd5 (0x5c72 bytes) 000846 1726304253.140553 fallocate 0x0028be thru 0x00da6a (0xb1ad bytes) 000847 1726304253.143008 read 0x0161aa thru 0x01f498 (0x92ef bytes) 000848 1726304253.147488 mapwrite 0x00201c thru 0x00aff0 (0x8fd5 bytes) 000851 1726304253.161244 mapwrite 0x00a9a5 thru 0x014c4d (0xa2a9 bytes) 000852 1726304253.170895 write 0x02c77e thru 0x032a04 (0x6287 bytes) 000853 1726304253.171919 mapwrite 0x02e4cf thru 0x037c0f (0x9741 bytes) 000854 1726304253.193080 fallocate 0x03a6ae thru 0x03ffff (0x5952 bytes) 000855 1726304253.195677 mapread 0x009c24 thru 0x011505 (0x78e2 bytes) 000857 1726304253.196768 mapwrite 0x026cbf thru 0x02ff45 (0x9287 bytes) 000858 1726304253.204609 write 0x0360c6 thru 0x03bbc6 (0x5b01 bytes) 000859 1726304253.207493 write 0x02b107 thru 0x0344ce (0x93c8 bytes) 000861 1726304253.208476 read 0x02129a thru 0x028354 (0x70bb bytes) 000862 1726304253.208872 mapwrite 0x0014e8 thru 0x009fc1 (0x8ada bytes) 000863 1726304253.220151 mapwrite 0x016533 thru 0x0235b5 (0xd083 bytes) 000864 1726304253.229121 mapread 0x0161aa thru 0x02245e (0xc2b5 bytes) 000865 1726304253.229736 mapwrite 0x0305d5 thru 0x036b08 (0x6534 bytes) 000867 1726304253.239152 MIRROR_RESYNC 10 mirrors 000868 1726304253.601471 write 0x01a7a7 thru 0x020580 (0x5dda bytes) 000869 1726304253.620411 mapread 0x01ab95 thru 0x02a5a0 (0xfa0c bytes) 000870 1726304253.627448 trunc from 0x040000 to 0x00c937 (0x336c8 bytes) 000871 1726304253.639484 write 0x02ed38 thru 0x03a195 (0xb45e bytes) 000872 1726304253.640683 read 0x01e854 thru 0x026dff (0x85ac bytes) 000873 1726304253.644158 trunc from 0x03a196 to 0x00cbf9 (0x2d59c bytes) 000874 1726304253.650874 read 0x000f13 thru 0x00a53a (0x9628 bytes) 000875 1726304253.654737 fallocate 0x016e8c thru 0x018207 (0x137c bytes) 000876 1726304253.657001 punch from 0x002cb4 to 0x00cbf9 (0x9f46 bytes) 000877 1726304253.659291 mapread 0x0068ea thru 0x00b866 (0x4f7d bytes) 000878 1726304253.667944 read 0x0077b9 thru 0x007faf (0x7f7 bytes) 000880 1726304253.671276 MIRROR_EXTEND to 11 mirrors 000881 1726304253.808215 write 0x03e4c6 thru 0x03ffff (0x1b3a bytes) 000882 1726304253.820954 MIRROR_RESYNC 11 mirrors 000883 1726304254.356006 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000884 1726304254.495481 trunc from 0x040000 to 0x00cb7f (0x33480 bytes) 000885 1726304254.516020 punch from 0x003ecc to 0x00cb7f (0x8cb4 bytes) 000887 1726304254.519032 read 0x00c568 thru 0x00cb7f (0x618 bytes) 000889 1726304254.524192 MIRROR_RESYNC 10 mirrors 000890 1726304254.897750 fallocate 0x012ce5 thru 0x01f5c2 (0xc8de bytes) 000892 1726304254.910735 read 0x00092f thru 0x008f9c (0x866e bytes) 000893 1726304254.913722 punch from 0x009891 to 0x00cb7f (0x32ef bytes) 000894 1726304254.916046 write 0x015e16 thru 0x0194ea (0x36d5 bytes) 000895 1726304254.916916 mapwrite 0x022e60 thru 0x02dc1b (0xadbc bytes) 000897 1726304254.948718 fallocate 0x005f26 thru 0x011100 (0xb1db bytes) 000898 1726304254.951088 fallocate 0x02229d thru 0x02e7f5 (0xc559 bytes) 000899 1726304254.955620 MIRROR_RESYNC 10 mirrors 000900 1726304255.613893 MIRROR_RESYNC 10 mirrors 000901 1726304255.688563 write 0x00c85c thru 0x011130 (0x48d5 bytes) 000902 1726304255.706007 MIRROR_EXTEND to 11 mirrors 000903 1726304255.848078 mapwrite 0x0324d1 thru 0x03ffff (0xdb2f bytes) 000904 1726304255.889092 mapread 0x02643b thru 0x02a940 (0x4506 bytes) 000906 1726304255.893324 write 0x005880 thru 0x0099d8 (0x4159 bytes) 000907 1726304255.897014 write 0x03a090 thru 0x03ffff (0x5f70 bytes) 000909 1726304255.897852 mapwrite 0x03decf thru 0x03ffff (0x2131 bytes) 000910 1726304255.904673 mapread 0x00f697 thru 0x0190d1 (0x9a3b bytes) 000912 1726304255.913581 punch from 0x0071fe to 0x00f908 (0x870b bytes) 000916 1726304255.917162 fallocate 0x007fcf thru 0x0124c7 (0xa4f9 bytes) 000917 1726304255.920406 fallocate 0x022de1 thru 0x023102 (0x322 bytes) 000918 1726304255.922788 fallocate 0x0240c2 thru 0x02f9be (0xb8fd bytes) 000919 1726304255.925041 mapread 0x0309ab thru 0x03adf9 (0xa44f bytes) 000920 1726304255.925647 mapread 0x00497e thru 0x00983d (0x4ec0 bytes) 000921 1726304255.931436 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000923 1726304255.998942 fallocate 0x01f94f thru 0x01faac (0x15e bytes) 000924 1726304256.001055 fallocate 0x026e19 thru 0x026f37 (0x11f bytes) 000925 1726304256.003116 fallocate 0x0276d8 thru 0x036072 (0xe99b bytes) 000926 1726304256.005489 mapread 0x00cb82 thru 0x01b852 (0xecd1 bytes) 000927 1726304256.014674 write 0x028905 thru 0x037973 (0xf06f bytes) 000928 1726304256.022904 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000929 1726304256.092592 read 0x02b9fc thru 0x02e6e2 (0x2ce7 bytes) 000931 1726304256.095793 mapwrite 0x03cf20 thru 0x03edf1 (0x1ed2 bytes) 000933 1726304256.109598 fallocate 0x00729b thru 0x00c9dd (0x5743 bytes) 000934 1726304256.111506 punch from 0x03f617 to 0x03ffff (0x9e9 bytes) 000936 1726304256.113596 mapwrite 0x01371b thru 0x01cf66 (0x984c bytes) 000937 1726304256.127033 write 0x012c19 thru 0x01e6b2 (0xba9a bytes) 000938 1726304256.129732 punch from 0x02a6fa to 0x036bff (0xc506 bytes) 000941 1726304256.132362 read 0x032f16 thru 0x038e60 (0x5f4b bytes) 000944 1726304256.136200 mapread 0x03e947 thru 0x03ffff (0x16b9 bytes) 000946 1726304256.136997 write 0x01050e thru 0x01cfff (0xcaf2 bytes) 000947 1726304256.137880 read 0x00924f thru 0x018373 (0xf125 bytes) 000948 1726304256.138270 punch from 0x035bb3 to 0x038737 (0x2b85 bytes) 000949 1726304256.140386 fallocate 0x03ecb4 thru 0x03ffff (0x134c bytes) 000950 1726304256.141948 trunc from 0x040000 to 0x02507b (0x1af84 bytes) 000951 1726304256.146169 read 0x010a70 thru 0x0111e8 (0x779 bytes) 000952 1726304256.146576 read 0x0181c9 thru 0x0202d0 (0x8108 bytes) 000953 1726304256.146990 read 0x01e0df thru 0x02507b (0x6f9d bytes) 000954 1726304256.148947 MIRROR_RESYNC 9 mirrors 000956 1726304256.536007 mapwrite 0x012268 thru 0x01f89a (0xd633 bytes) 000957 1726304256.587240 write 0x01c03f thru 0x0277c1 (0xb783 bytes) 000958 1726304256.588273 mapread 0x003f54 thru 0x007f65 (0x4012 bytes) 000959 1726304256.594548 mapwrite 0x034345 thru 0x03ffff (0xbcbb bytes) 000961 1726304256.638561 mapwrite 0x012c0b thru 0x01bc2f (0x9025 bytes) 000963 1726304256.646154 trunc from 0x040000 to 0x00d279 (0x32d86 bytes) 000965 1726304256.652662 punch from 0x006b27 to 0x00b039 (0x4513 bytes) 000966 1726304256.654987 write 0x039a48 thru 0x03ffff (0x65b8 bytes) 000967 1726304256.655800 mapread 0x013005 thru 0x01809b (0x5097 bytes) 000968 1726304256.659318 fallocate 0x0039b7 thru 0x012716 (0xed60 bytes) 000970 1726304256.661716 mapread 0x00b493 thru 0x018260 (0xcdce bytes) 000971 1726304256.665152 read 0x036140 thru 0x039285 (0x3146 bytes) 000974 1726304256.667643 read 0x03a2be thru 0x03ffff (0x5d42 bytes) 000975 1726304256.667942 trunc from 0x040000 to 0x0141a6 (0x2be59 bytes) 000976 1726304256.672704 punch from 0x0128e6 to 0x012f8a (0x6a5 bytes) 000977 1726304256.674360 trunc from 0x0141a7 to 0x03cedc (0x28d36 bytes) 000978 1726304256.679712 fallocate 0x0214b1 thru 0x02a302 (0x8e52 bytes) 000981 1726304256.682297 mapread 0x03c2c2 thru 0x03cedc (0xc1b bytes) 000982 1726304256.687615 trunc from 0x03cedd to 0x02e4e6 (0xe9f6 bytes) 000983 1726304256.693474 mapwrite 0x007323 thru 0x0139d3 (0xc6b1 bytes) 000986 1726304256.704570 trunc from 0x02e4e7 to 0x000f00 (0x2d5e6 bytes) 000987 1726304256.713340 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x84a:0x0]: Invalid argument (22) 000988 1726304256.796076 write 0x01a90b thru 0x01d150 (0x2846 bytes) 000989 1726304256.798411 fallocate 0x0141d3 thru 0x01561b (0x1449 bytes) 000990 1726304256.801313 write 0x0332bf thru 0x0337c0 (0x502 bytes) 000991 1726304256.802596 fallocate 0x0071d2 thru 0x00e00a (0x6e39 bytes) 000992 1726304256.805186 write 0x00f99b thru 0x016792 (0x6df8 bytes) 000993 1726304256.810660 read 0x003671 thru 0x008c21 (0x55b1 bytes) 000994 1726304256.815627 fallocate 0x03cf19 thru 0x03dbb7 (0xc9f bytes) 000995 1726304256.818771 write 0x028942 thru 0x036b77 (0xe236 bytes) 000996 1726304256.828192 write 0x009cd5 thru 0x01334c (0x9678 bytes) 000997 1726304256.834071 MIRROR_EXTEND to 9 mirrors 000999 1726304257.046840 fallocate 0x005c3f thru 0x012c15 (0xcfd7 bytes) 001000 1726304257.064810 trunc from 0x03dbb8 to 0x031886 (0xc331 bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 200a: stress test ===================== 04:57:38 (1726304258) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8359616 .. Extending file size to 5525760 .. done Extending file size to 7550368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8215488 .. Extending file size to 8982944 .. Extending file size to 8931232 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9182528 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9017952 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9231072 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2900 2901 2903 2904 2907 done 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) 192.168.201.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:58:42 (1726304322) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1377536 .. Extending file size to 938656 .. Extending file size to 8844480 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7573632 .. Extending file size to 8200768 .. Extending file size to 8616928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8925664 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8874080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8943072 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 5679 5680 5682 5684 5686 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) 192.168.201.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:59:46 (1726304386) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15500 3591520 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21188 3585832 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36688 7177352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15500 3591520 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21188 3585832 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36688 7177352 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.034471 s, 60.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0496552 s, 42.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:59:51 (1726304391) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.03012 s, 69.6 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 04:59:52 (1726304392) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:59:54 (1726304394) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0282695 s, 74.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:59:56 (1726304396) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274356 s, 76.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:59:58 (1726304398) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0260091 s, 80.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:00:21 (1726304421) Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150766 s, 69.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa0e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa0e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa0f:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:00:54 (1726304454) Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0262019 s, 80.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:01:26 (1726304486) Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg109-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg109-server: Pool lustre.test_204f created Adding targets to pool oleg109-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg109-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0243091 s, 86.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa0e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg109-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg109-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg109-server: Pool lustre.test_204f destroyed PASS 204f (44s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:02:10 (1726304530) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:02:12 (1726304532) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:02:14 (1726304534) oleg109-server: Pool lustre.test_206 created Adding targets to pool oleg109-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg109-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg109-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg109-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg109-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:02:27 (1726304547) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0222273 s, 47.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011963 s, 87.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xc297f6c Mirror 3: 0xc297f6c PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:02:30 (1726304550) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:02:31 (1726304551) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:02:31 (1726304551) Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00873458 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x820000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1410000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.201.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg109-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:02:45 (1726304565) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00279834 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:02:47 (1726304567) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116826 s, 89.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-09-14 05:02:47.000000000 -0400 Modify: 2024-09-14 05:02:47.000000000 -0400 Change: 2024-09-14 05:02:47.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:02:49 (1726304569) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00987346 s, 106 MB/s oleg109-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg109-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 oleg109-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg109-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg109-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test complete, duration 763 sec ============ 05:03:02 (1726304582) === sanity-flr: start cleanup 05:03:02 (1726304582) === === sanity-flr: finish cleanup 05:03:03 (1726304583) ===