-----============= acceptance-small: sanity-flr ============----- Fri Apr 25 18:23:18 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg445-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:23:25 (1745619805) === oleg445-client.virtnet: executing check_config_client /mnt/lustre oleg445-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg445-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b36f8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b36f8800.idle_timeout=debug disable quota as required oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:23:39 (1745619819) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7680] == sanity-flr test 0a: lfs mirror create with -N option == 18:23:40 (1745619820) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:23:47 (1745619827) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:23:48 (1745619828) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:23:50 (1745619830) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:24:00 (1745619840) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:24:01 (1745619841) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:24:03 (1745619843) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b36f8800.stats=clear osc.lustre-OST0001-osc-ffff8800b36f8800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:24:06 (1745619846) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:24:11 (1745619851) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:24:14 (1745619854) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:24:15 (1745619855) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 18:24:19 (1745619859) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:24:21 (1745619861) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:24:24 (1745619864) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:24:27 (1745619867) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:24:31 (1745619871) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:24:34 (1745619874) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.374698 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0315943 s, 33.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:24:37 (1745619877) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0340387 s, 123 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:24:40 (1745619880) Stopping /mnt/lustre-ost1 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg445-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg445-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:25:09 (1745619909) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.27056 s, 42.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg445-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (19s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:25:28 (1745619928) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg445-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg445-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (32s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:26:00 (1745619960) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.07131 s, 48.9 MB/s -rw-r--r-- 1 root root 52428800 Apr 25 18:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.402s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.443s user 0m0.001s sys 0m0.112s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg445-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg445-server oleg445-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.421s user 0m0.000s sys 0m0.122s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg445-server: oleg445-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg445-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b36f8800.ost_server_uuid in FULL state after 0 sec PASS 33b (32s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:26:32 (1745619992) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:26:33 (1745619993) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:26:34 (1745619994) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:26:35 (1745619995) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 18:26:38 (1745619998) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.212899 s, 59.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:26:43 (1745620003) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121555 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0292013 s, 35.9 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:26:45 (1745620005) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:26:49 (1745620009) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00054948 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000651811 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00375538 s, 273 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000699224 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 18:26:53 (1745620013) create test files with size 15739397 .. 40dab8df2026d5dd766a0c8bb327cad9 - 7f5191e8d2faad387239ee411ae0e495 - 0d3fc70ae7b724446efb0e01a6f31915 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 18:27:01 (1745620021) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:27:07 (1745620027) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:27:10 (1745620030) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0417111 s, 25.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161981 s, 64.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 18:27:13 (1745620033) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0377677 s, 55.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.118277 s, 35.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0703824 s, 29.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 18:27:17 (1745620037) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 18:27:18 (1745620038) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:27:20 (1745620040) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192146 s, 54.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 18:27:23 (1745620043) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.141384 s, 22.2 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:27:26 (1745620046) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:27:30 (1745620050) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.103643 s, 101 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:27:47 (1745620067) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b3484800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.148676 s, 70.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a9ad6000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:27:54 (1745620074) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:28:05 (1745620085) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:28:17 (1745620097) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:28:18 (1745620098) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0990838 s, 31.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0607207 s, 51.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 81120c630bd3e74a3e00c098751c3aea - mirror 1 checksum: ad13f7187f882f833cd1e46820fb17bb - mirror 2 checksum: 81120c630bd3e74a3e00c098751c3aea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 25 18:28:19 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 25 18:28:19 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 81120c630bd3e74a3e00c098751c3aea - mirror 1 checksum: ad13f7187f882f833cd1e46820fb17bb - mirror 2 checksum: ad13f7187f882f833cd1e46820fb17bb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 25 18:28:19 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 25 18:28:19 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:28:22 (1745620102) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133744 s, 78.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:28:24 (1745620104) Failing mds1 on oleg445-server Stopping /mnt/lustre-mds1 (opts:) on oleg445-server 18:28:32 (1745620112) shut down facet: mds1 facet_host: oleg445-server facet_failover_host: oleg445-server Failover mds1 to oleg445-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 18:28:45 (1745620125) targets are mounted 18:28:45 (1745620125) facet_failover done oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133344 s, 78.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:28:53 (1745620133) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.011678 s, 1.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00938474 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:28:56 (1745620136) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0189969 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00402501 s, 5.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:28:59 (1745620139) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00725753 s, 2.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:29:01 (1745620141) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0138191 s, 1.5 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:29:03 (1745620143) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00392315 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0151709 s, 270 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:29:06 (1745620146) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:29:12 (1745620152) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:29:31 (1745620171) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:29:49 (1745620189) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:30:07 (1745620207) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 18:30:09 (1745620209) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26369 mirror split pid 26370 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4892: 26369 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4892: 26370 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 18:31:11 (1745620271) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 272 skipping zero length punch hole fallocating to largest ever: 0x13c6e 000002 1745620272.685410 fallocate 0x0115a6 thru 0x013c6d (0x26c8 bytes) skipping zero size read skipping zero length zero range skipping zero length zero range skipping zero length zero range skipping zero size read 000009 1745620272.693818 write 0x016617 thru 0x01f690 (0x907a bytes) 000010 1745620272.694672 mapread 0x0189e8 thru 0x01d164 (0x477d bytes) truncating to largest ever: 0x27e3d 000012 1745620272.695625 trunc from 0x01f691 to 0x027e3c (0x87ac bytes) 000013 1745620272.705167 punch from 0x018250 to 0x01a57f (0x2330 bytes) fallocating to largest ever: 0x2ec9b 000014 1745620272.718323 fallocate 0x02a766 thru 0x02ec9a (0x4535 bytes) fallocating to largest ever: 0x40000 000015 1745620272.721989 fallocate 0x03b607 thru 0x03ffff (0x49f9 bytes) 000017 1745620272.728194 mapread 0x003de6 thru 0x011bc8 (0xdde3 bytes) 000019 1745620272.747974 MIRROR_EXTEND to 2 mirrors 000020 1745620273.022612 trunc from 0x01e468 to 0x02c01a (0xdbb3 bytes) 000021 1745620273.046228 mapread 0x0073b0 thru 0x0136fe (0xc34f bytes) 000022 1745620273.068838 write 0x02936e thru 0x02e942 (0x55d5 bytes) 000023 1745620273.075328 punch from 0x0136f7 to 0x01bfee (0x88f8 bytes) 000024 1745620273.081460 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000025 1745620273.171495 read 0x003756 thru 0x00dc65 (0xa510 bytes) 000026 1745620273.177015 mapwrite 0x03bb5a thru 0x03ffff (0x44a6 bytes) 000027 1745620273.206035 mapwrite 0x0002df thru 0x00c34f (0xc071 bytes) 000028 1745620273.230521 mapwrite 0x01972a thru 0x0278d3 (0xe1aa bytes) 000030 1745620273.263518 trunc from 0x017e25 to 0x018acc (0xca8 bytes) 000031 1745620273.267474 fallocate 0x0088ea thru 0x014c53 (0xc36a bytes) 000032 1745620273.270190 write 0x011226 thru 0x0192ed (0x80c8 bytes) 000033 1745620273.279678 write 0x0148dc thru 0x014eb8 (0x5dd bytes) 000034 1745620273.280097 fallocate 0x021759 thru 0x0307aa (0xf052 bytes) 000035 1745620273.284035 read 0x024f7b thru 0x0276d5 (0x275b bytes) 000036 1745620273.284438 write 0x031928 thru 0x036377 (0x4a50 bytes) 000037 1745620273.290680 fallocate 0x03886c thru 0x03f0b5 (0x684a bytes) 000038 1745620273.294776 fallocate 0x0285b2 thru 0x02aa35 (0x2484 bytes) 000039 1745620273.299750 punch from 0x034ea5 to 0x0375da (0x2736 bytes) 000041 1745620273.314918 write 0x02a816 thru 0x037d3b (0xd526 bytes) 000042 1745620273.319834 trunc from 0x040000 to 0x01a3a0 (0x25c5f bytes) 000043 1745620273.347756 write 0x035826 thru 0x03ffff (0xa7da bytes) 000044 1745620273.349057 mapwrite 0x01f75d thru 0x027f86 (0x882a bytes) 000045 1745620273.387717 fallocate 0x03fb21 thru 0x03ffff (0x4df bytes) 000046 1745620273.390113 trunc from 0x040000 to 0x01db46 (0x224b9 bytes) 000048 1745620273.397023 fallocate 0x01d1bb thru 0x025a17 (0x885d bytes) 000049 1745620273.400087 write 0x01afd3 thru 0x01fd1e (0x4d4c bytes) 000050 1745620273.407892 read 0x01713f thru 0x023340 (0xc202 bytes) 000051 1745620273.413609 mapwrite 0x00af9d thru 0x01547b (0xa4df bytes) 000052 1745620273.431182 read 0x008197 thru 0x00a2e1 (0x214b bytes) 000053 1745620273.431618 mapwrite 0x004522 thru 0x00ccf4 (0x87d3 bytes) 000054 1745620273.440440 read 0x01ac55 thru 0x024ca4 (0xa050 bytes) 000055 1745620273.440926 punch from 0x007d0d to 0x00da33 (0x5d27 bytes) 000056 1745620273.443868 trunc from 0x009c8c to 0x00d07c (0x33f1 bytes) 000058 1745620273.447500 read 0x00e8b5 thru 0x01d00d (0xe759 bytes) 000059 1745620273.447969 read 0x00b472 thru 0x016fe0 (0xbb6f bytes) 000061 1745620273.454188 write 0x036b52 thru 0x03ffff (0x94ae bytes) 000062 1745620273.454990 read 0x0012db thru 0x001b67 (0x88d bytes) 000063 1745620273.455263 trunc from 0x007408 to 0x00fb5d (0x8756 bytes) 000064 1745620273.457709 read 0x02562d thru 0x02b6b6 (0x608a bytes) 000065 1745620273.464496 mapwrite 0x00cb28 thru 0x01406a (0x7543 bytes) 000066 1745620273.477478 read 0x03d063 thru 0x03ffff (0x2f9d bytes) 000067 1745620273.479165 read 0x034cbe thru 0x03bd9b (0x70de bytes) 000069 1745620273.479912 fallocate 0x0100b8 thru 0x017b94 (0x7add bytes) 000070 1745620273.483486 mapwrite 0x03169c thru 0x03c932 (0xb297 bytes) 000071 1745620273.497679 write 0x01c71d thru 0x027297 (0xab7b bytes) 000072 1745620273.498917 punch from 0x028fb6 to 0x035643 (0xc68e bytes) 000073 1745620273.503414 fallocate 0x028b73 thru 0x033d6b (0xb1f9 bytes) 000074 1745620273.507122 punch from 0x03078c to 0x03f5e0 (0xee55 bytes) 000075 1745620273.510482 fallocate 0x03e650 thru 0x03ffff (0x19b0 bytes) 000076 1745620273.519293 MIRROR_EXTEND to 2 mirrors 000078 1745620273.795943 trunc from 0x037edf to 0x039834 (0x1956 bytes) 000079 1745620273.824844 write 0x03169e thru 0x034306 (0x2c69 bytes) 000080 1745620273.841345 MIRROR_EXTEND to 3 mirrors 000083 1745620274.170744 fallocate 0x0325fc thru 0x03a331 (0x7d36 bytes) 000085 1745620274.189696 write 0x02d1c3 thru 0x03c61a (0xf458 bytes) 000086 1745620274.198044 write 0x017e5d thru 0x021d8b (0x9f2f bytes) 000087 1745620274.203648 mapread 0x01e1cc thru 0x02d053 (0xee88 bytes) 000088 1745620274.210819 read 0x015203 thru 0x024daf (0xfbad bytes) 000089 1745620274.211531 mapwrite 0x01f46b thru 0x02345a (0x3ff0 bytes) 000090 1745620274.233226 mapread 0x021232 thru 0x02cf90 (0xbd5f bytes) 000091 1745620274.234278 mapwrite 0x03293d thru 0x03c8c0 (0x9f84 bytes) 000092 1745620274.248382 trunc from 0x040000 to 0x01e7a4 (0x2185b bytes) 000093 1745620274.261835 trunc from 0x01e7a5 to 0x021ba8 (0x3404 bytes) 000094 1745620274.269618 trunc from 0x0139ac to 0x015d88 (0x23dd bytes) 000095 1745620274.274767 write 0x010a59 thru 0x01e028 (0xd5d0 bytes) 000096 1745620274.275790 mapread 0x01d238 thru 0x020dd3 (0x3b9c bytes) 000097 1745620274.280169 read 0x002efd thru 0x010d68 (0xde6c bytes) 000099 1745620274.288041 write 0x01866f thru 0x01a9ca (0x235c bytes) 000100 1745620274.288740 trunc from 0x00d2f8 to 0x01551b (0x8224 bytes) 000101 1745620274.298902 fallocate 0x00f5cb thru 0x01e233 (0xec69 bytes) 000102 1745620274.308642 read 0x0197b8 thru 0x021ba8 (0x83f1 bytes) 000104 1745620274.309347 trunc from 0x021ba9 to 0x0219bc (0x1ec bytes) 000105 1745620274.315838 fallocate 0x007dac thru 0x00a711 (0x2966 bytes) 000106 1745620274.321741 read 0x00dc34 thru 0x01801e (0xa3eb bytes) 000107 1745620274.328420 trunc from 0x0219bd to 0x025063 (0x36a7 bytes) 000108 1745620274.338624 trunc from 0x025064 to 0x038fa6 (0x13f43 bytes) 000109 1745620274.348796 fallocate 0x03526c thru 0x0361e8 (0xf7d bytes) 000110 1745620274.352055 read 0x034cdb thru 0x038fa6 (0x42cc bytes) 000111 1745620274.359330 write 0x00e213 thru 0x00ee6c (0xc5a bytes) 000112 1745620274.360057 punch from 0x035bea to 0x038fa6 (0x33bd bytes) 000113 1745620274.362328 trunc from 0x02aa6c to 0x038fa6 (0xe53b bytes) 000114 1745620274.364424 trunc from 0x016f01 to 0x017108 (0x208 bytes) 000115 1745620274.366464 mapwrite 0x000798 thru 0x00cfe7 (0xc850 bytes) 000116 1745620274.378113 punch from 0x011465 to 0x013e94 (0x2a30 bytes) 000117 1745620274.383969 write 0x0275de thru 0x0355e9 (0xe00c bytes) 000118 1745620274.392717 MIRROR_RESYNC 3 mirrors 000119 1745620274.777656 fallocate 0x011d23 thru 0x0143f0 (0x26ce bytes) 000120 1745620274.810796 trunc from 0x011251 to 0x01eafe (0xd8ae bytes) 000122 1745620274.814363 write 0x0134af thru 0x015e6f (0x29c1 bytes) 000123 1745620274.824077 fallocate 0x02cb67 thru 0x03b4fa (0xe994 bytes) 000124 1745620274.828320 fallocate 0x010210 thru 0x01ea71 (0xe862 bytes) 000125 1745620274.833720 mapwrite 0x022a9c thru 0x0294e4 (0x6a49 bytes) 000126 1745620274.863004 mapwrite 0x00dc29 thru 0x00f9cb (0x1da3 bytes) 000127 1745620274.880370 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000128 1745620274.995194 trunc from 0x038fa7 to 0x012cf0 (0x262b6 bytes) 000130 1745620275.028982 punch from 0x00d80e to 0x00f725 (0x1f18 bytes) 000131 1745620275.037167 read 0x01129c thru 0x012cf0 (0x1a55 bytes) 000132 1745620275.052306 punch from 0x00243d to 0x005b65 (0x3729 bytes) 000133 1745620275.068107 read 0x00f8ac thru 0x012cf0 (0x3445 bytes) 000134 1745620275.081440 MIRROR_RESYNC 2 mirrors 000135 1745620275.329688 fallocate 0x02c92d thru 0x0342be (0x7992 bytes) 000137 1745620275.355983 punch from 0x00bfb1 to 0x012cf0 (0x6d40 bytes) 000138 1745620275.359603 punch from 0x00f1d4 to 0x012cf0 (0x3b1d bytes) 000140 1745620275.363539 read 0x0034a0 thru 0x00dda6 (0xa907 bytes) 000141 1745620275.370406 mapread 0x00e5cd thru 0x012cf0 (0x4724 bytes) 000142 1745620275.386043 mapread 0x012699 thru 0x012cf0 (0x658 bytes) 000143 1745620275.387540 fallocate 0x035b9d thru 0x039797 (0x3bfb bytes) 000144 1745620275.394465 write 0x01a59a thru 0x0246cb (0xa132 bytes) 000145 1745620275.403830 read 0x01f8ef thru 0x029665 (0x9d77 bytes) 000146 1745620275.410155 mapwrite 0x0098bb thru 0x0153b8 (0xbafe bytes) 000147 1745620275.420797 write 0x02e103 thru 0x03c282 (0xe180 bytes) 000148 1745620275.422383 trunc from 0x03c283 to 0x024fac (0x172d6 bytes) 000150 1745620275.446629 trunc from 0x024fad to 0x00c0d6 (0x18ed6 bytes) 000151 1745620275.456098 read 0x008810 thru 0x00c0d6 (0x38c7 bytes) 000152 1745620275.457385 fallocate 0x02c50f thru 0x03c340 (0xfe32 bytes) 000153 1745620275.462647 mapread 0x00503b thru 0x00c0d6 (0x709c bytes) 000154 1745620275.464540 write 0x01e4e9 thru 0x026516 (0x802e bytes) 000155 1745620275.466159 read 0x0240b7 thru 0x026516 (0x2460 bytes) 000156 1745620275.466577 trunc from 0x026517 to 0x01035a (0x161bc bytes) 000158 1745620275.475862 MIRROR_RESYNC 2 mirrors 000160 1745620275.671918 trunc from 0x00552b to 0x00c6c1 (0x7197 bytes) 000161 1745620275.697838 fallocate 0x030c8b thru 0x03380a (0x2b80 bytes) 000162 1745620275.700991 mapwrite 0x03c9f1 thru 0x03ffff (0x360f bytes) 000163 1745620275.756088 mapread 0x030b23 thru 0x038aca (0x7fa8 bytes) 000164 1745620275.761328 trunc from 0x03fbe2 to 0x03ffff (0x41e bytes) 000165 1745620275.767930 MIRROR_EXTEND to 3 mirrors 000166 1745620276.074858 mapwrite 0x022254 thru 0x031c8b (0xfa38 bytes) 000167 1745620276.151870 read 0x007a24 thru 0x0151b8 (0xd795 bytes) 000168 1745620276.160498 write 0x03149f thru 0x03ffff (0xeb61 bytes) 000169 1745620276.166168 fallocate 0x021708 thru 0x025551 (0x3e4a bytes) 000170 1745620276.172849 mapwrite 0x0218c2 thru 0x02e787 (0xcec6 bytes) 000171 1745620276.189132 MIRROR_EXTEND to 4 mirrors 000172 1745620276.505003 write 0x0391d7 thru 0x03ffff (0x6e29 bytes) 000173 1745620276.522788 mapread 0x008b38 thru 0x0123e9 (0x98b2 bytes) 000174 1745620276.540201 fallocate 0x035b3d thru 0x037fba (0x247e bytes) 000175 1745620276.544067 write 0x00ddc8 thru 0x0172af (0x94e8 bytes) 000176 1745620276.545373 mapread 0x00c3b7 thru 0x00c40f (0x59 bytes) 000177 1745620276.546147 read 0x031df2 thru 0x039bfc (0x7e0b bytes) 000178 1745620276.553068 punch from 0x02d6de to 0x0305fb (0x2f1e bytes) 000179 1745620276.558258 trunc from 0x019406 to 0x025814 (0xc40f bytes) 000180 1745620276.560869 fallocate 0x01b030 thru 0x02adb5 (0xfd86 bytes) 000181 1745620276.563966 write 0x039e5d thru 0x03bc86 (0x1e2a bytes) 000182 1745620276.564553 write 0x0145d3 thru 0x01b114 (0x6b42 bytes) 000183 1745620276.570001 mapread 0x000d9c thru 0x00e927 (0xdb8c bytes) 000184 1745620276.579313 punch from 0x02f75f to 0x030168 (0xa0a bytes) 000186 1745620276.584041 fallocate 0x0226c9 thru 0x02bf64 (0x989c bytes) 000187 1745620276.587032 read 0x0177b2 thru 0x01b276 (0x3ac5 bytes) 000188 1745620276.588446 trunc from 0x02f921 to 0x034430 (0x4b10 bytes) 000189 1745620276.591793 punch from 0x038eeb to 0x03cf1f (0x4035 bytes) 000190 1745620276.600950 punch from 0x016176 to 0x02596d (0xf7f8 bytes) 000191 1745620276.607723 mapwrite 0x02f006 thru 0x032358 (0x3353 bytes) 000192 1745620276.629385 fallocate 0x021d74 thru 0x02bd51 (0x9fde bytes) 000193 1745620276.639105 MIRROR_RESYNC 4 mirrors 000194 1745620277.151477 trunc from 0x01702b to 0x01a187 (0x315d bytes) 000195 1745620277.170595 mapwrite 0x00ac30 thru 0x01662a (0xb9fb bytes) 000196 1745620277.198983 MIRROR_EXTEND to 5 mirrors 000197 1745620277.490566 punch from 0x020dc7 to 0x02847c (0x76b6 bytes) 000198 1745620277.510114 mapread 0x00f948 thru 0x01bd86 (0xc43f bytes) 000199 1745620277.527245 read 0x016577 thru 0x01de3e (0x78c8 bytes) 000200 1745620277.528015 punch from 0x01ecc1 to 0x01f0a1 (0x3e1 bytes) 000201 1745620277.535010 MIRROR_EXTEND to 6 mirrors 000202 1745620277.853106 read 0x028b7f thru 0x032857 (0x9cd9 bytes) 000203 1745620277.861297 fallocate 0x0252a1 thru 0x026f15 (0x1c75 bytes) 000205 1745620277.885206 write 0x004d66 thru 0x00d733 (0x89ce bytes) 000206 1745620277.894591 mapread 0x03b412 thru 0x03d00c (0x1bfb bytes) 000207 1745620277.898970 read 0x0125f9 thru 0x01a6a0 (0x80a8 bytes) 000208 1745620277.906500 MIRROR_RESYNC 6 mirrors 000209 1745620278.723549 mapwrite 0x028593 thru 0x02b38b (0x2df9 bytes) 000210 1745620278.783602 mapwrite 0x00e740 thru 0x0195d9 (0xae9a bytes) 000211 1745620278.807295 mapread 0x01a5fe thru 0x02577a (0xb17d bytes) 000212 1745620278.812349 write 0x01b9c3 thru 0x024a10 (0x904e bytes) 000213 1745620278.813341 read 0x0248dd thru 0x02ccb5 (0x83d9 bytes) 000215 1745620278.814048 mapread 0x0261ed thru 0x026231 (0x45 bytes) 000216 1745620278.814593 read 0x030516 thru 0x039132 (0x8c1d bytes) 000218 1745620278.820060 mapwrite 0x00bdc7 thru 0x017b8c (0xbdc6 bytes) 000219 1745620278.834946 fallocate 0x025a2e thru 0x029390 (0x3963 bytes) 000221 1745620278.842596 MIRROR_RESYNC 6 mirrors 000222 1745620279.507290 mapwrite 0x031e53 thru 0x03cfe8 (0xb196 bytes) 000223 1745620279.576073 read 0x027ccb thru 0x02d3d6 (0x570c bytes) 000224 1745620279.579394 write 0x01780e thru 0x01aedd (0x36d0 bytes) 000225 1745620279.587965 mapread 0x0001b8 thru 0x009e22 (0x9c6b bytes) 000226 1745620279.599538 mapwrite 0x028e6e thru 0x02a47b (0x160e bytes) 000228 1745620279.612109 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000229 1745620279.714197 mapwrite 0x02645e thru 0x02d096 (0x6c39 bytes) 000230 1745620279.736758 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000231 1745620279.835495 punch from 0x0232e5 to 0x02cda1 (0x9abd bytes) 000232 1745620279.838791 write 0x000f78 thru 0x00c955 (0xb9de bytes) 000233 1745620279.849789 MIRROR_EXTEND to 5 mirrors 000234 1745620280.152263 mapread 0x019526 thru 0x01a813 (0x12ee bytes) 000235 1745620280.165487 read 0x017b47 thru 0x01e1a8 (0x6662 bytes) 000236 1745620280.172756 trunc from 0x00442f to 0x00d6ed (0x92bf bytes) 000237 1745620280.192685 trunc from 0x03bc54 to 0x03c767 (0xb14 bytes) 000238 1745620280.196367 read 0x018d74 thru 0x01aa57 (0x1ce4 bytes) 000239 1745620280.205186 mapwrite 0x01f02a thru 0x02a181 (0xb158 bytes) 000240 1745620280.238458 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000241 1745620280.369959 MIRROR_EXTEND to 5 mirrors 000242 1745620280.645381 trunc from 0x006e40 to 0x011ab3 (0xac74 bytes) 000244 1745620280.665745 fallocate 0x0217c5 thru 0x02e2ac (0xcae8 bytes) 000245 1745620280.667866 mapread 0x023daf thru 0x03199f (0xdbf1 bytes) 000247 1745620280.683322 trunc from 0x02bfa2 to 0x031161 (0x51c0 bytes) 000248 1745620280.688045 mapwrite 0x03c3e5 thru 0x03ffff (0x3c1b bytes) 000249 1745620280.707488 trunc from 0x01b291 to 0x0204d4 (0x5244 bytes) 000250 1745620280.710362 mapwrite 0x036146 thru 0x03ffff (0x9eba bytes) 000251 1745620280.720946 fallocate 0x01c93e thru 0x0216f0 (0x4db3 bytes) 000252 1745620280.724917 write 0x02375f thru 0x028a09 (0x52ab bytes) 000253 1745620280.728082 write 0x0312c8 thru 0x03ffff (0xed38 bytes) 000254 1745620280.735960 MIRROR_EXTEND to 6 mirrors 000255 1745620281.005608 mapread 0x00fbfa thru 0x00ff4d (0x354 bytes) 000256 1745620281.015339 fallocate 0x01f240 thru 0x020a39 (0x17fa bytes) 000257 1745620281.032022 punch from 0x021a14 to 0x02397e (0x1f6b bytes) 000259 1745620281.036666 fallocate 0x02ddc8 thru 0x03a634 (0xc86d bytes) 000260 1745620281.041745 MIRROR_EXTEND to 7 mirrors 000261 1745620281.322153 trunc from 0x03628f to 0x03ffff (0x9d71 bytes) 000262 1745620281.343059 trunc from 0x02ce52 to 0x032258 (0x5407 bytes) 000263 1745620281.345808 mapread 0x031811 thru 0x03ed73 (0xd563 bytes) 000264 1745620281.362372 mapread 0x012b9c thru 0x01d373 (0xa7d8 bytes) 000265 1745620281.376883 mapwrite 0x0172a4 thru 0x024b7a (0xd8d7 bytes) 000266 1745620281.400590 MIRROR_RESYNC 7 mirrors 000267 1745620281.843110 read 0x01712b thru 0x0184a0 (0x1376 bytes) 000268 1745620281.850154 MIRROR_RESYNC 7 mirrors 000269 1745620281.971179 write 0x031947 thru 0x03d319 (0xb9d3 bytes) 000270 1745620281.995853 read 0x00a72d thru 0x011956 (0x722a bytes) 000271 1745620282.002453 punch from 0x03dfeb to 0x03ffff (0x2015 bytes) 000272 1745620282.011441 trunc from 0x040000 to 0x01cfe3 (0x2301c bytes) 000273 1745620282.030191 punch from 0x01cd61 to 0x01cfe3 (0x283 bytes) 000274 1745620282.033134 trunc from 0x00e542 to 0x012d15 (0x47d4 bytes) 000275 1745620282.039030 write 0x033049 thru 0x03ffff (0xcfb7 bytes) 000277 1745620282.041874 fallocate 0x00698f thru 0x013cbb (0xd32d bytes) 000278 1745620282.044734 write 0x001262 thru 0x00b7fc (0xa59b bytes) 000279 1745620282.048857 trunc from 0x040000 to 0x0084f3 (0x37b0c bytes) 000280 1745620282.058754 punch from 0x004a11 to 0x0084f3 (0x3ae3 bytes) 000281 1745620282.066713 MIRROR_EXTEND to 8 mirrors 000282 1745620282.328027 trunc from 0x004cbd to 0x0084f3 (0x3837 bytes) 000283 1745620282.351425 trunc from 0x0084f4 to 0x014571 (0xc07e bytes) 000284 1745620282.376971 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000285 1745620282.493809 trunc from 0x014572 to 0x028ccc (0x1475b bytes) 000286 1745620282.508335 trunc from 0x028ccd to 0x038589 (0xf8bd bytes) 000287 1745620282.524399 MIRROR_RESYNC 7 mirrors 000288 1745620283.094663 write 0x015ded thru 0x0192d1 (0x34e5 bytes) 000289 1745620283.117069 trunc from 0x026824 to 0x0300c2 (0x989f bytes) 000290 1745620283.120713 MIRROR_EXTEND to 8 mirrors 000291 1745620283.424529 fallocate 0x00c541 thru 0x016bab (0xa66b bytes) 000292 1745620283.468541 MIRROR_EXTEND to 9 mirrors 000294 1745620283.774202 punch from 0x031245 to 0x038589 (0x7345 bytes) 000295 1745620283.797516 mapread 0x031334 thru 0x0334d8 (0x21a5 bytes) 000296 1745620283.807354 mapread 0x019f61 thru 0x0235f3 (0x9693 bytes) 000297 1745620283.818065 trunc from 0x03858a to 0x026266 (0x12323 bytes) 000298 1745620283.834807 MIRROR_EXTEND to 10 mirrors 000299 1745620284.118615 mapread 0x01c1d1 thru 0x01e3fd (0x222d bytes) 000300 1745620284.141805 fallocate 0x01c19b thru 0x02acbb (0xeb21 bytes) 000301 1745620284.165285 mapwrite 0x003233 thru 0x010290 (0xd05e bytes) 000302 1745620284.188641 trunc from 0x017c2b to 0x024ea4 (0xd27a bytes) 000303 1745620284.195871 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000304 1745620284.336868 read 0x02571f thru 0x026266 (0xb48 bytes) 000305 1745620284.339770 punch from 0x00ae98 to 0x013789 (0x88f2 bytes) 000306 1745620284.343305 fallocate 0x028728 thru 0x032480 (0x9d59 bytes) 000307 1745620284.348909 read 0x023449 thru 0x026266 (0x2e1e bytes) 000308 1745620284.355767 mapwrite 0x015f9f thru 0x01eba7 (0x8c09 bytes) 000309 1745620284.364765 read 0x00db43 thru 0x01531b (0x77d9 bytes) 000310 1745620284.365525 read 0x01ff37 thru 0x02511f (0x51e9 bytes) 000311 1745620284.366306 punch from 0x01efe6 to 0x0205be (0x15d9 bytes) 000313 1745620284.371760 fallocate 0x008e25 thru 0x016012 (0xd1ee bytes) 000314 1745620284.375640 write 0x031d66 thru 0x0347ce (0x2a69 bytes) 000315 1745620284.383248 MIRROR_EXTEND to 10 mirrors 000316 1745620284.713446 punch from 0x01b4ce to 0x023dd6 (0x8909 bytes) 000318 1745620284.739516 mapread 0x00223e thru 0x006559 (0x431c bytes) 000319 1745620284.756031 punch from 0x00cb4f to 0x01a8a2 (0xdd54 bytes) 000321 1745620284.761715 trunc from 0x0347cf to 0x00bf88 (0x28846 bytes) 000323 1745620284.807335 MIRROR_EXTEND to 11 mirrors 000324 1745620284.998400 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000325 1745620285.101517 trunc from 0x002b55 to 0x00bf88 (0x9434 bytes) 000326 1745620285.121851 punch from 0x00703f to 0x00bf88 (0x4f4a bytes) 000328 1745620285.124780 trunc from 0x00bf89 to 0x0020a5 (0x9ee3 bytes) 000329 1745620285.151254 mapwrite 0x025736 thru 0x0338d7 (0xe1a2 bytes) 000330 1745620285.186047 punch from 0x0222bf to 0x02ea50 (0xc792 bytes) 000331 1745620285.189457 read 0x032615 thru 0x0338d7 (0x12c3 bytes) 000332 1745620285.189902 trunc from 0x00affc to 0x01911e (0xe123 bytes) 000333 1745620285.193858 mapwrite 0x001099 thru 0x0042b1 (0x3219 bytes) 000334 1745620285.208101 mapwrite 0x0370e0 thru 0x03ffff (0x8f20 bytes) 000335 1745620285.237915 mapread 0x036d4a thru 0x03cf69 (0x6220 bytes) 000336 1745620285.244557 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000337 1745620285.339076 trunc from 0x040000 to 0x005dfb (0x3a204 bytes) 000338 1745620285.347385 fallocate 0x02d195 thru 0x032cc5 (0x5b31 bytes) 000339 1745620285.350313 fallocate 0x004cda thru 0x0102e4 (0xb60b bytes) 000340 1745620285.353273 fallocate 0x03a92c thru 0x03ffff (0x56d4 bytes) 000341 1745620285.355967 trunc from 0x040000 to 0x03e6a2 (0x195d bytes) 000342 1745620285.362793 punch from 0x01e6bc to 0x01ec9a (0x5df bytes) 000343 1745620285.366506 read 0x013bee thru 0x01423c (0x64f bytes) 000344 1745620285.370125 mapwrite 0x02505f thru 0x02a383 (0x5325 bytes) 000345 1745620285.391437 MIRROR_RESYNC 9 mirrors 000346 1745620286.095420 mapwrite 0x02f5b6 thru 0x03adb7 (0xb802 bytes) 000347 1745620286.162384 trunc from 0x0332b4 to 0x03e6a2 (0xb3ef bytes) 000348 1745620286.169430 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000349 1745620286.287259 write 0x02f702 thru 0x0318a6 (0x21a5 bytes) 000350 1745620286.294529 mapread 0x029f0f thru 0x02eebe (0x4fb0 bytes) 000351 1745620286.299722 mapwrite 0x03f3cc thru 0x03f54d (0x182 bytes) 000352 1745620286.335941 MIRROR_EXTEND to 9 mirrors 000353 1745620286.634398 mapread 0x0125fb thru 0x01f3ac (0xcdb2 bytes) 000354 1745620286.672376 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000355 1745620286.832206 write 0x02ff81 thru 0x03aa91 (0xab11 bytes) 000356 1745620286.858508 mapwrite 0x0001f7 thru 0x001243 (0x104d bytes) 000357 1745620286.875948 read 0x0373fc thru 0x03f54d (0x8152 bytes) 000358 1745620286.881682 mapread 0x022555 thru 0x0247ee (0x229a bytes) 000359 1745620286.885197 fallocate 0x02dad5 thru 0x035755 (0x7c81 bytes) 000360 1745620286.887909 write 0x002051 thru 0x007514 (0x54c4 bytes) 000361 1745620286.888592 trunc from 0x03f54e to 0x02b73b (0x13e12 bytes) 000362 1745620286.913113 trunc from 0x020366 to 0x025266 (0x4f01 bytes) 000363 1745620286.916063 punch from 0x00434c to 0x00d350 (0x9005 bytes) 000364 1745620286.931410 MIRROR_EXTEND to 9 mirrors 000365 1745620287.240324 mapread 0x002599 thru 0x00b4a8 (0x8f10 bytes) 000366 1745620287.257544 punch from 0x026ea0 to 0x02b73b (0x489c bytes) 000367 1745620287.283499 fallocate 0x003d99 thru 0x00aab8 (0x6d20 bytes) 000368 1745620287.286425 fallocate 0x00e1d9 thru 0x01370f (0x5537 bytes) 000369 1745620287.289382 read 0x021212 thru 0x02a8db (0x96ca bytes) 000370 1745620287.297958 trunc from 0x01ea32 to 0x0269cc (0x7f9b bytes) 000371 1745620287.305055 trunc from 0x02b73c to 0x0254b6 (0x6285 bytes) 000372 1745620287.316858 mapread 0x00dfb5 thru 0x01b450 (0xd49c bytes) 000373 1745620287.325134 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000375 1745620287.422524 read 0x00a978 thru 0x017286 (0xc90f bytes) 000376 1745620287.428032 punch from 0x00bfe4 to 0x01b744 (0xf761 bytes) 000377 1745620287.433307 MIRROR_RESYNC 8 mirrors 000380 1745620288.094069 read 0x0131ad thru 0x01e874 (0xb6c8 bytes) 000381 1745620288.100375 punch from 0x025288 to 0x0254b6 (0x22f bytes) 000382 1745620288.117303 trunc from 0x021d9b to 0x0254b6 (0x371c bytes) 000383 1745620288.120510 trunc from 0x0254b7 to 0x03ff10 (0x1aa5a bytes) 000384 1745620288.136073 read 0x0304ed thru 0x0352a4 (0x4db8 bytes) 000385 1745620288.148447 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000386 1745620288.240876 MIRROR_EXTEND to 8 mirrors 000387 1745620288.479971 fallocate 0x03e2fc thru 0x03ffff (0x1d04 bytes) 000389 1745620288.497190 trunc from 0x040000 to 0x003118 (0x3cee7 bytes) 000390 1745620288.523428 mapread 0x0010c1 thru 0x003118 (0x2058 bytes) 000391 1745620288.533941 mapread 0x000651 thru 0x002a64 (0x2414 bytes) 000392 1745620288.543586 read 0x000810 thru 0x003118 (0x2909 bytes) 000393 1745620288.546893 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000394 1745620288.626402 trunc from 0x000062 to 0x003118 (0x30b7 bytes) 000395 1745620288.628906 write 0x0160fd thru 0x01933b (0x323f bytes) 000396 1745620288.632726 punch from 0x00d1c1 to 0x013098 (0x5ed8 bytes) 000397 1745620288.634845 punch from 0x003f92 to 0x011f21 (0xdf90 bytes) 000398 1745620288.637467 mapwrite 0x02f700 thru 0x0393df (0x9ce0 bytes) 000399 1745620288.667055 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000400 1745620288.759167 trunc from 0x0393e0 to 0x028cb8 (0x10727 bytes) 000401 1745620288.767426 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000402 1745620288.849044 MIRROR_EXTEND to 6 mirrors 000403 1745620289.073062 MIRROR_EXTEND to 7 mirrors 000404 1745620289.271150 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000405 1745620289.384424 punch from 0x011e93 to 0x0132ee (0x145c bytes) 000406 1745620289.402431 trunc from 0x02787e to 0x028cb8 (0x143b bytes) 000407 1745620289.404813 trunc from 0x028cb9 to 0x01c366 (0xc952 bytes) 000408 1745620289.424833 trunc from 0x01c367 to 0x03365b (0x172f5 bytes) 000409 1745620289.432034 write 0x015bf8 thru 0x023f29 (0xe332 bytes) 000410 1745620289.440206 MIRROR_RESYNC 6 mirrors 000411 1745620289.849796 mapread 0x011272 thru 0x014893 (0x3622 bytes) 000412 1745620289.858754 mapread 0x01bed0 thru 0x026072 (0xa1a3 bytes) 000413 1745620289.864481 trunc from 0x00c8dc to 0x019a9e (0xd1c3 bytes) 000414 1745620289.883902 MIRROR_RESYNC 6 mirrors 000415 1745620290.411567 mapread 0x01a046 thru 0x0206d6 (0x6691 bytes) 000416 1745620290.419593 mapwrite 0x0077a4 thru 0x0084be (0xd1b bytes) 000417 1745620290.476923 fallocate 0x0245c3 thru 0x030366 (0xbda4 bytes) 000418 1745620290.480495 read 0x02a23f thru 0x03365b (0x941d bytes) 000419 1745620290.485758 trunc from 0x0186f6 to 0x01eda3 (0x66ae bytes) 000420 1745620290.490588 MIRROR_EXTEND to 7 mirrors 000421 1745620290.712544 MIRROR_EXTEND to 8 mirrors 000422 1745620290.940367 mapwrite 0x01f5ba thru 0x02bc1c (0xc663 bytes) 000423 1745620291.007575 MIRROR_EXTEND to 9 mirrors 000424 1745620291.296362 punch from 0x0099a5 to 0x00e953 (0x4faf bytes) 000425 1745620291.310650 write 0x039d8a thru 0x03be35 (0x20ac bytes) 000426 1745620291.315019 MIRROR_EXTEND to 10 mirrors 000427 1745620291.626309 trunc from 0x02b5cb to 0x037327 (0xbd5d bytes) 000428 1745620291.644551 mapread 0x0023f7 thru 0x00ec58 (0xc862 bytes) 000430 1745620291.660970 punch from 0x030160 to 0x0327dd (0x267e bytes) 000431 1745620291.664673 fallocate 0x02afdf thru 0x033439 (0x845b bytes) 000432 1745620291.668182 write 0x002755 thru 0x00ecc3 (0xc56f bytes) 000433 1745620291.673513 write 0x01f959 thru 0x020bfc (0x12a4 bytes) 000434 1745620291.680102 punch from 0x01216f to 0x01d0c9 (0xaf5b bytes) 000435 1745620291.682988 trunc from 0x03be36 to 0x02e044 (0xddf1 bytes) 000436 1745620291.697903 punch from 0x02834f to 0x02de5c (0x5b0e bytes) 000437 1745620291.700861 write 0x00a1a2 thru 0x01642e (0xc28d bytes) 000438 1745620291.706035 mapread 0x016f72 thru 0x020eb9 (0x9f48 bytes) 000439 1745620291.710062 read 0x009c2f thru 0x0157a3 (0xbb75 bytes) 000440 1745620291.712227 trunc from 0x02e045 to 0x03764f (0x960b bytes) 000441 1745620291.721900 MIRROR_RESYNC 10 mirrors 000442 1745620292.758115 fallocate 0x003bc8 thru 0x006d42 (0x317b bytes) 000443 1745620292.781110 fallocate 0x0319de thru 0x03ea62 (0xd085 bytes) 000444 1745620292.784799 write 0x008d89 thru 0x01494f (0xbbc7 bytes) 000445 1745620292.790470 trunc from 0x03ea63 to 0x02402d (0x1aa35 bytes) 000446 1745620292.801718 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000447 1745620292.907243 write 0x013bc8 thru 0x0208c0 (0xccf9 bytes) 000448 1745620292.917703 MIRROR_EXTEND to 10 mirrors 000449 1745620293.141159 fallocate 0x003147 thru 0x0110f4 (0xdfae bytes) 000450 1745620293.161530 trunc from 0x01bbc0 to 0x02402d (0x846e bytes) 000451 1745620293.164472 trunc from 0x003f0d to 0x0046ae (0x7a2 bytes) 000452 1745620293.167468 mapread 0x01cdcf thru 0x02402d (0x725f bytes) 000453 1745620293.181035 trunc from 0x01c2f9 to 0x02402d (0x7d35 bytes) 000455 1745620293.184670 mapwrite 0x0371b2 thru 0x03ffff (0x8e4e bytes) 000456 1745620293.211184 read 0x01788e thru 0x01d702 (0x5e75 bytes) 000457 1745620293.219310 trunc from 0x03de6f to 0x03ffff (0x2191 bytes) 000458 1745620293.222167 punch from 0x017426 to 0x01a791 (0x336c bytes) 000459 1745620293.225115 fallocate 0x02adf5 thru 0x031d7b (0x6f87 bytes) 000461 1745620293.230894 MIRROR_RESYNC 10 mirrors 000462 1745620294.163459 trunc from 0x040000 to 0x036158 (0x9ea7 bytes) 000463 1745620294.186671 fallocate 0x00d8bb thru 0x01895b (0xb0a1 bytes) 000464 1745620294.193024 trunc from 0x036159 to 0x012555 (0x23c03 bytes) 000465 1745620294.202500 trunc from 0x012556 to 0x03ffdf (0x2da8a bytes) 000466 1745620294.209572 mapwrite 0x0051ec thru 0x008472 (0x3287 bytes) 000467 1745620294.224268 read 0x021ff8 thru 0x02a9fb (0x8a04 bytes) 000468 1745620294.228392 trunc from 0x0262a1 to 0x033514 (0xd274 bytes) 000469 1745620294.230757 mapread 0x038e16 thru 0x03f9b1 (0x6b9c bytes) 000470 1745620294.235100 read 0x029928 thru 0x035fd6 (0xc6af bytes) 000471 1745620294.239916 write 0x03f1ce thru 0x03ffff (0xe32 bytes) 000472 1745620294.240567 punch from 0x0320dc to 0x0377e7 (0x570c bytes) 000473 1745620294.242647 punch from 0x034c0b to 0x03f3cd (0xa7c3 bytes) 000474 1745620294.247902 trunc from 0x00c2c2 to 0x00dd13 (0x1a52 bytes) 000475 1745620294.249764 write 0x001e35 thru 0x00c8dd (0xaaa9 bytes) 000476 1745620294.254045 fallocate 0x03d8cb thru 0x03ea8f (0x11c5 bytes) 000477 1745620294.258614 trunc from 0x012c16 to 0x022989 (0xfd74 bytes) 000478 1745620294.261351 write 0x01a650 thru 0x02a625 (0xffd6 bytes) 000479 1745620294.264969 punch from 0x02960d to 0x02ba0d (0x2401 bytes) 000480 1745620294.275416 punch from 0x025c97 to 0x026073 (0x3dd bytes) 000481 1745620294.279799 trunc from 0x036052 to 0x03ffff (0x9fae bytes) 000482 1745620294.286900 trunc from 0x040000 to 0x038ad9 (0x7526 bytes) 000483 1745620294.295108 trunc from 0x038ada to 0x01269d (0x2643c bytes) 000484 1745620294.305438 mapwrite 0x021700 thru 0x02c76e (0xb06f bytes) 000485 1745620294.351175 MIRROR_EXTEND to 11 mirrors 000486 1745620294.548061 mapwrite 0x012667 thru 0x015010 (0x29aa bytes) 000488 1745620294.598921 fallocate 0x00c5be thru 0x01713d (0xab80 bytes) 000489 1745620294.602918 punch from 0x0093c3 to 0x00c521 (0x315f bytes) 000490 1745620294.606378 mapwrite 0x02fa9f thru 0x03d978 (0xdeda bytes) 000491 1745620294.647694 mapwrite 0x024f75 thru 0x02710c (0x2198 bytes) 000492 1745620294.657562 trunc from 0x03d979 to 0x003735 (0x3a243 bytes) 000493 1745620294.663283 read 0x0001d2 thru 0x001d19 (0x1b48 bytes) 000494 1745620294.665761 fallocate 0x00ac04 thru 0x0120d2 (0x74cf bytes) 000495 1745620294.669291 MIRROR_RESYNC 11 mirrors 000496 1745620295.000941 read 0x00033b thru 0x003735 (0x33fb bytes) 000497 1745620295.004024 punch from 0x00289e to 0x003735 (0xe98 bytes) 000498 1745620295.020950 read 0x0002ec thru 0x003735 (0x344a bytes) 000499 1745620295.023106 fallocate 0x002143 thru 0x006cf0 (0x4bae bytes) 000500 1745620295.025826 punch from 0x002318 to 0x003735 (0x141e bytes) 000501 1745620295.028163 mapread 0x00009c thru 0x003735 (0x369a bytes) 000502 1745620295.034525 mapread 0x000037 thru 0x003735 (0x36ff bytes) 000503 1745620295.035079 fallocate 0x01f4a3 thru 0x02cfc2 (0xdb20 bytes) 000504 1745620295.037677 trunc from 0x003736 to 0x017748 (0x14013 bytes) 000506 1745620295.056964 write 0x03b9d4 thru 0x03ffff (0x462c bytes) 000507 1745620295.057843 mapread 0x013130 thru 0x01efca (0xbe9b bytes) 000508 1745620295.063520 trunc from 0x027e21 to 0x037792 (0xf972 bytes) 000509 1745620295.076019 MIRROR_RESYNC 11 mirrors 000510 1745620296.002305 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000511 1745620296.182507 write 0x03fc1d thru 0x03ffff (0x3e3 bytes) 000512 1745620296.205029 write 0x012519 thru 0x01695f (0x4447 bytes) 000514 1745620296.217124 MIRROR_RESYNC 10 mirrors 000515 1745620297.307646 fallocate 0x039583 thru 0x03ffff (0x6a7d bytes) 000516 1745620297.323620 mapread 0x0195b1 thru 0x01c4d0 (0x2f20 bytes) 000517 1745620297.331284 mapread 0x00dcc5 thru 0x013716 (0x5a52 bytes) 000518 1745620297.336546 trunc from 0x040000 to 0x0018ff (0x3e700 bytes) 000519 1745620297.345388 mapwrite 0x03feab thru 0x03ffff (0x155 bytes) 000520 1745620297.372466 mapread 0x0391a5 thru 0x03dfb4 (0x4e10 bytes) 000521 1745620297.373107 trunc from 0x040000 to 0x021d85 (0x1e27a bytes) 000522 1745620297.380627 trunc from 0x01fbb7 to 0x021d85 (0x21cf bytes) 000523 1745620297.382821 mapwrite 0x033569 thru 0x03ffff (0xca97 bytes) 000524 1745620297.408974 mapwrite 0x032f14 thru 0x038e6b (0x5f58 bytes) 000525 1745620297.424383 trunc from 0x040000 to 0x002d56 (0x3d2a9 bytes) 000526 1745620297.431352 mapread 0x001d95 thru 0x002d56 (0xfc2 bytes) 000527 1745620297.436250 trunc from 0x002d57 to 0x01c24c (0x194f6 bytes) 000528 1745620297.444028 write 0x02985b thru 0x0387ea (0xef90 bytes) 000529 1745620297.449888 MIRROR_EXTEND to 11 mirrors 000530 1745620297.678395 write 0x02c55c thru 0x034da5 (0x884a bytes) 000532 1745620297.714912 trunc from 0x0387eb to 0x010dcf (0x27a1b bytes) 000533 1745620297.729997 fallocate 0x037bd2 thru 0x03ffff (0x842e bytes) 000534 1745620297.732695 fallocate 0x006a69 thru 0x0160e9 (0xf681 bytes) 000535 1745620297.734783 mapwrite 0x00319a thru 0x00b1b5 (0x801c bytes) 000536 1745620297.749130 mapread 0x00b18d thru 0x00b495 (0x309 bytes) 000537 1745620297.750721 read 0x03ae6b thru 0x03ffff (0x5195 bytes) 000538 1745620297.756366 MIRROR_EXTEND to 12 mirrors 000539 1745620298.012792 trunc from 0x040000 to 0x0138c6 (0x2c739 bytes) 000540 1745620298.045031 mapread 0x010b17 thru 0x0138c6 (0x2db0 bytes) 000541 1745620298.053972 mapread 0x0035ea thru 0x0130ed (0xfb04 bytes) 000542 1745620298.058124 mapwrite 0x01846a thru 0x020eeb (0x8a82 bytes) 000543 1745620298.087515 trunc from 0x020eec to 0x023cab (0x2dc0 bytes) 000544 1745620298.093187 fallocate 0x01157b thru 0x01c1df (0xac65 bytes) 000545 1745620298.096298 trunc from 0x00b3ff to 0x017639 (0xc23b bytes) 000546 1745620298.099096 write 0x009778 thru 0x015a22 (0xc2ab bytes) 000547 1745620298.102408 read 0x0191f5 thru 0x01db78 (0x4984 bytes) 000548 1745620298.106452 MIRROR_RESYNC 12 mirrors 000549 1745620298.905301 fallocate 0x03407f thru 0x0378b4 (0x3836 bytes) 000550 1745620298.932434 MIRROR_EXTEND to 13 mirrors 000551 1745620299.214662 trunc from 0x0378b5 to 0x02e6a6 (0x920e bytes) 000552 1745620299.246408 write 0x01512b thru 0x01cd11 (0x7be7 bytes) 000553 1745620299.254451 write 0x0370ed thru 0x03ffff (0x8f13 bytes) 000554 1745620299.255463 punch from 0x02242d to 0x031d35 (0xf909 bytes) 000555 1745620299.259392 punch from 0x00899d to 0x017465 (0xeac9 bytes) 000556 1745620299.265956 mapread 0x00d950 thru 0x019cef (0xc3a0 bytes) 000557 1745620299.277134 trunc from 0x040000 to 0x03025f (0xfda0 bytes) 000558 1745620299.288710 trunc from 0x014ae9 to 0x01fdcc (0xb2e4 bytes) 000559 1745620299.291507 write 0x01dfb4 thru 0x02cd91 (0xedde bytes) 000560 1745620299.299549 trunc from 0x030260 to 0x03c745 (0xc4e6 bytes) 000561 1745620299.306098 read 0x0143b1 thru 0x01b0c3 (0x6d13 bytes) 000562 1745620299.308456 write 0x02ffdd thru 0x03ad97 (0xadbb bytes) 000563 1745620299.315225 write 0x006bd8 thru 0x00a78c (0x3bb5 bytes) 000564 1745620299.323092 read 0x00abfa thru 0x013081 (0x8488 bytes) 000565 1745620299.326301 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000566 1745620299.434077 fallocate 0x02172e thru 0x02cdf5 (0xb6c8 bytes) 000567 1745620299.443288 read 0x022d01 thru 0x023f1a (0x121a bytes) 000568 1745620299.450071 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000569 1745620299.550949 mapwrite 0x00c715 thru 0x019731 (0xd01d bytes) 000570 1745620299.584092 write 0x0269cc thru 0x02c9d0 (0x6005 bytes) 000571 1745620299.590581 read 0x039c36 thru 0x03c402 (0x27cd bytes) 000572 1745620299.594948 MIRROR_RESYNC 11 mirrors 000573 1745620300.685148 mapwrite 0x02c566 thru 0x036e12 (0xa8ad bytes) 000574 1745620300.734496 MIRROR_EXTEND to 12 mirrors 000575 1745620300.973331 mapwrite 0x00bcd9 thru 0x0119f2 (0x5d1a bytes) 000576 1745620301.030992 trunc from 0x0108f8 to 0x010e3c (0x545 bytes) 000578 1745620301.033824 trunc from 0x0234a8 to 0x030bde (0xd737 bytes) 000579 1745620301.038927 trunc from 0x03c746 to 0x01d83e (0x1ef07 bytes) 000580 1745620301.053741 read 0x007cfc thru 0x014090 (0xc395 bytes) 000581 1745620301.057115 fallocate 0x01bfa7 thru 0x01ffba (0x4014 bytes) 000582 1745620301.060092 mapwrite 0x03632b thru 0x03a45b (0x4131 bytes) 000584 1745620301.089272 fallocate 0x001bfc thru 0x00c2a3 (0xa6a8 bytes) 000585 1745620301.095314 write 0x00b4fc thru 0x017451 (0xbf56 bytes) 000586 1745620301.098883 mapread 0x0108ac thru 0x01d900 (0xd055 bytes) 000587 1745620301.107395 write 0x002978 thru 0x0066d8 (0x3d61 bytes) 000588 1745620301.110813 write 0x00b351 thru 0x0174ea (0xc19a bytes) 000589 1745620301.111780 write 0x01f0d1 thru 0x027817 (0x8747 bytes) 000590 1745620301.114976 mapread 0x035741 thru 0x03a45b (0x4d1b bytes) 000591 1745620301.120100 MIRROR_EXTEND to 13 mirrors 000592 1745620301.406075 mapwrite 0x003be6 thru 0x004384 (0x79f bytes) 000593 1745620301.482913 punch from 0x008488 to 0x01757d (0xf0f6 bytes) 000595 1745620301.489985 MIRROR_RESYNC 13 mirrors 000596 1745620302.694659 trunc from 0x017aaa to 0x02509e (0xd5f5 bytes) 000597 1745620302.720693 MIRROR_RESYNC 13 mirrors 000598 1745620303.905326 punch from 0x01fc8b to 0x02e9a6 (0xed1c bytes) 000599 1745620303.924658 mapread 0x01fc22 thru 0x023a32 (0x3e11 bytes) 000600 1745620303.930873 fallocate 0x0392d6 thru 0x03ffff (0x6d2a bytes) 000601 1745620303.933276 write 0x02cb32 thru 0x0313fa (0x48c9 bytes) 000603 1745620303.939669 fallocate 0x002069 thru 0x010f41 (0xeed9 bytes) 000604 1745620303.943063 trunc from 0x01e307 to 0x029565 (0xb25f bytes) 000605 1745620303.950275 mapread 0x019f23 thru 0x029b9f (0xfc7d bytes) 000606 1745620303.967308 write 0x00409d thru 0x00aed2 (0x6e36 bytes) 000607 1745620303.973857 mapread 0x014139 thru 0x0153ff (0x12c7 bytes) 000608 1745620303.974477 read 0x02fe2c thru 0x0339d2 (0x3ba7 bytes) 000609 1745620303.977306 MIRROR_EXTEND to 14 mirrors 000610 1745620304.273203 trunc from 0x01f90c to 0x02e700 (0xedf5 bytes) 000611 1745620304.289766 read 0x00e790 thru 0x019a5d (0xb2ce bytes) 000612 1745620304.293494 write 0x00930b thru 0x009c8e (0x984 bytes) 000613 1745620304.303911 MIRROR_RESYNC 14 mirrors 000614 1745620305.662781 trunc from 0x040000 to 0x02bd8a (0x14275 bytes) 000615 1745620305.689745 MIRROR_EXTEND to 15 mirrors 000616 1745620305.932850 mapread 0x005c2b thru 0x005d91 (0x167 bytes) 000617 1745620305.941168 punch from 0x0255fc to 0x029be8 (0x45ed bytes) 000618 1745620305.961714 fallocate 0x01f705 thru 0x02462a (0x4f26 bytes) 000619 1745620305.966113 trunc from 0x02043f to 0x0213de (0xfa0 bytes) 000620 1745620305.968371 fallocate 0x01b7cd thru 0x01fe16 (0x464a bytes) 000621 1745620305.972278 trunc from 0x02bd8b to 0x015e77 (0x15f13 bytes) 000622 1745620305.983268 fallocate 0x03117a thru 0x039bfb (0x8a82 bytes) 000623 1745620305.986052 mapread 0x015902 thru 0x015e77 (0x576 bytes) 000624 1745620305.989297 read 0x00bd07 thru 0x00db98 (0x1e92 bytes) 000625 1745620305.991655 punch from 0x007666 to 0x00ad50 (0x36eb bytes) 000627 1745620305.995145 trunc from 0x015e78 to 0x02c148 (0x162d1 bytes) 000628 1745620306.002861 trunc from 0x00ac36 to 0x00d973 (0x2d3e bytes) 000629 1745620306.005118 trunc from 0x025829 to 0x02b3ed (0x5bc5 bytes) 000630 1745620306.009764 fallocate 0x01b610 thru 0x02a9de (0xf3cf bytes) 000631 1745620306.012745 fallocate 0x005431 thru 0x00c58d (0x715d bytes) 000632 1745620306.015289 trunc from 0x02c149 to 0x006436 (0x25d12 bytes) 000633 1745620306.025009 fallocate 0x00052e thru 0x0070b0 (0x6b83 bytes) 000634 1745620306.029684 mapwrite 0x027d7e thru 0x036a46 (0xecc9 bytes) 000635 1745620306.056967 punch from 0x012e55 to 0x01b6bf (0x886b bytes) 000636 1745620306.059047 punch from 0x00020e to 0x004fb6 (0x4da9 bytes) 000637 1745620306.061362 mapread 0x011510 thru 0x01319a (0x1c8b bytes) 000638 1745620306.065457 mapwrite 0x01a15a thru 0x01e7c1 (0x4668 bytes) 000639 1745620306.079862 read 0x00efcd thru 0x013ac2 (0x4af6 bytes) 000640 1745620306.080309 punch from 0x00d533 to 0x01625b (0x8d29 bytes) 000641 1745620306.082685 read 0x0080cf thru 0x01629b (0xe1cd bytes) 000642 1745620306.088254 mapread 0x006aab thru 0x009555 (0x2aab bytes) 000643 1745620306.088993 write 0x0253ce thru 0x031bc7 (0xc7fa bytes) 000644 1745620306.090074 trunc from 0x036a47 to 0x02a705 (0xc341 bytes) 000645 1745620306.098338 read 0x018438 thru 0x0186c7 (0x290 bytes) 000648 1745620306.099163 read 0x0032e2 thru 0x008202 (0x4f21 bytes) 000649 1745620306.099599 fallocate 0x02963a thru 0x029a08 (0x3cf bytes) 000650 1745620306.104670 MIRROR_EXTEND to 16 mirrors 000651 1745620306.338148 punch from 0x013941 to 0x015166 (0x1826 bytes) 000652 1745620306.364523 trunc from 0x02a706 to 0x03525e (0xab59 bytes) 000653 1745620306.387168 trunc from 0x03525f to 0x00a94a (0x2a914 bytes) 000654 1745620306.399169 mapread 0x007a54 thru 0x00a94a (0x2ef7 bytes) 000655 1745620306.413506 mapwrite 0x03eca2 thru 0x03ffff (0x135e bytes) 000656 1745620306.444872 fallocate 0x02cdc5 thru 0x03c475 (0xf6b1 bytes) 000657 1745620306.448012 punch from 0x012b38 to 0x018e2b (0x62f4 bytes) 000658 1745620306.450592 read 0x03dcde thru 0x03ffff (0x2322 bytes) 000659 1745620306.454255 MIRROR_RESYNC 16 mirrors 000660 1745620307.725913 trunc from 0x0158aa to 0x01b00b (0x5762 bytes) 000662 1745620307.764350 mapwrite 0x0304a9 thru 0x032c9f (0x27f7 bytes) 000663 1745620307.790693 mapread 0x00c3a4 thru 0x01493e (0x859b bytes) 000664 1745620307.802801 trunc from 0x040000 to 0x02c2fe (0x13d01 bytes) 000665 1745620307.818092 mapwrite 0x01254b thru 0x0165a5 (0x405b bytes) 000666 1745620307.837815 MIRROR_RESYNC 16 mirrors 000667 1745620309.149719 MIRROR_RESYNC 16 mirrors 000668 1745620309.295526 trunc from 0x02c2ff to 0x03dd55 (0x11a57 bytes) 000669 1745620309.323108 punch from 0x00998d to 0x00b452 (0x1ac6 bytes) 000670 1745620309.326651 trunc from 0x03dd56 to 0x01b8ea (0x2246b bytes) 000672 1745620309.343416 write 0x02ae60 thru 0x034512 (0x96b3 bytes) 000673 1745620309.344986 punch from 0x00d5b1 to 0x010501 (0x2f51 bytes) 000674 1745620309.350452 mapwrite 0x032255 thru 0x036d0a (0x4ab6 bytes) 000675 1745620309.373005 write 0x01b9f5 thru 0x022930 (0x6f3c bytes) 000676 1745620309.380634 punch from 0x0329c2 to 0x036d0a (0x4349 bytes) 000677 1745620309.383020 mapread 0x024ac9 thru 0x02e5a1 (0x9ad9 bytes) 000678 1745620309.389389 trunc from 0x001404 to 0x00fb78 (0xe775 bytes) 000680 1745620309.396966 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000682 1745620309.504632 fallocate 0x02c5ab thru 0x032ff3 (0x6a49 bytes) 000683 1745620309.508019 punch from 0x007045 to 0x011553 (0xa50f bytes) 000685 1745620309.512478 write 0x010485 thru 0x01607e (0x5bfa bytes) 000686 1745620309.522903 trunc from 0x0198de to 0x01aee9 (0x160c bytes) 000687 1745620309.527304 fallocate 0x03d72b thru 0x03ffff (0x28d5 bytes) 000688 1745620309.530027 mapwrite 0x02a0c6 thru 0x036b47 (0xca82 bytes) 000689 1745620309.552431 mapwrite 0x01ae1d thru 0x02ad55 (0xff39 bytes) 000690 1745620309.577239 mapread 0x016048 thru 0x01c2cc (0x6285 bytes) 000691 1745620309.577919 trunc from 0x00b153 to 0x016059 (0xaf07 bytes) 000692 1745620309.580756 read 0x02cb29 thru 0x036d0a (0xa1e2 bytes) 000693 1745620309.581229 trunc from 0x001387 to 0x006d85 (0x59ff bytes) 000694 1745620309.583540 read 0x03569f thru 0x036d0a (0x166c bytes) 000695 1745620309.583805 trunc from 0x036d0b to 0x00b91b (0x2b3ef bytes) 000696 1745620309.590725 fallocate 0x00ea0b thru 0x01264d (0x3c43 bytes) 000697 1745620309.597703 MIRROR_RESYNC 15 mirrors 000698 1745620310.881624 write 0x025027 thru 0x029ede (0x4eb8 bytes) 000699 1745620310.913547 MIRROR_EXTEND to 16 mirrors 000700 1745620311.188238 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000701 1745620311.339919 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000703 1745620311.509648 read 0x021bce thru 0x029ede (0x8311 bytes) 000704 1745620311.523689 fallocate 0x0071b6 thru 0x0091f6 (0x2041 bytes) 000705 1745620311.550430 write 0x03babd thru 0x03e5c1 (0x2b05 bytes) 000706 1745620311.555466 MIRROR_EXTEND to 15 mirrors 000707 1745620311.855605 fallocate 0x03faee thru 0x03ffff (0x512 bytes) 000708 1745620311.876069 fallocate 0x009684 thru 0x00ff4d (0x68ca bytes) 000709 1745620311.878270 mapwrite 0x01e392 thru 0x02569b (0x730a bytes) 000710 1745620311.903955 MIRROR_EXTEND to 16 mirrors 000711 1745620312.158516 mapwrite 0x013f17 thru 0x01af4e (0x7038 bytes) 000713 1745620312.220446 mapwrite 0x03a152 thru 0x03b9ef (0x189e bytes) 000714 1745620312.231923 trunc from 0x03e5c2 to 0x018b28 (0x25a99 bytes) 000715 1745620312.243090 trunc from 0x018b29 to 0x030997 (0x17e6f bytes) 000716 1745620312.256048 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000717 1745620312.370360 mapwrite 0x0191b4 thru 0x01ba64 (0x28b1 bytes) 000718 1745620312.397107 read 0x022cb6 thru 0x02f59b (0xc8e6 bytes) 000719 1745620312.403599 punch from 0x02a5a4 to 0x030997 (0x63f4 bytes) 000720 1745620312.406711 trunc from 0x0238ca to 0x030997 (0xd0ce bytes) 000722 1745620312.413503 MIRROR_EXTEND to 16 mirrors 000723 1745620312.657493 read 0x0240e5 thru 0x02d6ef (0x960b bytes) 000725 1745620312.670249 fallocate 0x0219de thru 0x0256dc (0x3cff bytes) 000727 1745620312.691126 mapread 0x02af30 thru 0x03074f (0x5820 bytes) 000728 1745620312.703249 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000729 1745620312.810324 mapread 0x014786 thru 0x0188a0 (0x411b bytes) 000730 1745620312.823966 read 0x01f292 thru 0x02f101 (0xfe70 bytes) 000732 1745620312.830274 fallocate 0x0267a9 thru 0x02d46b (0x6cc3 bytes) 000733 1745620312.833303 trunc from 0x02f7b7 to 0x030997 (0x11e1 bytes) 000734 1745620312.836853 fallocate 0x037865 thru 0x03efd4 (0x7770 bytes) 000735 1745620312.843900 mapread 0x022a5a thru 0x023ecc (0x1473 bytes) 000736 1745620312.844599 trunc from 0x030998 to 0x01a02b (0x1696c bytes) 000737 1745620312.865417 trunc from 0x005ecf to 0x00fd54 (0x9e86 bytes) 000738 1745620312.869812 MIRROR_EXTEND to 16 mirrors 000739 1745620313.106758 read 0x005cf3 thru 0x01331b (0xd629 bytes) 000740 1745620313.118280 trunc from 0x01a02c to 0x029db9 (0xfd8e bytes) 000741 1745620313.142950 fallocate 0x034649 thru 0x037dd1 (0x3789 bytes) 000742 1745620313.146076 punch from 0x025bd6 to 0x029bab (0x3fd6 bytes) 000743 1745620313.149137 write 0x013d92 thru 0x01bc71 (0x7ee0 bytes) 000744 1745620313.154545 read 0x02bfa5 thru 0x02ff3f (0x3f9b bytes) 000745 1745620313.161185 trunc from 0x012f05 to 0x02205c (0xf158 bytes) 000746 1745620313.168758 trunc from 0x02cf6a to 0x037dd1 (0xae68 bytes) 000747 1745620313.173148 trunc from 0x037dd2 to 0x03fb87 (0x7db6 bytes) 000748 1745620313.182897 trunc from 0x01840a to 0x020273 (0x7e6a bytes) 000749 1745620313.184923 trunc from 0x008341 to 0x010468 (0x8128 bytes) 000750 1745620313.187565 mapwrite 0x002399 thru 0x006a25 (0x468d bytes) 000751 1745620313.200430 mapread 0x00c7a5 thru 0x014ad5 (0x8331 bytes) 000752 1745620313.218619 MIRROR_RESYNC 16 mirrors 000753 1745620315.287675 fallocate 0x0251cb thru 0x02bdeb (0x6c21 bytes) 000754 1745620315.301534 punch from 0x0085c5 to 0x008c86 (0x6c2 bytes) 000755 1745620315.303391 read 0x00b157 thru 0x012614 (0x74be bytes) 000756 1745620315.306955 mapread 0x010aff thru 0x012b52 (0x2054 bytes) 000757 1745620315.308098 write 0x026028 thru 0x028bc1 (0x2b9a bytes) 000758 1745620315.313549 trunc from 0x017dd1 to 0x025cad (0xdedd bytes) 000759 1745620315.315897 trunc from 0x0162ad to 0x01f438 (0x918c bytes) 000760 1745620315.323116 MIRROR_RESYNC 16 mirrors 000761 1745620316.253716 mapwrite 0x01529f thru 0x017555 (0x22b7 bytes) 000762 1745620316.312690 punch from 0x0383d4 to 0x03fb87 (0x77b4 bytes) 000763 1745620316.316030 write 0x02a6c1 thru 0x02c1ab (0x1aeb bytes) 000764 1745620316.322291 fallocate 0x0019e8 thru 0x01060c (0xec25 bytes) 000765 1745620316.324686 fallocate 0x031a2b thru 0x03fb37 (0xe10d bytes) 000766 1745620316.327941 write 0x022921 thru 0x028b19 (0x61f9 bytes) 000767 1745620316.331082 read 0x033f70 thru 0x03fb87 (0xbc18 bytes) 000768 1745620316.336245 mapread 0x02b104 thru 0x02cc1a (0x1b17 bytes) 000769 1745620316.337117 mapread 0x0394de thru 0x03d941 (0x4464 bytes) 000770 1745620316.339709 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000772 1745620316.437496 mapread 0x02b0a2 thru 0x032988 (0x78e7 bytes) 000773 1745620316.446714 trunc from 0x03fb88 to 0x00cac9 (0x330be bytes) 000774 1745620316.456151 fallocate 0x03f989 thru 0x03ffff (0x677 bytes) 000775 1745620316.459272 mapwrite 0x02732b thru 0x0315ba (0xa290 bytes) 000776 1745620316.480282 punch from 0x01eab7 to 0x02e9e9 (0xff33 bytes) 000777 1745620316.483474 fallocate 0x01e570 thru 0x02c0f5 (0xdb86 bytes) 000778 1745620316.488796 mapwrite 0x027553 thru 0x03293b (0xb3e9 bytes) 000779 1745620316.504538 punch from 0x016589 to 0x01f287 (0x8cff bytes) 000780 1745620316.507151 fallocate 0x01aa88 thru 0x01e2b6 (0x382f bytes) 000781 1745620316.513848 punch from 0x03850d to 0x03ffff (0x7af3 bytes) 000782 1745620316.516743 trunc from 0x013c9f to 0x0190dc (0x543e bytes) 000783 1745620316.518964 read 0x00b37d thru 0x012946 (0x75ca bytes) 000785 1745620316.522852 fallocate 0x00d230 thru 0x010a15 (0x37e6 bytes) 000786 1745620316.525638 trunc from 0x009093 to 0x011a66 (0x89d4 bytes) 000787 1745620316.528341 mapwrite 0x01b771 thru 0x01bf13 (0x7a3 bytes) 000788 1745620316.540774 fallocate 0x01294b thru 0x0151d2 (0x2888 bytes) 000789 1745620316.543053 mapwrite 0x0358aa thru 0x03ffff (0xa756 bytes) 000790 1745620316.557106 trunc from 0x040000 to 0x0202c5 (0x1fd3a bytes) 000791 1745620316.565018 read 0x00454d thru 0x006ff8 (0x2aac bytes) 000792 1745620316.570757 read 0x017fed thru 0x01ee2f (0x6e43 bytes) 000793 1745620316.571199 punch from 0x01a648 to 0x0202c5 (0x5c7e bytes) 000794 1745620316.575941 punch from 0x0179c7 to 0x0202c5 (0x88ff bytes) 000795 1745620316.578787 trunc from 0x00802c to 0x010852 (0x8827 bytes) 000796 1745620316.584264 mapwrite 0x02bf86 thru 0x03726f (0xb2ea bytes) 000797 1745620316.615973 trunc from 0x037270 to 0x001e03 (0x3546c bytes) 000798 1745620316.622581 mapread 0x00020f thru 0x001e03 (0x1bf5 bytes) 000800 1745620316.624173 trunc from 0x000686 to 0x001e03 (0x177e bytes) 000801 1745620316.627310 mapread 0x00129c thru 0x001e03 (0xb68 bytes) 000803 1745620316.631083 mapread 0x00056e thru 0x001e03 (0x1896 bytes) 000804 1745620316.635479 trunc from 0x0018e0 to 0x001e03 (0x524 bytes) 000805 1745620316.641383 MIRROR_EXTEND to 16 mirrors 000806 1745620316.816190 trunc from 0x000898 to 0x001e03 (0x156c bytes) 000807 1745620316.839776 mapread 0x000f80 thru 0x001e03 (0xe84 bytes) 000808 1745620316.848628 mapread 0x0016af thru 0x001e03 (0x755 bytes) 000809 1745620316.849161 write 0x021031 thru 0x02fbd1 (0xeba1 bytes) 000811 1745620316.853378 read 0x01774d thru 0x01b87b (0x412f bytes) 000813 1745620316.860339 read 0x02254f thru 0x02fbd1 (0xd683 bytes) 000814 1745620316.860825 write 0x02d153 thru 0x0355ee (0x849c bytes) 000815 1745620316.861706 fallocate 0x00eb70 thru 0x013122 (0x45b3 bytes) 000816 1745620316.863717 read 0x018afd thru 0x0281be (0xf6c2 bytes) 000817 1745620316.868101 mapread 0x01bca0 thru 0x029631 (0xd992 bytes) 000818 1745620316.870827 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000819 1745620316.971641 MIRROR_RESYNC 15 mirrors 000820 1745620317.661039 punch from 0x01e71f to 0x0246a3 (0x5f85 bytes) 000822 1745620317.689456 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000823 1745620317.770573 write 0x00452c thru 0x00a0b5 (0x5b8a bytes) 000824 1745620317.783098 write 0x0330a9 thru 0x03ffff (0xcf57 bytes) 000825 1745620317.786670 punch from 0x022428 to 0x02e12f (0xbd08 bytes) 000826 1745620317.789623 mapwrite 0x03c4f4 thru 0x03ffff (0x3b0c bytes) 000827 1745620317.802432 mapread 0x01bf59 thru 0x023f01 (0x7fa9 bytes) 000828 1745620317.811537 read 0x00bb3a thru 0x0140e7 (0x85ae bytes) 000829 1745620317.815098 read 0x006cea thru 0x00c35d (0x5674 bytes) 000830 1745620317.815564 fallocate 0x02d3f3 thru 0x03beea (0xeaf8 bytes) 000831 1745620317.818049 trunc from 0x040000 to 0x03446c (0xbb93 bytes) 000832 1745620317.826927 MIRROR_RESYNC 14 mirrors 000833 1745620318.714469 fallocate 0x03d790 thru 0x03ecbe (0x152f bytes) 000834 1745620318.730286 MIRROR_RESYNC 14 mirrors 000835 1745620319.699573 trunc from 0x03ecbf to 0x02b369 (0x13955 bytes) 000836 1745620319.718168 MIRROR_EXTEND to 15 mirrors 000837 1745620319.905906 read 0x00eda2 thru 0x01a890 (0xbaef bytes) 000838 1745620319.912990 fallocate 0x00e112 thru 0x013897 (0x5786 bytes) 000839 1745620319.927733 read 0x016eb4 thru 0x023bc5 (0xcd12 bytes) 000841 1745620319.933104 trunc from 0x02b36a to 0x0075a8 (0x23dc1 bytes) 000842 1745620319.943434 fallocate 0x009f8e thru 0x012eec (0x8f5f bytes) 000843 1745620319.945498 mapread 0x005af8 thru 0x00d3fe (0x7907 bytes) 000844 1745620319.955570 trunc from 0x00e94f to 0x012eec (0x459e bytes) 000845 1745620319.957643 write 0x02faae thru 0x03a0a2 (0xa5f5 bytes) 000846 1745620319.958646 punch from 0x02ea58 to 0x037901 (0x8eaa bytes) 000847 1745620319.964251 punch from 0x01cd0f to 0x02b2d0 (0xe5c2 bytes) 000848 1745620319.966177 mapread 0x036df4 thru 0x03a0a2 (0x32af bytes) 000849 1745620319.971397 read 0x00d2f6 thru 0x00d991 (0x69c bytes) 000850 1745620319.974439 MIRROR_EXTEND to 16 mirrors 000851 1745620320.188089 punch from 0x013ff7 to 0x0203e1 (0xc3eb bytes) 000852 1745620320.208997 mapwrite 0x00b220 thru 0x0134b3 (0x8294 bytes) 000853 1745620320.237299 trunc from 0x03a0a3 to 0x037dca (0x22d8 bytes) 000854 1745620320.250169 fallocate 0x02f4df thru 0x03d540 (0xe062 bytes) 000855 1745620320.253449 trunc from 0x006afd to 0x013978 (0xce7c bytes) 000856 1745620320.256753 trunc from 0x03d541 to 0x00121b (0x3c325 bytes) 000857 1745620320.266664 mapwrite 0x0104b7 thru 0x01cc03 (0xc74d bytes) 000858 1745620320.294697 mapwrite 0x03851b thru 0x03ffff (0x7ae5 bytes) 000859 1745620320.317056 read 0x01976a thru 0x01c459 (0x2cf0 bytes) 000861 1745620320.317833 read 0x017830 thru 0x020c60 (0x9431 bytes) 000863 1745620320.323537 read 0x02bccd thru 0x03a9c8 (0xecfc bytes) 000865 1745620320.324614 mapwrite 0x028672 thru 0x02ef4f (0x68de bytes) 000866 1745620320.333862 trunc from 0x01d71e to 0x02b26f (0xdb52 bytes) 000867 1745620320.337099 read 0x0232eb thru 0x02957c (0x6292 bytes) 000868 1745620320.342544 fallocate 0x01f29a thru 0x02e7a9 (0xf510 bytes) 000869 1745620320.345526 write 0x0032f3 thru 0x006ea5 (0x3bb3 bytes) 000870 1745620320.346316 fallocate 0x00342a thru 0x00fb7c (0xc753 bytes) 000872 1745620320.349541 mapread 0x02d0e7 thru 0x033134 (0x604e bytes) 000873 1745620320.350145 trunc from 0x007d2a to 0x010e98 (0x916f bytes) 000875 1745620320.354532 write 0x025a72 thru 0x03550e (0xfa9d bytes) 000876 1745620320.356122 fallocate 0x019a04 thru 0x01fbc3 (0x61c0 bytes) 000877 1745620320.359628 read 0x02bfd7 thru 0x031f93 (0x5fbd bytes) 000878 1745620320.360049 fallocate 0x0058ed thru 0x00e470 (0x8b84 bytes) 000879 1745620320.362717 write 0x020b3a thru 0x02418c (0x3653 bytes) 000880 1745620320.365158 MIRROR_RESYNC 16 mirrors 000881 1745620321.519866 mapwrite 0x00f945 thru 0x01a19f (0xa85b bytes) 000882 1745620321.584332 write 0x009cea thru 0x01848f (0xe7a6 bytes) 000883 1745620321.586011 punch from 0x014e9b to 0x018116 (0x327c bytes) 000884 1745620321.593793 trunc from 0x007e2d to 0x00c21b (0x43ef bytes) 000885 1745620321.596253 mapread 0x03e6e4 thru 0x03ffff (0x191c bytes) 000886 1745620321.601375 punch from 0x00626d to 0x00632b (0xbf bytes) 000887 1745620321.606261 mapwrite 0x00b1fb thru 0x015b0f (0xa915 bytes) 000888 1745620321.627700 trunc from 0x040000 to 0x03a611 (0x59ee bytes) 000889 1745620321.637288 MIRROR_RESYNC 16 mirrors 000891 1745620322.756036 write 0x00c3f6 thru 0x00fcc4 (0x38cf bytes) 000892 1745620322.778634 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000893 1745620322.856399 punch from 0x0341e8 to 0x03a611 (0x642a bytes) 000894 1745620322.859015 mapwrite 0x006c82 thru 0x00e827 (0x7ba6 bytes) 000895 1745620322.879476 fallocate 0x03347f thru 0x03ffff (0xcb81 bytes) 000896 1745620322.881699 write 0x00e078 thru 0x012793 (0x471c bytes) 000897 1745620322.887731 MIRROR_RESYNC 15 mirrors 000899 1745620324.158088 write 0x02ea76 thru 0x0355f1 (0x6b7c bytes) 000900 1745620324.178558 fallocate 0x019b79 thru 0x01bd2d (0x21b5 bytes) 000901 1745620324.182340 mapwrite 0x013883 thru 0x01ae74 (0x75f2 bytes) 000902 1745620324.206282 fallocate 0x02c3d0 thru 0x02d303 (0xf34 bytes) 000903 1745620324.209677 trunc from 0x02600c to 0x031b47 (0xbb3c bytes) 000904 1745620324.215393 mapread 0x034183 thru 0x03a611 (0x648f bytes) 000905 1745620324.222705 read 0x0124cd thru 0x01354c (0x1080 bytes) 000906 1745620324.223046 mapread 0x014af6 thru 0x01fdc2 (0xb2cd bytes) 000907 1745620324.227245 trunc from 0x013917 to 0x02159c (0xdc86 bytes) 000908 1745620324.231041 trunc from 0x027c39 to 0x02aea4 (0x326c bytes) 000909 1745620324.233555 trunc from 0x007d51 to 0x00fcee (0x7f9e bytes) 000910 1745620324.235957 mapread 0x01bec0 thru 0x0284ee (0xc62f bytes) 000911 1745620324.248558 fallocate 0x01ffa1 thru 0x027533 (0x7593 bytes) 000912 1745620324.254981 mapread 0x0011d2 thru 0x00ddbc (0xcbeb bytes) 000913 1745620324.270351 fallocate 0x03b0dc thru 0x03ffff (0x4f24 bytes) 000914 1745620324.276534 mapwrite 0x017f72 thru 0x020ff2 (0x9081 bytes) 000915 1745620324.293667 fallocate 0x018de3 thru 0x01b33b (0x2559 bytes) 000916 1745620324.298073 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000917 1745620324.387337 punch from 0x034aab to 0x03ba96 (0x6fec bytes) 000918 1745620324.389492 trunc from 0x040000 to 0x00ac2c (0x353d3 bytes) 000919 1745620324.397502 mapwrite 0x036daa thru 0x03ffff (0x9256 bytes) 000920 1745620324.434917 fallocate 0x019e16 thru 0x02647e (0xc669 bytes) 000921 1745620324.441956 trunc from 0x040000 to 0x00d892 (0x3276d bytes) 000922 1745620324.451666 read 0x000202 thru 0x00d892 (0xd691 bytes) 000923 1745620324.456525 trunc from 0x00d893 to 0x03a222 (0x2c990 bytes) 000924 1745620324.466082 write 0x03eece thru 0x03ffff (0x1132 bytes) 000925 1745620324.470921 MIRROR_EXTEND to 15 mirrors 000927 1745620324.772793 mapread 0x027a36 thru 0x036ee0 (0xf4ab bytes) 000928 1745620324.791508 trunc from 0x040000 to 0x0218e5 (0x1e71a bytes) 000929 1745620324.838767 trunc from 0x0218e6 to 0x022304 (0xa1f bytes) 000930 1745620324.847541 fallocate 0x02d150 thru 0x034485 (0x7336 bytes) 000931 1745620324.850344 trunc from 0x022305 to 0x004b0e (0x1d7f6 bytes) 000932 1745620324.858869 punch from 0x004131 to 0x004b0e (0x9de bytes) 000933 1745620324.864302 trunc from 0x001b09 to 0x004b0e (0x3006 bytes) 000934 1745620324.867603 mapread 0x00211b thru 0x004b0e (0x29f4 bytes) 000935 1745620324.871545 mapread 0x003f51 thru 0x004b0e (0xbbe bytes) 000936 1745620324.872437 write 0x024458 thru 0x02d0ee (0x8c97 bytes) 000937 1745620324.873605 write 0x038637 thru 0x03c5c3 (0x3f8d bytes) 000938 1745620324.874557 fallocate 0x01a507 thru 0x024ae8 (0xa5e2 bytes) 000939 1745620324.878374 punch from 0x029ad8 to 0x02b84d (0x1d76 bytes) 000940 1745620324.886979 write 0x0032e4 thru 0x00beaa (0x8bc7 bytes) 000941 1745620324.890061 mapread 0x02d12e thru 0x03c0ee (0xefc1 bytes) 000942 1745620324.896139 mapread 0x037c5e thru 0x03c5c3 (0x4966 bytes) 000943 1745620324.901739 MIRROR_RESYNC 15 mirrors 000944 1745620326.276323 mapwrite 0x0302e1 thru 0x032a0b (0x272b bytes) 000945 1745620326.354950 read 0x023b0d thru 0x031311 (0xd805 bytes) 000946 1745620326.364119 MIRROR_EXTEND to 16 mirrors 000947 1745620326.641516 MIRROR_RESYNC 16 mirrors 000948 1745620328.267187 mapread 0x012e4d thru 0x01691d (0x3ad1 bytes) 000949 1745620328.274092 punch from 0x031bf1 to 0x03c5c3 (0xa9d3 bytes) 000951 1745620328.291111 trunc from 0x01e74f to 0x01ed47 (0x5f9 bytes) 000952 1745620328.293360 trunc from 0x03c5c4 to 0x034b07 (0x7abc bytes) 000953 1745620328.309874 mapread 0x00e4df thru 0x01da20 (0xf542 bytes) 000954 1745620328.319088 trunc from 0x0294e2 to 0x033b23 (0xa642 bytes) 000955 1745620328.323862 punch from 0x01fc37 to 0x02c538 (0xc902 bytes) 000957 1745620328.327592 mapread 0x02503a thru 0x02e9ba (0x9981 bytes) 000958 1745620328.337428 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000959 1745620328.426132 mapwrite 0x03b54d thru 0x03ffff (0x4ab3 bytes) 000960 1745620328.456644 mapwrite 0x0115e9 thru 0x0212f1 (0xfd09 bytes) 000961 1745620328.475027 read 0x02f954 thru 0x032ff0 (0x369d bytes) 000962 1745620328.478019 read 0x03b9b8 thru 0x03ffff (0x4648 bytes) 000963 1745620328.478997 fallocate 0x03410c thru 0x039191 (0x5086 bytes) 000964 1745620328.488560 MIRROR_RESYNC 15 mirrors 000965 1745620329.598757 punch from 0x013f71 to 0x016e36 (0x2ec6 bytes) 000967 1745620329.616324 MIRROR_RESYNC 15 mirrors 000968 1745620330.936935 read 0x03c302 thru 0x03ffff (0x3cfe bytes) 000969 1745620330.939168 trunc from 0x040000 to 0x00185d (0x3e7a2 bytes) 000970 1745620330.961055 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000971 1745620331.040143 trunc from 0x00185e to 0x02aa51 (0x291f4 bytes) 000973 1745620331.055930 MIRROR_RESYNC 14 mirrors 000974 1745620331.749319 trunc from 0x01cd3c to 0x027d3c (0xb001 bytes) 000975 1745620331.782910 write 0x01686c thru 0x0233cf (0xcb64 bytes) 000977 1745620331.791263 trunc from 0x02aa52 to 0x029760 (0x12f1 bytes) 000978 1745620331.806035 fallocate 0x03249a thru 0x03ffff (0xdb66 bytes) 000979 1745620331.811987 read 0x015951 thru 0x020d48 (0xb3f8 bytes) 000980 1745620331.815088 punch from 0x02273e to 0x029760 (0x7023 bytes) 000981 1745620331.822393 fallocate 0x027ef3 thru 0x0301f4 (0x8302 bytes) 000982 1745620331.825631 punch from 0x024db1 to 0x029760 (0x49b0 bytes) 000983 1745620331.830841 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6c8:0x0]: Invalid argument (22) 000984 1745620331.943388 read 0x000eac thru 0x00bba8 (0xacfd bytes) 000985 1745620331.947998 punch from 0x01bd29 to 0x020377 (0x464f bytes) 000986 1745620331.950738 trunc from 0x028317 to 0x028771 (0x45b bytes) 000987 1745620331.953745 punch from 0x00ed7c to 0x00f376 (0x5fb bytes) 000988 1745620331.956924 mapwrite 0x0200bc thru 0x021e7e (0x1dc3 bytes) 000989 1745620331.975198 MIRROR_EXTEND to 14 mirrors 000990 1745620332.204410 read 0x00adcf thru 0x00ef75 (0x41a7 bytes) 000991 1745620332.211309 MIRROR_EXTEND to 15 mirrors 000992 1745620332.485681 fallocate 0x006093 thru 0x01492a (0xe898 bytes) 000993 1745620332.511246 punch from 0x009593 to 0x018aa0 (0xf50e bytes) 000994 1745620332.514300 mapwrite 0x019602 thru 0x026f43 (0xd942 bytes) 000995 1745620332.538666 mapwrite 0x007044 thru 0x00ee3d (0x7dfa bytes) 000996 1745620332.554883 read 0x026827 thru 0x029760 (0x2f3a bytes) 000997 1745620332.555245 punch from 0x01bea3 to 0x0245ff (0x875d bytes) 000998 1745620332.557746 mapwrite 0x02037c thru 0x028330 (0x7fb5 bytes) 000999 1745620332.572111 mapread 0x015363 thru 0x01e56a (0x9208 bytes) 001000 1745620332.575092 read 0x025e05 thru 0x029760 (0x395c bytes) All operations completed A-OK! PASS 70a (63s) == sanity-flr test 71: check mirror extend parent fid ==== 18:32:14 (1745620334) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x71e:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 18:32:17 (1745620337) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1207008 .. Extending file size to 370752 .. done Extending file size to 7568288 .. Extending file size to 1577600 .. Extending file size to 5878080 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7846208 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8435456 .. failed Extending file size to 8321792 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9222016 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8487744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8620800 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8708352 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9043712 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 32762 32763 32764 301 302 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) 192.168.204.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:33:21 (1745620401) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6491488 .. Extending file size to 2853344 .. Extending file size to 4017792 .. Extending file size to 7257312 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8145568 .. Extending file size to 7817600 .. done Extending file size to 8397888 .. Extending file size to 7847040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8269536 .. failed Extending file size to 8578560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8560416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8765184 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8794624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9098944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8883840 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9206720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9348640 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 4150 4151 4153 4154 4157 done 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) 192.168.204.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:34:25 (1745620465) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2672 1285016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3116 1284572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15924 3591096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22040 3584980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37964 7176076 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2672 1285016 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3116 1284572 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15924 3591096 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22040 3584980 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37964 7176076 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0375204 s, 55.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0489749 s, 42.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:34:30 (1745620470) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0305237 s, 68.7 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 18:34:31 (1745620471) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:34:33 (1745620473) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263672 s, 79.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7f7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:34:35 (1745620475) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0258463 s, 81.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7f9:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:34:37 (1745620477) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0307262 s, 68.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:35:00 (1745620500) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg445-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191231 s, 54.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7f9:0x0] } PASS 204d (30s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:35:30 (1745620530) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg445-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0563157 s, 37.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:36:05 (1745620565) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg445-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg445-server: Pool lustre.test_204f created Adding targets to pool oleg445-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg445-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.055457 s, 37.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x7fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x7fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg445-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg445-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg445-server: Pool lustre.test_204f destroyed PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:36:57 (1745620617) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:37:01 (1745620621) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:37:04 (1745620624) oleg445-server: Pool lustre.test_206 created Adding targets to pool oleg445-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg445-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg445-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg445-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg445-server: Pool lustre.test_206 destroyed PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:37:23 (1745620643) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0529004 s, 19.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016203 s, 64.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xcfab1f5a Mirror 3: 0xcfab1f5a PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:37:27 (1745620647) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:37:28 (1745620648) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:37:30 (1745620650) Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0270497 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x890000 Read this (4 bytes): BBBB 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:37:48 (1745620668) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00614828 s, 1.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:37:50 (1745620670) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247841 s, 42.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x78 lmm_fid: [0x200000405:0x78:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x78 lmm_fid: [0x200000405:0x78:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611256 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-25 18:37:51.000000000 -0400 Modify: 2025-04-25 18:37:51.000000000 -0400 Change: 2025-04-25 18:37:52.000000000 -0400 Birth: - PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:37:54 (1745620674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174198 s, 60.2 MB/s oleg445-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg445-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg445-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg445-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg445-server: O/280000401/d21/b5: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:38:08 (1745620688) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0616285 s, 665 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0122301 s, 335 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 25 18:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 25 18:38 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 892 sec ============ 18:38:11 (1745620691) === sanity-flr: start cleanup 18:38:12 (1745620692) === === sanity-flr: finish cleanup 18:38:17 (1745620697) ===