-----============= acceptance-small: sanity-flr ============----- Mon Dec 2 13:25:30 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 13:25:39 (1733163939) === oleg125-client.virtnet: executing check_config_client /mnt/lustre oleg125-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg125-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3ac7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3ac7000.idle_timeout=debug disable quota as required oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:25:55 (1733163955) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7571] == sanity-flr test 0a: lfs mirror create with -N option == 13:25:56 (1733163956) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:26:04 (1733163964) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:26:05 (1733163965) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:26:07 (1733163967) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:26:16 (1733163976) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:26:18 (1733163978) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 13:26:20 (1733163980) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b3ac7000.stats=clear osc.lustre-OST0001-osc-ffff8800b3ac7000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:26:23 (1733163983) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:26:28 (1733163988) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:26:31 (1733163991) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:26:33 (1733163993) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 13:26:37 (1733163997) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:26:40 (1733164000) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:26:43 (1733164003) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:26:47 (1733164007) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:26:52 (1733164012) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:26:55 (1733164015) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.490669 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0564845 s, 18.6 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:26:59 (1733164019) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0894475 s, 70.3 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:27:03 (1733164023) Stopping /mnt/lustre-ost1 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg125-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg125-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:27:32 (1733164052) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0480769 s, 43.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg125-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:27:47 (1733164067) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg125-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg125-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:28:12 (1733164092) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.04737 s, 50.1 MB/s -rw-r--r-- 1 root root 52428800 Dec 2 13:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.318s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.362s user 0m0.000s sys 0m0.122s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg125-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg125-server oleg125-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.357s user 0m0.000s sys 0m0.132s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg125-server: oleg125-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg125-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3ac7000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:28:39 (1733164119) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:28:40 (1733164120) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:28:41 (1733164121) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 13:28:43 (1733164123) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 13:28:45 (1733164125) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.199996 s, 62.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:28:49 (1733164129) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119756 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172727 s, 60.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:28:51 (1733164131) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:28:55 (1733164135) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00194721 s, 526 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00100491 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00203039 s, 504 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000906961 s, 1.1 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 13:28:58 (1733164138) create test files with size 15738563 .. cedaba1103b342a16ff326907888d14c - eb37e708e0f005f6b5bd2f7211f638e7 - 6d28a67ea96d67adb006b3e58d401477 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg125-client.virtnet /mnt/lustre (opts:) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 13:29:06 (1733164146) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:29:10 (1733164150) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:29:12 (1733164152) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0275037 s, 38.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125514 s, 83.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 13:29:14 (1733164154) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.038718 s, 54.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.120263 s, 34.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.076002 s, 27.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 13:29:18 (1733164158) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:29:19 (1733164159) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:29:20 (1733164160) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121049 s, 86.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 13:29:23 (1733164163) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.099862 s, 31.5 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:29:26 (1733164166) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:29:29 (1733164169) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.104013 s, 101 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg125-client.virtnet /mnt/lustre (opts:) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (4s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:29:33 (1733164173) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012b80b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0883477 s, 119 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg125-client.virtnet /mnt/lustre (opts:) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a39ca000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:29:38 (1733164178) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (10s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:29:49 (1733164189) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:29:59 (1733164199) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:30:00 (1733164200) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.102704 s, 30.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0632587 s, 49.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 1e9c4c07eaa5569c481dba1e9e93979c - mirror 1 checksum: e4b3039bb03ab625963b8ba87dde114e - mirror 2 checksum: 1e9c4c07eaa5569c481dba1e9e93979c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 2 13:30:01 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 2 13:30:01 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1e9c4c07eaa5569c481dba1e9e93979c - mirror 1 checksum: e4b3039bb03ab625963b8ba87dde114e - mirror 2 checksum: e4b3039bb03ab625963b8ba87dde114e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 2 13:30:01 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 2 13:30:01 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x55:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:30:03 (1733164203) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124561 s, 84.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 525312.. 526335: 1024: 0001: net,eof 1: 0.. 1023: 529408.. 530431: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:30:06 (1733164206) Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 13:30:13 (1733164213) shut down Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 13:30:27 (1733164227) targets are mounted 13:30:27 (1733164227) facet_failover done oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167383 s, 62.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:30:34 (1733164234) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0148821 s, 1.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00827927 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:30:37 (1733164237) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0194568 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00373883 s, 5.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:30:40 (1733164240) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00280531 s, 5.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:30:42 (1733164242) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0137417 s, 1.5 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:30:44 (1733164244) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00545387 s, 751 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0111443 s, 368 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:30:46 (1733164246) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:30:51 (1733164251) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:31:10 (1733164270) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:31:27 (1733164287) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:31:44 (1733164304) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 13:31:46 (1733164306) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26111 mirror split pid 26112 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26111 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26112 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 13:32:48 (1733164368) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4368 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x14d43 000001 1733164368.973487 fallocate 0x012a2d thru 0x014d42 (0x2316 bytes) 000002 1733164368.978857 write 0x031d1a thru 0x038384 (0x666b bytes) 000004 1733164368.980049 mapwrite 0x027bf4 thru 0x030fe5 (0x93f2 bytes) truncating to largest ever: 0x1a52d 000005 1733164369.002184 trunc from 0x038385 to 0x01a52c (0x1de58 bytes) 000006 1733164369.008481 mapread 0x00501a thru 0x00ea4e (0x9a35 bytes) 000007 1733164369.016208 mapwrite 0x01247e thru 0x01b991 (0x9514 bytes) fallocating to largest ever: 0x20864 000010 1733164369.038721 fallocate 0x014fe0 thru 0x020863 (0xb884 bytes) 000011 1733164369.041308 punch from 0x015f89 to 0x0189a4 (0x2a1c bytes) fallocating to largest ever: 0x37242 000012 1733164369.043624 fallocate 0x0289c6 thru 0x037241 (0xe87c bytes) 000013 1733164369.046346 fallocate 0x018b6f thru 0x01c7c2 (0x3c54 bytes) 000017 1733164369.049392 read 0x011db3 thru 0x015e89 (0x40d7 bytes) 000018 1733164369.052165 trunc from 0x01b992 to 0x01916c (0x2825 bytes) fallocating to largest ever: 0x40000 000020 1733164369.057491 fallocate 0x03bf74 thru 0x03ffff (0x408c bytes) 000022 1733164369.060609 punch from 0x01aeaa to 0x01f836 (0x498d bytes) 000023 1733164369.063355 punch from 0x02b596 to 0x034a10 (0x947b bytes) 000024 1733164369.065303 mapwrite 0x0258d9 thru 0x0306f7 (0xae1f bytes) 000025 1733164369.083162 mapread 0x00f58e thru 0x01727f (0x7cf2 bytes) 000026 1733164369.086862 trunc from 0x040000 to 0x004559 (0x3baa6 bytes) 000027 1733164369.093420 fallocate 0x008d8a thru 0x0156db (0xc952 bytes) 000028 1733164369.095629 mapwrite 0x002940 thru 0x00551e (0x2bdf bytes) 000029 1733164369.106272 read 0x00c7c7 thru 0x00e97c (0x21b6 bytes) 000031 1733164369.107246 read 0x014898 thru 0x0156db (0xe44 bytes) 000033 1733164369.111152 punch from 0x006d5d to 0x00ee89 (0x812d bytes) 000037 1733164369.114226 read 0x011702 thru 0x0156db (0x3fda bytes) 000038 1733164369.114563 trunc from 0x0156dc to 0x00cf78 (0x8763 bytes) 000039 1733164369.120782 trunc from 0x00cf79 to 0x02493f (0x179c7 bytes) 000040 1733164369.127396 fallocate 0x03b1af thru 0x03ffff (0x4e51 bytes) 000041 1733164369.129515 read 0x02ee2e thru 0x0340c5 (0x5298 bytes) 000042 1733164369.135379 mapwrite 0x03b8a2 thru 0x03ffff (0x475e bytes) 000045 1733164369.146183 mapwrite 0x01b21b thru 0x029256 (0xe03c bytes) 000047 1733164369.157963 write 0x00522f thru 0x00cc93 (0x7a65 bytes) 000050 1733164369.159308 fallocate 0x02caa3 thru 0x03af3c (0xe49a bytes) 000051 1733164369.161623 punch from 0x01c2d3 to 0x02a829 (0xe557 bytes) 000054 1733164369.163882 fallocate 0x01b96b thru 0x026fd9 (0xb66f bytes) 000055 1733164369.165991 punch from 0x03b1bf to 0x03ffff (0x4e41 bytes) 000057 1733164369.168300 punch from 0x03110c to 0x03d501 (0xc3f6 bytes) 000059 1733164369.170970 write 0x021de3 thru 0x026a85 (0x4ca3 bytes) 000060 1733164369.174472 read 0x00c704 thru 0x018388 (0xbc85 bytes) 000061 1733164369.174941 mapwrite 0x02157b thru 0x025277 (0x3cfd bytes) 000062 1733164369.181616 trunc from 0x040000 to 0x00faba (0x30545 bytes) 000063 1733164369.186917 mapwrite 0x02d949 thru 0x038df0 (0xb4a8 bytes) 000064 1733164369.209678 punch from 0x003498 to 0x013240 (0xfda9 bytes) 000065 1733164369.213577 punch from 0x012617 to 0x018b92 (0x657c bytes) 000066 1733164369.215661 punch from 0x01f424 to 0x0249b6 (0x5593 bytes) 000067 1733164369.218298 trunc from 0x038df1 to 0x00137f (0x37a71 bytes) 000068 1733164369.225005 mapwrite 0x0170f2 thru 0x02363a (0xc549 bytes) 000071 1733164369.268390 trunc from 0x02363b to 0x03f0d5 (0x1ba9b bytes) 000072 1733164369.273448 write 0x02e0a9 thru 0x0331cb (0x5123 bytes) 000073 1733164369.279522 write 0x00d00b thru 0x00d333 (0x329 bytes) 000074 1733164369.281643 trunc from 0x03f0d6 to 0x008bff (0x364d6 bytes) 000076 1733164369.290751 write 0x03df85 thru 0x03ffff (0x207b bytes) 000077 1733164369.291567 mapread 0x03a7f1 thru 0x03ff34 (0x5744 bytes) 000078 1733164369.298864 punch from 0x039896 to 0x03cbc3 (0x332e bytes) 000079 1733164369.300964 mapwrite 0x02dfa9 thru 0x037cee (0x9d46 bytes) 000080 1733164369.329692 fallocate 0x02e360 thru 0x037fbf (0x9c60 bytes) 000081 1733164369.331647 fallocate 0x00fa85 thru 0x01b956 (0xbed2 bytes) 000082 1733164369.333395 fallocate 0x012974 thru 0x01d212 (0xa89f bytes) 000083 1733164369.335013 read 0x03bf5c thru 0x03cd7f (0xe24 bytes) 000084 1733164369.341751 trunc from 0x040000 to 0x0254f4 (0x1ab0b bytes) 000085 1733164369.346801 fallocate 0x00e6f0 thru 0x011f35 (0x3846 bytes) 000086 1733164369.350405 fallocate 0x00b1d2 thru 0x010304 (0x5133 bytes) 000087 1733164369.352842 mapread 0x01de73 thru 0x0254f4 (0x7682 bytes) 000090 1733164369.354301 trunc from 0x0254f5 to 0x015cd7 (0xf81d bytes) 000091 1733164369.359907 punch from 0x00ddc5 to 0x015cd7 (0x7f13 bytes) 000095 1733164369.363913 write 0x03feac thru 0x03ffff (0x154 bytes) 000096 1733164369.364618 trunc from 0x040000 to 0x022889 (0x1d776 bytes) 000097 1733164369.370946 punch from 0x003067 to 0x00816a (0x5104 bytes) 000098 1733164369.373760 trunc from 0x02288a to 0x009d7c (0x18b0d bytes) 000101 1733164369.381696 read 0x000360 thru 0x009d7c (0x9a1d bytes) 000102 1733164369.386066 mapwrite 0x000e96 thru 0x007970 (0x6adb bytes) 000103 1733164369.397677 fallocate 0x0040eb thru 0x00f255 (0xb16b bytes) 000104 1733164369.401833 fallocate 0x005419 thru 0x013dd7 (0xe9bf bytes) 000105 1733164369.404594 fallocate 0x02eb46 thru 0x03dc5c (0xf117 bytes) 000106 1733164369.406895 mapwrite 0x033847 thru 0x03546d (0x1c27 bytes) 000107 1733164369.423719 mapread 0x03774f thru 0x03dc5c (0x650e bytes) 000110 1733164369.425573 fallocate 0x038c78 thru 0x03ffff (0x7388 bytes) 000111 1733164369.429310 read 0x027fdc thru 0x0335f3 (0xb618 bytes) 000112 1733164369.429748 write 0x03048d thru 0x039d61 (0x98d5 bytes) 000113 1733164369.430653 write 0x02df7b thru 0x033531 (0x55b7 bytes) 000114 1733164369.431364 mapwrite 0x020295 thru 0x024fb5 (0x4d21 bytes) 000116 1733164369.439972 mapwrite 0x0048b3 thru 0x01333c (0xea8a bytes) 000117 1733164369.449478 trunc from 0x03dc5d to 0x03b8b2 (0x23aa bytes) 000118 1733164369.454974 punch from 0x022c03 to 0x02cac0 (0x9ebe bytes) 000121 1733164369.457828 mapwrite 0x022d96 thru 0x02538f (0x25fa bytes) 000122 1733164369.470314 trunc from 0x03b8b3 to 0x0145bb (0x272f7 bytes) 000123 1733164369.476581 read 0x003e1c thru 0x010640 (0xc825 bytes) 000124 1733164369.477023 trunc from 0x0145bc to 0x0040e0 (0x104db bytes) 000126 1733164369.483418 mapwrite 0x02535a thru 0x0299e0 (0x4687 bytes) 000127 1733164369.504835 fallocate 0x00c494 thru 0x011f1d (0x5a8a bytes) 000128 1733164369.507719 trunc from 0x0299e1 to 0x036626 (0xcc46 bytes) 000131 1733164369.515326 read 0x031a97 thru 0x036626 (0x4b90 bytes) 000132 1733164369.520816 trunc from 0x036627 to 0x002559 (0x340cd bytes) 000134 1733164369.528658 write 0x02c81b thru 0x037c6c (0xb452 bytes) 000135 1733164369.529764 write 0x004f1e thru 0x00c3dd (0x74c0 bytes) 000136 1733164369.534064 mapread 0x0179a3 thru 0x020969 (0x8fc7 bytes) 000137 1733164369.548852 mapread 0x002353 thru 0x0119d1 (0xf67f bytes) 000138 1733164369.558381 read 0x01ff05 thru 0x02bace (0xbbca bytes) 000141 1733164369.565229 MIRROR_EXTEND to 2 mirrors 000142 1733164369.744912 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000143 1733164369.797163 fallocate 0x001de7 thru 0x00c5e5 (0xa7ff bytes) 000144 1733164369.804357 read 0x025da8 thru 0x0268b9 (0xb12 bytes) 000145 1733164369.809000 write 0x02c6b3 thru 0x02e6e1 (0x202f bytes) 000146 1733164369.815224 trunc from 0x037c6d to 0x00d92f (0x2a33d bytes) 000148 1733164369.835462 write 0x00c9c3 thru 0x00ee0f (0x244d bytes) 000149 1733164369.840824 mapread 0x006eab thru 0x008c46 (0x1d9c bytes) 000150 1733164369.841715 read 0x00cbed thru 0x00ee0f (0x2223 bytes) 000151 1733164369.842050 read 0x008357 thru 0x00ee0f (0x6ab9 bytes) 000152 1733164369.842452 write 0x02b1ae thru 0x031ced (0x6b40 bytes) 000153 1733164369.843306 punch from 0x02cbc5 to 0x031ced (0x5129 bytes) 000156 1733164369.856879 trunc from 0x031cee to 0x01ac7a (0x17073 bytes) 000157 1733164369.862941 mapwrite 0x0229b7 thru 0x02cf41 (0xa58b bytes) 000159 1733164369.889432 read 0x01d2a5 thru 0x0271f2 (0x9f4e bytes) 000160 1733164369.889884 write 0x028e4f thru 0x034ea9 (0xc05b bytes) 000161 1733164369.890852 mapwrite 0x006d00 thru 0x01184f (0xab50 bytes) 000162 1733164369.904631 mapread 0x0054be thru 0x00a412 (0x4f55 bytes) 000163 1733164369.905250 punch from 0x02bfbc to 0x02e353 (0x2398 bytes) 000164 1733164369.909982 trunc from 0x034eaa to 0x00274a (0x3275f bytes) 000166 1733164369.920318 punch from 0x00152f to 0x00274a (0x121c bytes) 000168 1733164369.924642 punch from 0x000f74 to 0x00274a (0x17d7 bytes) 000171 1733164369.929427 fallocate 0x00fbec thru 0x01d3d6 (0xd7eb bytes) 000173 1733164369.935713 mapread 0x0006bc thru 0x00274a (0x208f bytes) 000175 1733164369.940517 fallocate 0x03a087 thru 0x03ffff (0x5f79 bytes) 000176 1733164369.944445 read 0x0284e0 thru 0x031283 (0x8da4 bytes) 000177 1733164369.950733 punch from 0x022830 to 0x02fd8d (0xd55e bytes) 000179 1733164369.956934 read 0x00ea97 thru 0x0194c8 (0xaa32 bytes) 000180 1733164369.957581 mapwrite 0x016de1 thru 0x020b02 (0x9d22 bytes) 000181 1733164369.974533 punch from 0x000a7c to 0x00e23a (0xd7bf bytes) 000182 1733164369.979224 fallocate 0x01d322 thru 0x01e7cd (0x14ac bytes) 000183 1733164369.982369 mapwrite 0x03092c thru 0x03fc26 (0xf2fb bytes) 000184 1733164369.995699 mapread 0x03ef87 thru 0x03ffff (0x1079 bytes) 000185 1733164369.996212 write 0x00bd31 thru 0x0135b1 (0x7881 bytes) 000186 1733164370.004714 MIRROR_EXTEND to 2 mirrors 000187 1733164370.221023 mapread 0x0327dc thru 0x03ffff (0xd824 bytes) 000188 1733164370.237323 fallocate 0x00ed4b thru 0x010b34 (0x1dea bytes) 000189 1733164370.254126 punch from 0x033796 to 0x03e7cb (0xb036 bytes) 000190 1733164370.258311 fallocate 0x00a836 thru 0x013c7e (0x9449 bytes) 000191 1733164370.261008 mapwrite 0x0366ee thru 0x03ffff (0x9912 bytes) 000192 1733164370.281895 mapread 0x01e83e thru 0x025060 (0x6823 bytes) 000193 1733164370.285645 write 0x039cd8 thru 0x03ffff (0x6328 bytes) 000194 1733164370.288083 mapread 0x03062d thru 0x0370ca (0x6a9e bytes) 000195 1733164370.288700 trunc from 0x040000 to 0x03679a (0x9865 bytes) 000196 1733164370.297648 mapread 0x010355 thru 0x0185a1 (0x824d bytes) 000197 1733164370.309210 MIRROR_RESYNC 2 mirrors 000198 1733164370.489704 fallocate 0x01c54c thru 0x021853 (0x5308 bytes) 000199 1733164370.503318 read 0x0060f7 thru 0x0094a1 (0x33ab bytes) 000202 1733164370.506621 mapread 0x033b6f thru 0x03611d (0x25af bytes) 000203 1733164370.514498 punch from 0x002035 to 0x00a19b (0x8167 bytes) 000204 1733164370.516605 write 0x03fbe2 thru 0x03ffff (0x41e bytes) 000205 1733164370.519263 MIRROR_RESYNC 2 mirrors 000208 1733164370.738955 MIRROR_EXTEND to 3 mirrors 000209 1733164370.946722 read 0x039dc6 thru 0x03ffff (0x623a bytes) 000210 1733164370.950745 mapread 0x022182 thru 0x02a6ac (0x852b bytes) 000211 1733164370.957705 punch from 0x02fa0c to 0x039809 (0x9dfe bytes) 000212 1733164370.973502 write 0x0086fc thru 0x008d50 (0x655 bytes) 000213 1733164370.979572 trunc from 0x040000 to 0x00368b (0x3c974 bytes) 000214 1733164370.997504 read 0x002635 thru 0x00368b (0x1057 bytes) 000215 1733164370.997998 mapwrite 0x019f83 thru 0x01ce9a (0x2f18 bytes) 000216 1733164371.018884 write 0x00e1ee thru 0x019fa7 (0xbdba bytes) 000217 1733164371.019922 mapwrite 0x038126 thru 0x03b19c (0x3077 bytes) 000218 1733164371.036368 write 0x02b0af thru 0x03395c (0x88ae bytes) 000219 1733164371.037353 mapread 0x034360 thru 0x03b19c (0x6e3d bytes) 000220 1733164371.038333 trunc from 0x03b19d to 0x00e4aa (0x2ccf2 bytes) 000221 1733164371.044979 mapwrite 0x038c1d thru 0x03ffff (0x73e3 bytes) 000222 1733164371.073798 MIRROR_EXTEND to 4 mirrors 000224 1733164371.248794 fallocate 0x02420d thru 0x02fca4 (0xba98 bytes) 000225 1733164371.261908 fallocate 0x02ccbc thru 0x03c83e (0xfb83 bytes) 000226 1733164371.264732 read 0x03b9e8 thru 0x03baa9 (0xc2 bytes) 000227 1733164371.267649 mapwrite 0x030338 thru 0x0333ec (0x30b5 bytes) 000228 1733164371.284240 punch from 0x0278b2 to 0x028cfb (0x144a bytes) 000229 1733164371.286884 mapwrite 0x0335a3 thru 0x03ffff (0xca5d bytes) 000230 1733164371.303894 punch from 0x036dcc to 0x03b091 (0x42c6 bytes) 000232 1733164371.307636 write 0x02106b thru 0x025550 (0x44e6 bytes) 000233 1733164371.313794 write 0x026079 thru 0x029ce6 (0x3c6e bytes) 000234 1733164371.315541 write 0x02f1c5 thru 0x03f01a (0xfe56 bytes) 000235 1733164371.318966 MIRROR_EXTEND to 5 mirrors 000236 1733164371.493428 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000237 1733164371.572906 mapwrite 0x001673 thru 0x00ffc4 (0xe952 bytes) 000238 1733164371.614953 MIRROR_RESYNC 4 mirrors 000239 1733164371.818557 read 0x012a54 thru 0x013d2e (0x12db bytes) 000241 1733164371.822256 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000242 1733164371.908421 mapwrite 0x007238 thru 0x00b93e (0x4707 bytes) 000243 1733164371.949182 write 0x02afd3 thru 0x02eed4 (0x3f02 bytes) 000244 1733164371.953949 read 0x023de4 thru 0x024ea4 (0x10c1 bytes) 000245 1733164371.954440 fallocate 0x039641 thru 0x03fe78 (0x6838 bytes) 000248 1733164371.957501 write 0x000e9d thru 0x00843d (0x75a1 bytes) 000249 1733164371.960394 write 0x00c3d7 thru 0x019000 (0xcc2a bytes) 000250 1733164371.963887 punch from 0x01c790 to 0x01d2c5 (0xb36 bytes) 000251 1733164371.965891 fallocate 0x03b631 thru 0x03ffff (0x49cf bytes) 000252 1733164371.967969 punch from 0x00a5c5 to 0x00c41b (0x1e57 bytes) 000253 1733164371.976016 read 0x037fdd thru 0x03c564 (0x4588 bytes) 000254 1733164371.981047 punch from 0x025e7f to 0x028bf3 (0x2d75 bytes) 000255 1733164371.983752 mapwrite 0x00706b thru 0x01561f (0xe5b5 bytes) 000256 1733164371.993238 mapwrite 0x0366fb thru 0x03bedd (0x57e3 bytes) 000257 1733164372.000003 punch from 0x03b4f1 to 0x03ffff (0x4b0f bytes) 000260 1733164372.005194 MIRROR_RESYNC 3 mirrors 000261 1733164372.251039 trunc from 0x040000 to 0x00c61f (0x339e0 bytes) 000263 1733164372.268498 write 0x02e022 thru 0x0397a4 (0xb783 bytes) 000264 1733164372.269528 punch from 0x01c327 to 0x02511c (0x8df6 bytes) 000265 1733164372.273898 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000267 1733164372.361739 MIRROR_RESYNC 2 mirrors 000268 1733164372.552849 read 0x00d65b thru 0x015c6c (0x8612 bytes) 000269 1733164372.558733 read 0x0331ef thru 0x039238 (0x604a bytes) 000270 1733164372.564978 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000273 1733164372.617588 write 0x000813 thru 0x000a86 (0x274 bytes) 000274 1733164372.626315 mapread 0x0066d9 thru 0x00f3b5 (0x8cdd bytes) 000275 1733164372.635962 mapwrite 0x0082e4 thru 0x01461c (0xc339 bytes) 000276 1733164372.649608 fallocate 0x0162e8 thru 0x01723c (0xf55 bytes) 000277 1733164372.652763 read 0x004020 thru 0x006e07 (0x2de8 bytes) 000278 1733164372.655989 write 0x033196 thru 0x0338d9 (0x744 bytes) 000280 1733164372.661679 trunc from 0x0397a5 to 0x015b48 (0x23c5c bytes) 000281 1733164372.674612 MIRROR_EXTEND to 2 mirrors 000282 1733164372.878458 fallocate 0x00d01a thru 0x00f4db (0x24c2 bytes) 000283 1733164372.898467 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000284 1733164372.957621 MIRROR_EXTEND to 2 mirrors 000285 1733164373.205666 write 0x019c21 thru 0x01ac41 (0x1021 bytes) 000286 1733164373.228384 trunc from 0x01ac42 to 0x010069 (0xabd8 bytes) 000287 1733164373.236897 fallocate 0x00d46b thru 0x01d3b0 (0xff46 bytes) 000288 1733164373.240107 mapread 0x00444f thru 0x010069 (0xbc1b bytes) 000289 1733164373.253934 mapread 0x00b636 thru 0x00df72 (0x293d bytes) 000290 1733164373.254565 fallocate 0x01c0ab thru 0x028906 (0xc85c bytes) 000291 1733164373.257583 punch from 0x02243b to 0x028743 (0x6309 bytes) 000292 1733164373.261085 mapwrite 0x00ec5a thru 0x01a7eb (0xbb92 bytes) 000293 1733164373.286092 fallocate 0x027a2f thru 0x035a92 (0xe064 bytes) 000294 1733164373.288872 fallocate 0x02ea13 thru 0x03865d (0x9c4b bytes) 000295 1733164373.299722 MIRROR_RESYNC 2 mirrors 000296 1733164373.459514 fallocate 0x0233b1 thru 0x030c02 (0xd852 bytes) 000297 1733164373.474970 write 0x0084b9 thru 0x00f3f5 (0x6f3d bytes) 000298 1733164373.479765 mapread 0x00050d thru 0x005ca5 (0x5799 bytes) 000299 1733164373.485523 mapwrite 0x026fab thru 0x035471 (0xe4c7 bytes) 000300 1733164373.525898 trunc from 0x035472 to 0x03bac7 (0x6656 bytes) 000301 1733164373.535289 read 0x00a8c1 thru 0x0103d1 (0x5b11 bytes) 000303 1733164373.535866 read 0x003ac4 thru 0x011565 (0xdaa2 bytes) 000304 1733164373.540199 read 0x012208 thru 0x0203f9 (0xe1f2 bytes) 000305 1733164373.540843 fallocate 0x019dfb thru 0x02230d (0x8513 bytes) 000306 1733164373.543792 write 0x02d9dc thru 0x033bc1 (0x61e6 bytes) 000307 1733164373.544754 read 0x002e61 thru 0x003fbb (0x115b bytes) 000308 1733164373.545963 read 0x006776 thru 0x008e15 (0x26a0 bytes) 000309 1733164373.546349 fallocate 0x005cb5 thru 0x0065b6 (0x902 bytes) 000310 1733164373.549995 punch from 0x000be0 to 0x00c86e (0xbc8f bytes) 000311 1733164373.553528 punch from 0x0129f9 to 0x01f11f (0xc727 bytes) 000312 1733164373.556233 read 0x002d91 thru 0x0032e8 (0x558 bytes) 000313 1733164373.561817 trunc from 0x03bac8 to 0x01a947 (0x21180 bytes) 000315 1733164373.569178 fallocate 0x005424 thru 0x00cae7 (0x76c4 bytes) 000316 1733164373.572030 read 0x0187f1 thru 0x01a947 (0x2157 bytes) 000317 1733164373.572483 write 0x007af7 thru 0x00db3f (0x6049 bytes) 000319 1733164373.576580 MIRROR_EXTEND to 3 mirrors 000321 1733164373.741251 read 0x00ff72 thru 0x01a947 (0xa9d6 bytes) 000323 1733164373.747919 write 0x02650d thru 0x0324dd (0xbfd1 bytes) 000325 1733164373.760961 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000326 1733164373.827287 mapread 0x009ef3 thru 0x017a44 (0xdb52 bytes) 000327 1733164373.837766 read 0x025207 thru 0x025a0e (0x808 bytes) 000328 1733164373.839887 mapread 0x01c93d thru 0x01e44c (0x1b10 bytes) 000329 1733164373.843866 punch from 0x006a9a to 0x00c054 (0x55bb bytes) 000330 1733164373.846927 mapwrite 0x03853f thru 0x03ffff (0x7ac1 bytes) 000331 1733164373.873405 fallocate 0x013fd2 thru 0x016117 (0x2146 bytes) 000332 1733164373.875359 punch from 0x02895a to 0x02e1e7 (0x588e bytes) 000334 1733164373.878980 trunc from 0x040000 to 0x02ce51 (0x131ae bytes) 000335 1733164373.885244 trunc from 0x02ce52 to 0x0143c3 (0x18a8e bytes) 000336 1733164373.890197 mapread 0x00fc8d thru 0x0143c3 (0x4737 bytes) 000337 1733164373.891082 mapread 0x01273f thru 0x013570 (0xe32 bytes) 000339 1733164373.891914 mapwrite 0x038367 thru 0x03ffff (0x7c99 bytes) 000340 1733164373.909326 punch from 0x01dcae to 0x02afba (0xd30d bytes) 000341 1733164373.913607 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000344 1733164373.960496 mapread 0x010db6 thru 0x01cd24 (0xbf6f bytes) 000345 1733164373.971410 read 0x01f6fb thru 0x02c1dd (0xcae3 bytes) 000348 1733164373.975337 fallocate 0x03897a thru 0x0399ce (0x1055 bytes) 000349 1733164373.977664 write 0x02d0eb thru 0x02eea5 (0x1dbb bytes) 000351 1733164373.984029 MIRROR_EXTEND to 2 mirrors 000352 1733164374.181620 write 0x031efc thru 0x0350fd (0x3202 bytes) 000353 1733164374.199075 punch from 0x00ccf6 to 0x01300a (0x6315 bytes) 000354 1733164374.201173 read 0x02d390 thru 0x037564 (0xa1d5 bytes) 000355 1733164374.204778 trunc from 0x040000 to 0x034c6f (0xb390 bytes) 000356 1733164374.211446 trunc from 0x034c70 to 0x03c657 (0x79e8 bytes) 000357 1733164374.215772 mapwrite 0x032eed thru 0x03a38d (0x74a1 bytes) 000358 1733164374.227954 punch from 0x0144b2 to 0x01af8d (0x6adc bytes) 000359 1733164374.230258 read 0x0391cb thru 0x03c657 (0x348d bytes) 000364 1733164374.233548 MIRROR_EXTEND to 3 mirrors 000365 1733164374.389426 mapwrite 0x0376ab thru 0x03abeb (0x3541 bytes) 000366 1733164374.435703 mapread 0x005c69 thru 0x012fa3 (0xd33b bytes) 000367 1733164374.442512 punch from 0x01aae7 to 0x02a9a0 (0xfeba bytes) 000368 1733164374.444575 mapread 0x036204 thru 0x039692 (0x348f bytes) 000369 1733164374.445164 fallocate 0x016c2c thru 0x0241de (0xd5b3 bytes) 000370 1733164374.448669 trunc from 0x03c658 to 0x02c135 (0x10522 bytes) 000371 1733164374.464502 mapwrite 0x02c959 thru 0x039ae4 (0xd18c bytes) 000373 1733164374.484712 trunc from 0x039ae5 to 0x02d430 (0xc6b4 bytes) 000374 1733164374.489061 mapwrite 0x013b1f thru 0x01c7c7 (0x8ca9 bytes) 000375 1733164374.501016 write 0x036f95 thru 0x03ffff (0x906b bytes) 000376 1733164374.502013 trunc from 0x040000 to 0x026775 (0x1988a bytes) 000377 1733164374.507980 write 0x01a71b thru 0x01ad29 (0x60f bytes) 000378 1733164374.510224 MIRROR_EXTEND to 4 mirrors 000379 1733164374.668333 punch from 0x00f17d to 0x01be20 (0xcca4 bytes) 000380 1733164374.684126 punch from 0x005b93 to 0x013cb5 (0xe123 bytes) 000381 1733164374.686302 punch from 0x001860 to 0x00edbd (0xd55e bytes) 000384 1733164374.688917 read 0x01056d thru 0x012192 (0x1c26 bytes) 000386 1733164374.693894 mapread 0x0029d0 thru 0x00c763 (0x9d94 bytes) 000387 1733164374.707385 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000388 1733164374.781790 trunc from 0x026776 to 0x03b2ba (0x14b45 bytes) 000389 1733164374.800639 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000390 1733164374.879785 mapwrite 0x034a0b thru 0x03573d (0xd33 bytes) 000391 1733164374.892423 MIRROR_RESYNC 2 mirrors 000392 1733164375.061487 trunc from 0x03b2bb to 0x00abad (0x3070d bytes) 000393 1733164375.075889 mapwrite 0x00337c thru 0x004d1a (0x199f bytes) 000394 1733164375.089272 mapread 0x000590 thru 0x00abad (0xa61e bytes) 000396 1733164375.096355 fallocate 0x0188da thru 0x021c29 (0x9350 bytes) 000398 1733164375.098935 read 0x02184e thru 0x021c29 (0x3dc bytes) 000399 1733164375.101786 read 0x010934 thru 0x01e4c7 (0xdb94 bytes) 000400 1733164375.106176 punch from 0x01762c to 0x021c29 (0xa5fe bytes) 000401 1733164375.108116 mapwrite 0x02ef23 thru 0x03655c (0x763a bytes) 000402 1733164375.127243 trunc from 0x03655d to 0x03f864 (0x9308 bytes) 000403 1733164375.133171 read 0x026a91 thru 0x0313eb (0xa95b bytes) 000404 1733164375.135464 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000408 1733164375.180971 mapwrite 0x0274c4 thru 0x02b6ff (0x423c bytes) 000409 1733164375.200350 trunc from 0x03f865 to 0x01db80 (0x21ce4 bytes) 000410 1733164375.208294 trunc from 0x01db81 to 0x0144de (0x96a2 bytes) 000413 1733164375.215420 trunc from 0x0144df to 0x03c4ed (0x2800f bytes) 000414 1733164375.223817 MIRROR_EXTEND to 2 mirrors 000415 1733164375.382658 MIRROR_EXTEND to 3 mirrors 000417 1733164375.551738 write 0x001c5b thru 0x00e842 (0xcbe8 bytes) 000419 1733164375.565682 mapread 0x0323cd thru 0x03c4ed (0xa121 bytes) 000420 1733164375.573607 mapread 0x03135d thru 0x03ada0 (0x9a44 bytes) 000421 1733164375.574224 trunc from 0x03c4ee to 0x02f9bf (0xcb2e bytes) 000423 1733164375.587961 mapread 0x01ecfe thru 0x029866 (0xab69 bytes) 000424 1733164375.594266 trunc from 0x02f9c0 to 0x02ec10 (0xdaf bytes) 000425 1733164375.600345 trunc from 0x02ec11 to 0x01ee83 (0xfd8d bytes) 000426 1733164375.607407 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000428 1733164375.686542 write 0x00516d thru 0x00722b (0x20bf bytes) 000429 1733164375.693820 fallocate 0x01497c thru 0x01da77 (0x90fc bytes) 000432 1733164375.696667 trunc from 0x01ee84 to 0x026656 (0x77d3 bytes) 000433 1733164375.701994 mapread 0x007305 thru 0x0083a4 (0x10a0 bytes) 000434 1733164375.705750 fallocate 0x003d78 thru 0x00f9c6 (0xbc4f bytes) 000435 1733164375.708050 mapwrite 0x014b86 thru 0x017736 (0x2bb1 bytes) 000437 1733164375.719811 read 0x00bb4d thru 0x01ad7a (0xf22e bytes) 000438 1733164375.720374 punch from 0x008ba2 to 0x0104fa (0x7959 bytes) 000440 1733164375.723552 read 0x01df84 thru 0x021fdf (0x405c bytes) 000441 1733164375.728458 punch from 0x022d78 to 0x026656 (0x38df bytes) 000442 1733164375.733202 MIRROR_RESYNC 2 mirrors 000443 1733164375.859518 write 0x01fe3c thru 0x02ed49 (0xef0e bytes) 000444 1733164375.877881 mapread 0x008f8e thru 0x017396 (0xe409 bytes) 000445 1733164375.891734 fallocate 0x0153f1 thru 0x016981 (0x1591 bytes) 000446 1733164375.894245 read 0x0143e4 thru 0x014a68 (0x685 bytes) 000448 1733164375.894960 write 0x01187b thru 0x0201e0 (0xe966 bytes) 000450 1733164375.896365 mapread 0x012c6c thru 0x0139aa (0xd3f bytes) 000451 1733164375.896928 write 0x0135b5 thru 0x017a56 (0x44a2 bytes) 000452 1733164375.899989 MIRROR_EXTEND to 3 mirrors 000453 1733164376.041774 read 0x009ba8 thru 0x015612 (0xba6b bytes) 000455 1733164376.049704 punch from 0x001989 to 0x0040fe (0x2776 bytes) 000456 1733164376.062097 write 0x0062f1 thru 0x007445 (0x1155 bytes) 000457 1733164376.067206 write 0x01d83e thru 0x01fc3e (0x2401 bytes) 000458 1733164376.070747 mapread 0x02a374 thru 0x02ed49 (0x49d6 bytes) 000459 1733164376.075030 read 0x0097eb thru 0x00c8d3 (0x30e9 bytes) 000461 1733164376.081225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000462 1733164376.144530 write 0x003c7c thru 0x007be7 (0x3f6c bytes) 000463 1733164376.151765 MIRROR_EXTEND to 3 mirrors 000464 1733164376.293495 punch from 0x00c4ea to 0x011dd9 (0x58f0 bytes) 000465 1733164376.305173 fallocate 0x02c3f2 thru 0x02d132 (0xd41 bytes) 000466 1733164376.307104 punch from 0x0015a8 to 0x0040ac (0x2b05 bytes) 000467 1733164376.309011 mapwrite 0x00f64e thru 0x01eea1 (0xf854 bytes) 000468 1733164376.326620 mapread 0x01753c thru 0x01ae62 (0x3927 bytes) 000469 1733164376.327231 punch from 0x010a5c to 0x01fb03 (0xf0a8 bytes) 000470 1733164376.329422 read 0x00a9ca thru 0x00e299 (0x38d0 bytes) 000472 1733164376.330151 mapwrite 0x000119 thru 0x00be2f (0xbd17 bytes) 000473 1733164376.340416 punch from 0x01980f to 0x022413 (0x8c05 bytes) 000474 1733164376.342629 write 0x00fb30 thru 0x0157e7 (0x5cb8 bytes) 000475 1733164376.346602 trunc from 0x02ed4a to 0x039e9a (0xb151 bytes) 000476 1733164376.357237 read 0x008f80 thru 0x014ca8 (0xbd29 bytes) 000478 1733164376.358024 punch from 0x02dff2 to 0x02f8a8 (0x18b7 bytes) 000479 1733164376.361434 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000480 1733164376.419712 mapwrite 0x023222 thru 0x031073 (0xde52 bytes) 000481 1733164376.441350 MIRROR_EXTEND to 3 mirrors 000482 1733164376.593183 MIRROR_EXTEND to 4 mirrors 000483 1733164376.769518 read 0x01e3bc thru 0x02ccc3 (0xe908 bytes) 000485 1733164376.775406 mapwrite 0x0215d3 thru 0x02702e (0x5a5c bytes) 000486 1733164376.814017 mapwrite 0x0079d8 thru 0x009fb5 (0x25de bytes) 000488 1733164376.824942 mapwrite 0x0351ed thru 0x03ffff (0xae13 bytes) 000489 1733164376.861223 fallocate 0x0341ee thru 0x03f044 (0xae57 bytes) 000490 1733164376.863536 mapwrite 0x01907c thru 0x019890 (0x815 bytes) 000491 1733164376.873100 fallocate 0x019d2f thru 0x024d36 (0xb008 bytes) 000492 1733164376.875386 mapread 0x000e9d thru 0x00342d (0x2591 bytes) 000493 1733164376.878053 write 0x03f1e4 thru 0x03ffff (0xe1c bytes) 000494 1733164376.878781 mapwrite 0x005b35 thru 0x0123b5 (0xc881 bytes) 000495 1733164376.890152 mapread 0x036113 thru 0x03b5df (0x54cd bytes) 000496 1733164376.890687 trunc from 0x040000 to 0x02e21b (0x11de4 bytes) 000497 1733164376.896370 punch from 0x00e996 to 0x011c13 (0x327e bytes) 000498 1733164376.898348 write 0x00ad47 thru 0x0176aa (0xc964 bytes) 000500 1733164376.899474 write 0x03938c thru 0x03ffff (0x6c74 bytes) 000501 1733164376.900280 write 0x02effd thru 0x032d52 (0x3d56 bytes) 000502 1733164376.904032 write 0x01391c thru 0x01e3ae (0xaa93 bytes) 000503 1733164376.906928 MIRROR_RESYNC 4 mirrors 000504 1733164377.093070 trunc from 0x040000 to 0x03e3c7 (0x1c38 bytes) 000505 1733164377.105039 mapread 0x022ae7 thru 0x023bf3 (0x110d bytes) 000507 1733164377.108557 trunc from 0x03e3c8 to 0x012ddd (0x2b5ea bytes) 000508 1733164377.114799 punch from 0x00b6ba to 0x012ddd (0x7724 bytes) 000509 1733164377.116776 write 0x03bd3c thru 0x03ffff (0x42c4 bytes) 000510 1733164377.119373 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000511 1733164377.178281 trunc from 0x040000 to 0x02c542 (0x13abd bytes) 000512 1733164377.184631 fallocate 0x0281af thru 0x02e669 (0x64bb bytes) 000513 1733164377.188362 MIRROR_EXTEND to 4 mirrors 000514 1733164377.360571 read 0x01071b thru 0x0150bb (0x49a1 bytes) 000517 1733164377.363201 punch from 0x00088b to 0x00f545 (0xecbb bytes) 000518 1733164377.374822 write 0x016b8e thru 0x01eeeb (0x835e bytes) 000519 1733164377.380225 fallocate 0x03f781 thru 0x03ffff (0x87f bytes) 000520 1733164377.382269 fallocate 0x02a997 thru 0x02e2fa (0x3964 bytes) 000521 1733164377.384236 punch from 0x019169 to 0x01fe96 (0x6d2e bytes) 000522 1733164377.389737 write 0x00ce72 thru 0x016936 (0x9ac5 bytes) 000523 1733164377.394825 mapread 0x022484 thru 0x02d26e (0xadeb bytes) 000525 1733164377.401916 punch from 0x01a618 to 0x01bd4b (0x1734 bytes) 000526 1733164377.403673 mapwrite 0x03b047 thru 0x03b528 (0x4e2 bytes) 000527 1733164377.415166 mapwrite 0x02fd45 thru 0x03cee1 (0xd19d bytes) 000528 1733164377.426706 MIRROR_EXTEND to 5 mirrors 000529 1733164377.579313 MIRROR_EXTEND to 6 mirrors 000530 1733164377.734925 mapwrite 0x0175b7 thru 0x023419 (0xbe63 bytes) 000531 1733164377.776273 write 0x0173d0 thru 0x017b40 (0x771 bytes) 000532 1733164377.778597 write 0x0107b8 thru 0x019cb6 (0x94ff bytes) 000533 1733164377.781837 read 0x00684d thru 0x009272 (0x2a26 bytes) 000535 1733164377.784244 trunc from 0x040000 to 0x000622 (0x3f9dd bytes) 000536 1733164377.798669 MIRROR_RESYNC 6 mirrors 000537 1733164377.973898 read 0x00001b thru 0x000622 (0x608 bytes) 000538 1733164377.974233 punch from 0x0001ec to 0x000622 (0x437 bytes) 000539 1733164377.990826 MIRROR_RESYNC 6 mirrors 000540 1733164378.168584 mapread 0x000258 thru 0x000622 (0x3cb bytes) 000541 1733164378.169016 mapwrite 0x0299f9 thru 0x029c3c (0x244 bytes) 000542 1733164378.195894 trunc from 0x029c3d to 0x038e71 (0xf235 bytes) 000543 1733164378.205372 MIRROR_RESYNC 6 mirrors 000544 1733164378.530483 fallocate 0x02876f thru 0x031c92 (0x9524 bytes) 000546 1733164378.545831 MIRROR_RESYNC 6 mirrors 000547 1733164378.862850 fallocate 0x02e62c thru 0x033f54 (0x5929 bytes) 000548 1733164378.877550 read 0x01525c thru 0x01d188 (0x7f2d bytes) 000549 1733164378.881126 trunc from 0x038e72 to 0x007d21 (0x31150 bytes) 000550 1733164378.889166 punch from 0x003ef6 to 0x007d21 (0x3e2c bytes) 000551 1733164378.891114 trunc from 0x007d22 to 0x0370e1 (0x2f3c0 bytes) 000552 1733164378.897065 read 0x009b85 thru 0x01314b (0x95c7 bytes) 000553 1733164378.904317 punch from 0x02772c to 0x02ebe7 (0x74bc bytes) 000554 1733164378.906740 write 0x02909d thru 0x02c3c7 (0x332b bytes) 000556 1733164378.913279 mapread 0x02ec0c thru 0x031188 (0x257d bytes) 000557 1733164378.916066 mapread 0x018d75 thru 0x01a826 (0x1ab2 bytes) 000558 1733164378.916714 punch from 0x019392 to 0x01c02c (0x2c9b bytes) 000559 1733164378.918835 fallocate 0x014900 thru 0x01fd3f (0xb440 bytes) 000560 1733164378.922613 mapwrite 0x03c2de thru 0x03ffff (0x3d22 bytes) 000562 1733164378.945049 punch from 0x000ec0 to 0x0071ba (0x62fb bytes) 000563 1733164378.948493 mapread 0x012861 thru 0x02013c (0xd8dc bytes) 000564 1733164378.956512 mapread 0x030499 thru 0x030c57 (0x7bf bytes) 000565 1733164378.956942 mapread 0x015135 thru 0x01b9c3 (0x688f bytes) 000566 1733164378.957509 trunc from 0x040000 to 0x0343b7 (0xbc48 bytes) 000567 1733164378.964119 mapwrite 0x0114f2 thru 0x0181ea (0x6cf9 bytes) 000569 1733164378.976264 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000571 1733164379.042039 MIRROR_RESYNC 5 mirrors 000573 1733164379.362937 fallocate 0x00e708 thru 0x0157dd (0x70d6 bytes) 000574 1733164379.374424 fallocate 0x010da9 thru 0x017a9f (0x6cf7 bytes) 000575 1733164379.376471 mapread 0x032a75 thru 0x0343b7 (0x1943 bytes) 000576 1733164379.383845 trunc from 0x0343b8 to 0x021eef (0x124c8 bytes) 000578 1733164379.393635 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000579 1733164379.464183 MIRROR_EXTEND to 5 mirrors 000582 1733164379.662015 mapread 0x010fb4 thru 0x020e40 (0xfe8d bytes) 000586 1733164379.689689 punch from 0x008144 to 0x013dc5 (0xbc82 bytes) 000588 1733164379.704589 write 0x007e89 thru 0x00f5be (0x7736 bytes) 000589 1733164379.712320 mapread 0x01c3fa thru 0x021eef (0x5af6 bytes) 000590 1733164379.718480 mapwrite 0x019a24 thru 0x026967 (0xcf44 bytes) 000592 1733164379.748720 punch from 0x001a6d to 0x004ddb (0x336f bytes) 000593 1733164379.752447 mapread 0x0182a4 thru 0x01bf53 (0x3cb0 bytes) 000595 1733164379.753640 punch from 0x0097fd to 0x0135f1 (0x9df5 bytes) 000596 1733164379.762328 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000599 1733164379.841611 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000600 1733164379.947392 mapwrite 0x0151b7 thru 0x019f2e (0x4d78 bytes) 000601 1733164379.960580 mapwrite 0x00a7d2 thru 0x011ca0 (0x74cf bytes) 000602 1733164379.973536 fallocate 0x02ba60 thru 0x02da4c (0x1fed bytes) 000605 1733164379.978181 trunc from 0x02da4d to 0x035163 (0x7717 bytes) 000606 1733164379.989532 write 0x01fd1c thru 0x0253a9 (0x568e bytes) 000607 1733164379.993802 read 0x02e510 thru 0x035163 (0x6c54 bytes) 000608 1733164380.001960 mapread 0x0171fd thru 0x025a98 (0xe89c bytes) 000609 1733164380.005570 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000610 1733164380.080305 fallocate 0x0220c0 thru 0x030dfb (0xed3c bytes) 000611 1733164380.085812 punch from 0x02dc32 to 0x035163 (0x7532 bytes) 000612 1733164380.088159 mapread 0x03072c thru 0x035163 (0x4a38 bytes) 000613 1733164380.096978 mapwrite 0x02399c thru 0x0300ca (0xc72f bytes) 000614 1733164380.112400 punch from 0x003479 to 0x00bb5f (0x86e7 bytes) 000615 1733164380.115061 fallocate 0x01a24f thru 0x022176 (0x7f28 bytes) 000616 1733164380.116811 read 0x00fb01 thru 0x0146f2 (0x4bf2 bytes) 000617 1733164380.120505 punch from 0x034ff5 to 0x035163 (0x16f bytes) 000618 1733164380.122793 mapwrite 0x02d24e thru 0x03a4d3 (0xd286 bytes) 000619 1733164380.145837 write 0x00dc39 thru 0x01ad75 (0xd13d bytes) 000620 1733164380.154970 fallocate 0x025f04 thru 0x0268cb (0x9c8 bytes) 000621 1733164380.157956 mapread 0x003f78 thru 0x009f52 (0x5fdb bytes) 000622 1733164380.162710 read 0x02c642 thru 0x037c65 (0xb624 bytes) 000623 1733164380.163174 mapwrite 0x0357f9 thru 0x03ed48 (0x9550 bytes) 000624 1733164380.189224 mapwrite 0x0223c7 thru 0x02ac3f (0x8879 bytes) 000625 1733164380.198417 write 0x00af35 thru 0x00e87f (0x394b bytes) 000626 1733164380.201622 MIRROR_RESYNC 2 mirrors 000627 1733164380.355099 trunc from 0x03ed49 to 0x003d90 (0x3afb8 bytes) 000628 1733164380.378512 mapwrite 0x014aa2 thru 0x017b7b (0x30da bytes) 000629 1733164380.401259 trunc from 0x017b7c to 0x020dc7 (0x924c bytes) 000630 1733164380.408686 read 0x00382a thru 0x012e2d (0xf604 bytes) 000631 1733164380.413352 write 0x0007cf thru 0x008692 (0x7ec4 bytes) 000632 1733164380.420884 MIRROR_RESYNC 2 mirrors 000633 1733164380.569985 MIRROR_EXTEND to 3 mirrors 000635 1733164380.743968 fallocate 0x004a9a thru 0x0122d1 (0xd838 bytes) 000637 1733164380.761608 fallocate 0x0150d7 thru 0x01adb3 (0x5cdd bytes) 000640 1733164380.765250 fallocate 0x0115f8 thru 0x019c62 (0x866b bytes) 000641 1733164380.768230 mapwrite 0x026b24 thru 0x02e0a7 (0x7584 bytes) 000642 1733164380.809640 MIRROR_RESYNC 3 mirrors 000643 1733164380.996687 mapwrite 0x02b322 thru 0x02c2b3 (0xf92 bytes) 000644 1733164381.036469 read 0x004e20 thru 0x009fb8 (0x5199 bytes) 000647 1733164381.040976 trunc from 0x02e0a8 to 0x010456 (0x1dc51 bytes) 000648 1733164381.057943 MIRROR_EXTEND to 4 mirrors 000649 1733164381.200624 write 0x0369d0 thru 0x03b248 (0x4879 bytes) 000650 1733164381.222508 read 0x0206fc thru 0x02f078 (0xe97d bytes) 000653 1733164381.230462 MIRROR_RESYNC 4 mirrors 000654 1733164381.461123 read 0x004d16 thru 0x00ee63 (0xa14e bytes) 000657 1733164381.464956 trunc from 0x03b249 to 0x03f96e (0x4726 bytes) 000658 1733164381.483774 read 0x0310ec thru 0x036b33 (0x5a48 bytes) 000659 1733164381.489458 mapread 0x011f60 thru 0x0161fc (0x429d bytes) 000661 1733164381.490544 mapwrite 0x03df46 thru 0x03ffff (0x20ba bytes) 000663 1733164381.507771 write 0x005a68 thru 0x0112f7 (0xb890 bytes) 000665 1733164381.508988 mapread 0x017faa thru 0x0261e8 (0xe23f bytes) 000666 1733164381.509677 read 0x021bfe thru 0x02f914 (0xdd17 bytes) 000667 1733164381.510052 mapread 0x0236a4 thru 0x031fd0 (0xe92d bytes) 000668 1733164381.510638 punch from 0x02c58a to 0x02e60d (0x2084 bytes) 000669 1733164381.513903 mapwrite 0x014280 thru 0x01e6a0 (0xa421 bytes) 000671 1733164381.526033 mapwrite 0x01f83c thru 0x0221f5 (0x29ba bytes) 000672 1733164381.534195 punch from 0x034680 to 0x03c2e7 (0x7c68 bytes) 000673 1733164381.537083 trunc from 0x040000 to 0x02ea07 (0x115f8 bytes) 000674 1733164381.544226 mapread 0x00362a thru 0x00bb2a (0x8501 bytes) 000675 1733164381.546746 MIRROR_RESYNC 4 mirrors 000676 1733164381.824738 punch from 0x00e08f to 0x01c507 (0xe479 bytes) 000677 1733164381.839688 read 0x01bed7 thru 0x0246f9 (0x8823 bytes) 000678 1733164381.844848 mapread 0x00781c thru 0x012239 (0xaa1e bytes) 000679 1733164381.859266 read 0x02ae72 thru 0x02ca59 (0x1be8 bytes) 000680 1733164381.864695 trunc from 0x02ea08 to 0x0397c1 (0xadba bytes) 000681 1733164381.879208 mapwrite 0x01e14d thru 0x02990e (0xb7c2 bytes) 000682 1733164381.894063 trunc from 0x0397c2 to 0x02b121 (0xe6a0 bytes) 000683 1733164381.903988 mapwrite 0x0340da thru 0x03723d (0x3164 bytes) 000684 1733164381.927232 write 0x0029ed thru 0x010a53 (0xe067 bytes) 000685 1733164381.929205 mapread 0x0207e2 thru 0x024eed (0x470c bytes) 000686 1733164381.929928 mapread 0x024f74 thru 0x025e9a (0xf27 bytes) 000690 1733164381.931971 trunc from 0x03723e to 0x00d9e5 (0x29858 bytes) 000691 1733164381.941681 write 0x00a2e7 thru 0x012d2b (0x8a45 bytes) 000692 1733164381.942623 punch from 0x002d00 to 0x008108 (0x5409 bytes) 000693 1733164381.953124 mapwrite 0x00ece1 thru 0x012ee9 (0x4209 bytes) 000694 1733164381.973552 punch from 0x009b30 to 0x012ee9 (0x93ba bytes) 000695 1733164381.976153 mapread 0x00bac1 thru 0x012ee9 (0x7429 bytes) 000696 1733164381.993301 write 0x01fb2d thru 0x02c30c (0xc7e0 bytes) 000698 1733164381.994734 punch from 0x000988 to 0x00f1f6 (0xe86f bytes) 000700 1733164381.998255 mapread 0x02a83d thru 0x02c30c (0x1ad0 bytes) 000701 1733164381.999021 fallocate 0x00c595 thru 0x014ec0 (0x892c bytes) 000702 1733164382.002481 write 0x00228b thru 0x00e5f0 (0xc366 bytes) 000703 1733164382.007238 fallocate 0x019566 thru 0x022c3d (0x96d8 bytes) 000704 1733164382.010689 write 0x027ea8 thru 0x0333f3 (0xb54c bytes) 000705 1733164382.011776 mapwrite 0x0187dd thru 0x0224f1 (0x9d15 bytes) 000707 1733164382.040391 mapread 0x01c9e2 thru 0x029b88 (0xd1a7 bytes) 000708 1733164382.041032 fallocate 0x0306ab thru 0x03200f (0x1965 bytes) 000709 1733164382.044087 trunc from 0x0333f4 to 0x01befb (0x174f8 bytes) 000710 1733164382.051797 trunc from 0x01befc to 0x025389 (0x948e bytes) 000711 1733164382.061304 mapread 0x00ed12 thru 0x01e459 (0xf748 bytes) 000712 1733164382.086064 punch from 0x017857 to 0x020896 (0x9040 bytes) 000713 1733164382.089967 punch from 0x018c57 to 0x01cdf9 (0x41a3 bytes) 000714 1733164382.092270 mapwrite 0x03c365 thru 0x03d5e1 (0x127d bytes) 000715 1733164382.115600 mapread 0x023e5a thru 0x02bf3e (0x80e5 bytes) 000716 1733164382.132529 mapread 0x00d785 thru 0x00e042 (0x8be bytes) 000718 1733164382.133408 mapread 0x027dc2 thru 0x02bf4f (0x418e bytes) 000719 1733164382.133935 read 0x023d72 thru 0x02ea88 (0xad17 bytes) 000721 1733164382.140281 trunc from 0x03d5e2 to 0x01da4a (0x1fb97 bytes) 000722 1733164382.147733 trunc from 0x01da4b to 0x02a885 (0xce3b bytes) 000723 1733164382.158339 trunc from 0x02a886 to 0x018795 (0x120f0 bytes) 000724 1733164382.165966 punch from 0x00f1df to 0x0125fe (0x3420 bytes) 000725 1733164382.170923 trunc from 0x018796 to 0x020711 (0x7f7c bytes) 000726 1733164382.179606 trunc from 0x020712 to 0x021c2c (0x151b bytes) 000727 1733164382.187508 mapwrite 0x01ee9e thru 0x028457 (0x95ba bytes) 000729 1733164382.245371 trunc from 0x028458 to 0x010999 (0x17abe bytes) 000730 1733164382.254618 mapread 0x005f7d thru 0x008338 (0x23bc bytes) 000731 1733164382.255128 mapwrite 0x02d0f6 thru 0x02d7f9 (0x704 bytes) 000732 1733164382.274110 fallocate 0x00c866 thru 0x01aacb (0xe266 bytes) 000733 1733164382.276783 trunc from 0x02d7fa to 0x03d88d (0x10094 bytes) 000734 1733164382.286267 fallocate 0x01660f thru 0x01829c (0x1c8e bytes) 000735 1733164382.289071 trunc from 0x03d88e to 0x004a88 (0x38e05 bytes) 000736 1733164382.299786 MIRROR_EXTEND to 5 mirrors 000737 1733164382.490344 MIRROR_EXTEND to 6 mirrors 000738 1733164382.680779 trunc from 0x004a89 to 0x00c80c (0x7d84 bytes) 000739 1733164382.706626 mapwrite 0x02b7f1 thru 0x02f10b (0x391b bytes) 000740 1733164382.730285 write 0x002a45 thru 0x009c91 (0x724d bytes) 000741 1733164382.735544 write 0x02c90a thru 0x032001 (0x56f8 bytes) 000742 1733164382.736424 write 0x01d189 thru 0x027530 (0xa3a8 bytes) 000743 1733164382.742718 read 0x01e692 thru 0x0259d6 (0x7345 bytes) 000744 1733164382.745861 MIRROR_RESYNC 6 mirrors 000745 1733164383.281229 mapread 0x0116c3 thru 0x0207d8 (0xf116 bytes) 000746 1733164383.294191 MIRROR_RESYNC 6 mirrors 000747 1733164383.379322 MIRROR_RESYNC 6 mirrors 000749 1733164383.461103 mapwrite 0x026e98 thru 0x030fc6 (0xa12f bytes) 000751 1733164383.511834 mapread 0x01c441 thru 0x0268ed (0xa4ad bytes) 000752 1733164383.518382 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000753 1733164383.597913 punch from 0x011c54 to 0x016d6e (0x511b bytes) 000754 1733164383.599784 mapwrite 0x025e1c thru 0x0277b9 (0x199e bytes) 000755 1733164383.615900 write 0x018ddb thru 0x02750d (0xe733 bytes) 000756 1733164383.624760 MIRROR_EXTEND to 6 mirrors 000757 1733164383.837377 punch from 0x02b21c to 0x032001 (0x6de6 bytes) 000758 1733164383.854533 MIRROR_EXTEND to 7 mirrors 000759 1733164384.047012 mapread 0x025f45 thru 0x02d465 (0x7521 bytes) 000760 1733164384.059821 mapread 0x02b554 thru 0x030562 (0x500f bytes) 000762 1733164384.071993 punch from 0x03131b to 0x032001 (0xce7 bytes) 000764 1733164384.086801 trunc from 0x032002 to 0x00a5fb (0x27a06 bytes) 000765 1733164384.102614 fallocate 0x01c045 thru 0x0209a3 (0x495f bytes) 000766 1733164384.104979 write 0x017ace thru 0x025890 (0xddc3 bytes) 000767 1733164384.106269 write 0x01d9ec thru 0x029429 (0xba3e bytes) 000769 1733164384.107739 read 0x019592 thru 0x01c8d9 (0x3348 bytes) 000770 1733164384.109173 trunc from 0x02942a to 0x013151 (0x162d8 bytes) 000774 1733164384.126037 mapread 0x005b8a thru 0x00cbd4 (0x704b bytes) 000776 1733164384.132342 fallocate 0x0022b6 thru 0x0114ba (0xf205 bytes) 000777 1733164384.137063 trunc from 0x013152 to 0x03384d (0x206fc bytes) 000778 1733164384.147926 write 0x00a539 thru 0x010c98 (0x6760 bytes) 000779 1733164384.151793 MIRROR_EXTEND to 8 mirrors 000780 1733164384.391362 mapwrite 0x017fab thru 0x01b732 (0x3788 bytes) 000781 1733164384.441049 MIRROR_EXTEND to 9 mirrors 000782 1733164384.675719 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000783 1733164384.785265 read 0x008af0 thru 0x00aac2 (0x1fd3 bytes) 000784 1733164384.788479 mapwrite 0x0169e4 thru 0x01c477 (0x5a94 bytes) 000786 1733164384.848548 read 0x020f38 thru 0x025502 (0x45cb bytes) 000787 1733164384.853649 punch from 0x011244 to 0x012554 (0x1311 bytes) 000788 1733164384.856903 read 0x008f62 thru 0x00eaed (0x5b8c bytes) 000789 1733164384.857389 punch from 0x0189db to 0x022714 (0x9d3a bytes) 000791 1733164384.860783 mapread 0x030ed0 thru 0x03384d (0x297e bytes) 000792 1733164384.861593 read 0x01dd68 thru 0x02bf1b (0xe1b4 bytes) 000793 1733164384.868000 MIRROR_EXTEND to 9 mirrors 000794 1733164385.103416 punch from 0x0056ef to 0x0061e4 (0xaf6 bytes) 000795 1733164385.121626 read 0x00e4b0 thru 0x015e34 (0x7985 bytes) 000796 1733164385.127204 MIRROR_EXTEND to 10 mirrors 000797 1733164385.361678 mapread 0x02d5b4 thru 0x031778 (0x41c5 bytes) 000798 1733164385.376206 fallocate 0x02d0f5 thru 0x03206f (0x4f7b bytes) 000800 1733164385.392447 write 0x006044 thru 0x00c431 (0x63ee bytes) 000801 1733164385.401404 mapread 0x01c459 thru 0x023002 (0x6baa bytes) 000802 1733164385.405239 fallocate 0x02e87d thru 0x03a696 (0xbe1a bytes) 000803 1733164385.407674 trunc from 0x03a697 to 0x036b91 (0x3b05 bytes) 000804 1733164385.416390 punch from 0x009d70 to 0x00c53b (0x27cc bytes) 000805 1733164385.423856 mapwrite 0x00de03 thru 0x01013b (0x2339 bytes) 000806 1733164385.436453 trunc from 0x036b92 to 0x03cd54 (0x61c3 bytes) 000808 1733164385.445092 mapread 0x02346f thru 0x02532e (0x1ec0 bytes) 000809 1733164385.452281 MIRROR_RESYNC 10 mirrors 000810 1733164386.502179 mapread 0x023524 thru 0x02b871 (0x834e bytes) 000811 1733164386.512840 MIRROR_RESYNC 10 mirrors 000812 1733164386.619706 mapwrite 0x00130b thru 0x0047e4 (0x34da bytes) 000813 1733164386.668868 mapwrite 0x03ace3 thru 0x03ffff (0x531d bytes) 000814 1733164386.698967 mapwrite 0x007763 thru 0x00dfa2 (0x6840 bytes) 000815 1733164386.713851 write 0x012e79 thru 0x014961 (0x1ae9 bytes) 000816 1733164386.719326 write 0x005fbf thru 0x01157e (0xb5c0 bytes) 000817 1733164386.720612 fallocate 0x00f894 thru 0x0121ed (0x295a bytes) 000819 1733164386.725080 read 0x0274f9 thru 0x036bf9 (0xf701 bytes) 000820 1733164386.732416 fallocate 0x02a2e5 thru 0x02d6c0 (0x33dc bytes) 000822 1733164386.737309 write 0x011a69 thru 0x01c9dd (0xaf75 bytes) 000823 1733164386.741397 fallocate 0x0085b5 thru 0x00d55b (0x4fa7 bytes) 000824 1733164386.745038 write 0x03b35e thru 0x03eead (0x3b50 bytes) 000825 1733164386.745768 mapread 0x02fbd9 thru 0x0370b7 (0x74df bytes) 000826 1733164386.746378 fallocate 0x0235ad thru 0x024bcb (0x161f bytes) 000828 1733164386.751504 trunc from 0x040000 to 0x0186d8 (0x27927 bytes) 000829 1733164386.763029 MIRROR_RESYNC 10 mirrors 000830 1733164387.104715 MIRROR_RESYNC 10 mirrors 000831 1733164387.199874 read 0x00190b thru 0x00d073 (0xb769 bytes) 000834 1733164387.205744 fallocate 0x01e277 thru 0x0244c9 (0x6253 bytes) 000836 1733164387.225405 fallocate 0x001641 thru 0x00939a (0x7d5a bytes) 000837 1733164387.232507 MIRROR_EXTEND to 11 mirrors 000838 1733164387.435840 MIRROR_EXTEND to 12 mirrors 000839 1733164387.591035 mapwrite 0x012870 thru 0x01b04b (0x87dc bytes) 000841 1733164387.630653 mapread 0x008867 thru 0x015767 (0xcf01 bytes) 000843 1733164387.639828 fallocate 0x024176 thru 0x02554b (0x13d6 bytes) 000844 1733164387.645956 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000845 1733164387.743738 trunc from 0x02554c to 0x03297c (0xd431 bytes) 000846 1733164387.753364 trunc from 0x03297d to 0x02c83b (0x6141 bytes) 000847 1733164387.759607 mapread 0x00315b thru 0x005295 (0x213b bytes) 000848 1733164387.772823 mapread 0x002b9a thru 0x0092fd (0x6764 bytes) 000850 1733164387.774617 read 0x006545 thru 0x00d26b (0x6d27 bytes) 000851 1733164387.775087 mapread 0x0175b0 thru 0x01b18d (0x3bde bytes) 000852 1733164387.781714 write 0x00b8a1 thru 0x00cd63 (0x14c3 bytes) 000854 1733164387.782849 mapread 0x020331 thru 0x026d69 (0x6a39 bytes) 000856 1733164387.790951 write 0x00d8be thru 0x018573 (0xacb6 bytes) 000858 1733164387.793661 write 0x016886 thru 0x01dae9 (0x7264 bytes) 000859 1733164387.794394 mapwrite 0x00535e thru 0x00a236 (0x4ed9 bytes) 000862 1733164387.808865 punch from 0x00866c to 0x0139d0 (0xb365 bytes) 000863 1733164387.812254 write 0x001e1f thru 0x005d73 (0x3f55 bytes) 000864 1733164387.813239 mapwrite 0x03c30f thru 0x03ffff (0x3cf1 bytes) 000866 1733164387.832644 mapwrite 0x01ca79 thru 0x0248ce (0x7e56 bytes) 000867 1733164387.844002 mapread 0x01d76e thru 0x020f6f (0x3802 bytes) 000868 1733164387.844666 trunc from 0x040000 to 0x038d03 (0x72fc bytes) 000869 1733164387.852782 trunc from 0x038d04 to 0x038f67 (0x264 bytes) 000870 1733164387.858915 punch from 0x01d1bc to 0x020948 (0x378d bytes) 000871 1733164387.862094 write 0x008081 thru 0x00ca9f (0x4a1f bytes) 000872 1733164387.866424 mapread 0x022a96 thru 0x02ccb5 (0xa220 bytes) 000873 1733164387.867376 mapread 0x02aa61 thru 0x034372 (0x9912 bytes) 000874 1733164387.870857 fallocate 0x00faaf thru 0x01707c (0x75ce bytes) 000876 1733164387.873488 read 0x0110aa thru 0x0141cb (0x3122 bytes) 000877 1733164387.877908 fallocate 0x01762d thru 0x019b5c (0x2530 bytes) 000878 1733164387.880937 write 0x02687b thru 0x02b39b (0x4b21 bytes) 000880 1733164387.881923 fallocate 0x01f0d5 thru 0x020fdd (0x1f09 bytes) 000881 1733164387.884270 punch from 0x01fc74 to 0x0223d5 (0x2762 bytes) 000882 1733164387.886420 trunc from 0x038f68 to 0x002ef7 (0x36070 bytes) 000883 1733164387.892936 read 0x002ea7 thru 0x002ef7 (0x51 bytes) 000884 1733164387.893533 read 0x001ac3 thru 0x002ef7 (0x1435 bytes) 000886 1733164387.894258 write 0x031f48 thru 0x03acce (0x8d87 bytes) 000887 1733164387.895236 mapwrite 0x011796 thru 0x0206d1 (0xef3c bytes) 000888 1733164387.912406 trunc from 0x03accf to 0x0252a6 (0x15a28 bytes) 000889 1733164387.919184 punch from 0x0047b3 to 0x013989 (0xf1d7 bytes) 000890 1733164387.922097 write 0x01677f thru 0x02130d (0xab8f bytes) 000891 1733164387.923171 punch from 0x02323f to 0x0252a6 (0x2068 bytes) 000892 1733164387.925427 read 0x00c7cb thru 0x01ae19 (0xe64f bytes) 000893 1733164387.931922 MIRROR_EXTEND to 12 mirrors 000894 1733164388.181177 fallocate 0x0012e3 thru 0x006c22 (0x5940 bytes) 000895 1733164388.196065 read 0x01364c thru 0x01d3e3 (0x9d98 bytes) 000896 1733164388.203703 MIRROR_RESYNC 12 mirrors 000897 1733164389.003016 read 0x021c87 thru 0x0252a6 (0x3620 bytes) 000898 1733164389.005097 write 0x01f53e thru 0x0225b7 (0x307a bytes) 000899 1733164389.029864 mapwrite 0x03ecce thru 0x03f41c (0x74f bytes) 000900 1733164389.059118 trunc from 0x03f41d to 0x037a14 (0x7a08 bytes) 000901 1733164389.071649 read 0x0183db thru 0x027fa3 (0xfbc9 bytes) 000902 1733164389.076794 mapread 0x0347b2 thru 0x037a14 (0x3263 bytes) 000903 1733164389.077592 punch from 0x0262db to 0x0304c6 (0xa1ec bytes) 000904 1733164389.080255 fallocate 0x03af81 thru 0x03ffff (0x507f bytes) 000905 1733164389.082598 trunc from 0x037a15 to 0x03d5dd (0x5bc9 bytes) 000906 1733164389.087832 read 0x007c08 thru 0x00e111 (0x650a bytes) 000907 1733164389.088286 punch from 0x01866d to 0x01c528 (0x3ebc bytes) 000910 1733164389.090891 mapread 0x0389bb thru 0x03b925 (0x2f6b bytes) 000913 1733164389.098303 MIRROR_EXTEND to 13 mirrors 000915 1733164389.335981 fallocate 0x03d5fc thru 0x03ffff (0x2a04 bytes) 000916 1733164389.355303 write 0x02920d thru 0x0326e6 (0x94da bytes) 000917 1733164389.368301 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000918 1733164389.441489 write 0x03673f thru 0x03b0b1 (0x4973 bytes) 000919 1733164389.448915 punch from 0x01eeeb to 0x021f19 (0x302f bytes) 000920 1733164389.455858 mapread 0x00dbc0 thru 0x00f967 (0x1da8 bytes) 000921 1733164389.459812 mapread 0x025690 thru 0x02dda3 (0x8714 bytes) 000922 1733164389.468394 write 0x002487 thru 0x0032d8 (0xe52 bytes) 000923 1733164389.473334 punch from 0x0187b1 to 0x019625 (0xe75 bytes) 000924 1733164389.475442 fallocate 0x0316ac thru 0x03d039 (0xb98e bytes) 000925 1733164389.478007 write 0x031aea thru 0x031f0a (0x421 bytes) 000926 1733164389.478603 fallocate 0x032aa5 thru 0x03df3a (0xb496 bytes) 000927 1733164389.483483 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000928 1733164389.581124 MIRROR_RESYNC 11 mirrors 000929 1733164390.839066 trunc from 0x040000 to 0x006e0d (0x391f2 bytes) 000930 1733164390.861290 mapread 0x0031d4 thru 0x004743 (0x1570 bytes) 000931 1733164390.867063 fallocate 0x00c1b1 thru 0x012ce9 (0x6b39 bytes) 000932 1733164390.870032 read 0x003695 thru 0x006e0d (0x3779 bytes) 000933 1733164390.874482 punch from 0x0004d6 to 0x006e0d (0x6938 bytes) 000934 1733164390.879066 fallocate 0x0101a2 thru 0x0186b4 (0x8513 bytes) 000935 1733164390.882278 punch from 0x003d25 to 0x0065f7 (0x28d3 bytes) 000936 1733164390.885515 fallocate 0x03aad9 thru 0x03ffff (0x5527 bytes) 000939 1733164390.889653 trunc from 0x006e0e to 0x01e583 (0x17776 bytes) 000940 1733164390.904081 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000941 1733164390.996657 read 0x003b98 thru 0x0138ea (0xfd53 bytes) 000942 1733164391.001394 trunc from 0x01e584 to 0x03f96d (0x213ea bytes) 000943 1733164391.012520 mapwrite 0x031ad7 thru 0x03e22c (0xc756 bytes) 000945 1733164391.040497 trunc from 0x03f96e to 0x014c66 (0x2ad07 bytes) 000946 1733164391.057201 write 0x008cf0 thru 0x00f976 (0x6c87 bytes) 000950 1733164391.060972 trunc from 0x014c67 to 0x03289d (0x1dc37 bytes) 000952 1733164391.073705 write 0x0337b6 thru 0x035aad (0x22f8 bytes) 000953 1733164391.075440 mapread 0x017a4a thru 0x018d29 (0x12e0 bytes) 000954 1733164391.080785 fallocate 0x022271 thru 0x026ac9 (0x4859 bytes) 000955 1733164391.083234 mapread 0x0094d7 thru 0x018d2f (0xf859 bytes) 000956 1733164391.083865 fallocate 0x01199c thru 0x01e040 (0xc6a5 bytes) 000957 1733164391.086738 read 0x001680 thru 0x007426 (0x5da7 bytes) 000958 1733164391.093070 MIRROR_RESYNC 10 mirrors 000964 1733164392.036852 trunc from 0x035aae to 0x015f97 (0x1fb16 bytes) 000965 1733164392.060492 MIRROR_RESYNC 10 mirrors 000967 1733164392.867231 fallocate 0x03c2be thru 0x03ffff (0x3d42 bytes) 000968 1733164392.887940 MIRROR_RESYNC 10 mirrors 000969 1733164393.565633 MIRROR_EXTEND to 11 mirrors 000970 1733164393.818526 mapread 0x00c84d thru 0x015664 (0x8e18 bytes) 000971 1733164393.825526 write 0x03b7f1 thru 0x03cb41 (0x1351 bytes) 000973 1733164393.842588 fallocate 0x00d2a7 thru 0x00f393 (0x20ed bytes) 000974 1733164393.845053 write 0x03e619 thru 0x03ffff (0x19e7 bytes) 000975 1733164393.848037 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000977 1733164393.950799 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000978 1733164394.026127 read 0x02ab8f thru 0x02b099 (0x50b bytes) 000980 1733164394.028953 read 0x02b7cc thru 0x037170 (0xb9a5 bytes) 000981 1733164394.035055 read 0x03886b thru 0x03f41c (0x6bb2 bytes) 000982 1733164394.035572 mapwrite 0x02c141 thru 0x036d46 (0xac06 bytes) 000984 1733164394.047987 fallocate 0x0365e3 thru 0x03ffff (0x9a1d bytes) 000985 1733164394.050444 fallocate 0x01cf33 thru 0x01d021 (0xef bytes) 000987 1733164394.054557 read 0x01cc5d thru 0x01e6b8 (0x1a5c bytes) 000989 1733164394.055348 mapwrite 0x02de7e thru 0x038e30 (0xafb3 bytes) 000990 1733164394.067287 mapread 0x00ec1e thru 0x01ae6b (0xc24e bytes) 000993 1733164394.068528 mapwrite 0x022410 thru 0x02ceb2 (0xaaa3 bytes) 000994 1733164394.077968 fallocate 0x010fb4 thru 0x01c74c (0xb799 bytes) 000996 1733164394.080816 trunc from 0x040000 to 0x0330fe (0xcf01 bytes) 000997 1733164394.088986 mapwrite 0x00ee9e thru 0x015d63 (0x6ec6 bytes) 000999 1733164394.100777 trunc from 0x0330ff to 0x01999e (0x19760 bytes) 001000 1733164394.111296 fallocate 0x03c656 thru 0x03ffff (0x39aa bytes) All operations completed A-OK! PASS 70a (27s) == sanity-flr test 200a: stress test ===================== 13:33:15 (1733164395) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre2 Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1909728 .. Extending file size to 856288 .. failed Extending file size to 5152640 .. Extending file size to 7407680 .. Extending file size to 8718848 .. Extending file size to 7754912 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7840576 .. Extending file size to 8581824 .. Extending file size to 9238784 .. Extending file size to 9241568 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9326400 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 3380 3381 3382 3384 3387 192.168.201.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre2 (opts:) 192.168.201.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:34:20 (1733164460) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre2 Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7863296 .. Extending file size to 3983680 .. done Extending file size to 4756896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6847328 .. Extending file size to 7709888 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8964032 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8737504 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9038048 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8999168 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9094016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9176128 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9275168 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 6064 6065 6067 6069 6071 failed 192.168.201.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre2 (opts:) 192.168.201.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:35:24 (1733164524) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3244 1284444 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15764 3591256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21728 3585292 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37492 7176548 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3244 1284444 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2156 1285532 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15764 3591256 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21728 3585292 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37492 7176548 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0355697 s, 59.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0476532 s, 44.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:35:29 (1733164529) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0314944 s, 66.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 13:35:31 (1733164531) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7e5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa40:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa21:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:35:33 (1733164533) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0239115 s, 87.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:35:35 (1733164535) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263404 s, 79.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa41:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa42:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa22:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa41:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa42:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa22:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa23:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:35:37 (1733164537) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0221739 s, 94.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa24:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa24:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa25:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:36:00 (1733164560) Waiting for MDT destroys to complete oleg125-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg125-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg125-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163866 s, 64.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:36:32 (1733164592) Waiting for MDT destroys to complete oleg125-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg125-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg125-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0251375 s, 83.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:37:07 (1733164627) Waiting for MDT destroys to complete oleg125-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg125-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg125-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg125-server: Pool lustre.test_204f created Adding targets to pool oleg125-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg125-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0230906 s, 90.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg125-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg125-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg125-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:37:55 (1733164675) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:37:56 (1733164676) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:37:58 (1733164678) oleg125-server: Pool lustre.test_206 created Adding targets to pool oleg125-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg125-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg125-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg125-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg125-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:38:13 (1733164693) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019538 s, 53.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119091 s, 88.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3eb421e7 Mirror 3: 0x3eb421e7 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:38:15 (1733164695) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:38:16 (1733164696) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:38:16 (1733164696) Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00933878 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xea0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xea0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xea0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB 192.168.201.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg125-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:38:30 (1733164710) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0022275 s, 3.7 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:38:31 (1733164711) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136531 s, 76.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x802 lmm_fid: [0x200000405:0x802:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x802 lmm_fid: [0x200000405:0x802:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4e:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613186 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-02 13:38:32.000000000 -0500 Modify: 2024-12-02 13:38:32.000000000 -0500 Change: 2024-12-02 13:38:32.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:38:33 (1733164713) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104128 s, 101 MB/s oleg125-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg125-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg125-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 oleg125-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 oleg125-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg125-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg125-server: O/280000401/d15/a4f: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test complete, duration 793 sec ============ 13:38:44 (1733164724) === sanity-flr: start cleanup 13:38:44 (1733164724) === === sanity-flr: finish cleanup 13:38:46 (1733164726) ===