-----============= acceptance-small: sanity-flr ============----- Tue Feb 25 07:00:18 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg252-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:00:23 (1740484823) === oleg252-client.virtnet: executing check_config_client /mnt/lustre oleg252-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg252-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b616b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b616b000.idle_timeout=debug disable quota as required oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:00:31 (1740484831) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanity-flr test 0a: lfs mirror create with -N option == 07:00:32 (1740484832) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:00:35 (1740484835) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:00:36 (1740484836) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:00:37 (1740484837) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:00:41 (1740484841) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:00:42 (1740484842) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 07:00:42 (1740484842) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b616b000.stats=clear osc.lustre-OST0001-osc-ffff8800b616b000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:00:44 (1740484844) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:00:46 (1740484846) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:00:47 (1740484847) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:00:48 (1740484848) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 07:00:50 (1740484850) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:00:51 (1740484851) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:00:53 (1740484853) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:00:55 (1740484855) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:00:57 (1740484857) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:00:59 (1740484859) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.357915 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227637 s, 46.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:01:01 (1740484861) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.0533534 s, 138 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:01:03 (1740484863) Stopping /mnt/lustre-ost1 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg252-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg252-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:01:29 (1740484889) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.114756 s, 64.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg252-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:01:37 (1740484897) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg252-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg252-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:01:58 (1740484918) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.800623 s, 65.5 MB/s -rw-r--r-- 1 root root 52428800 Feb 25 07:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.257s user 0m0.001s sys 0m0.109s Stopping /mnt/lustre-ost1 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.246s user 0m0.000s sys 0m0.108s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg252-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg252-server oleg252-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.273s user 0m0.000s sys 0m0.120s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg252-server: oleg252-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg252-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b616b000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:02:19 (1740484939) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:02:20 (1740484940) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:02:21 (1740484941) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 07:02:21 (1740484941) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 07:02:23 (1740484943) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.126012 s, 99.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:02:26 (1740484946) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00968722 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00933762 s, 112 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:02:27 (1740484947) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:02:30 (1740484950) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000626787 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000640881 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000818698 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000790587 s, 1.3 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 07:02:32 (1740484952) create test files with size 15748559 .. b2628402000f84087f81f9bd7608a209 - f525f1a2ae23ac1e82148548e353894f - eba28ca97b3ffbdde76f81aeb7a763d8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg252-client.virtnet /mnt/lustre (opts:) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 07:02:36 (1740484956) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:02:38 (1740484958) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:02:40 (1740484960) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159082 s, 65.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00861901 s, 122 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 07:02:41 (1740484961) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0225787 s, 92.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0867176 s, 48.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0324103 s, 64.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 07:02:44 (1740484964) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 07:02:44 (1740484964) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:02:45 (1740484965) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096614 s, 109 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 07:02:47 (1740484967) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0677339 s, 46.4 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:02:49 (1740484969) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:02:51 (1740484971) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0630616 s, 166 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg252-client.virtnet /mnt/lustre (opts:) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:03:14 (1740484994) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012fe1e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0644653 s, 163 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg252-client.virtnet /mnt/lustre (opts:) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012bdbe800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:03:17 (1740484997) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:03:23 (1740485003) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:03:29 (1740485009) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:03:30 (1740485010) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0663308 s, 47.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0539934 s, 58.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 6ce65c00407df2f5e2c68ec01aeab889 - mirror 1 checksum: a9092eec6581f3826fb075894880b53a - mirror 2 checksum: 6ce65c00407df2f5e2c68ec01aeab889 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Feb 25 07:03:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Feb 25 07:03:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ce65c00407df2f5e2c68ec01aeab889 - mirror 1 checksum: a9092eec6581f3826fb075894880b53a - mirror 2 checksum: a9092eec6581f3826fb075894880b53a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Feb 25 07:03:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Feb 25 07:03:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:03:32 (1740485012) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914055 s, 115 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:03:33 (1740485013) Failing mds1 on oleg252-server Stopping /mnt/lustre-mds1 (opts:) on oleg252-server 07:03:41 (1740485021) shut down facet: mds1 facet_host: oleg252-server facet_failover_host: oleg252-server Failover mds1 to oleg252-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0000 07:03:53 (1740485033) targets are mounted 07:03:53 (1740485033) facet_failover done oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0093201 s, 113 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:04:01 (1740485041) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00780364 s, 2.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00392497 s, 5.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:04:03 (1740485043) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00940763 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00334828 s, 6.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:04:05 (1740485045) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00259497 s, 6.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:04:07 (1740485047) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00966243 s, 2.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:04:09 (1740485049) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00160367 s, 2.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00860565 s, 476 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:04:10 (1740485050) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:04:14 (1740485054) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:04:32 (1740485072) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:04:49 (1740485089) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:05:06 (1740485106) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 07:05:07 (1740485107) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26284 mirror split pid 26285 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26284 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26285 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 07:06:09 (1740485169) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5169 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3120b 000001 1740485169.645275 fallocate 0x02d719 thru 0x03120a (0x3af2 bytes) 000003 1740485169.647969 write 0x0068e5 thru 0x00e687 (0x7da3 bytes) fallocating to largest ever: 0x40000 000004 1740485169.648778 fallocate 0x03f92a thru 0x03ffff (0x6d6 bytes) 000006 1740485169.650331 trunc from 0x00e688 to 0x024b96 (0x1650f bytes) 000007 1740485169.654002 mapwrite 0x00f95f thru 0x01f1a2 (0xf844 bytes) 000008 1740485169.667236 punch from 0x002f9f to 0x00926a (0x62cc bytes) 000010 1740485169.669089 mapread 0x0082ec thru 0x016610 (0xe325 bytes) 000011 1740485169.671549 trunc from 0x024b97 to 0x039752 (0x14bbc bytes) 000013 1740485169.675092 punch from 0x0176e2 to 0x020fb4 (0x98d3 bytes) 000014 1740485169.676691 punch from 0x0306c9 to 0x031081 (0x9b9 bytes) 000015 1740485169.677939 trunc from 0x039753 to 0x0003c0 (0x39392 bytes) 000016 1740485169.681379 read 0x000038 thru 0x0003c0 (0x389 bytes) 000017 1740485169.682813 write 0x023c71 thru 0x02dbed (0x9f7d bytes) 000019 1740485169.684166 mapwrite 0x00f3d6 thru 0x015c7a (0x68a5 bytes) 000021 1740485169.692498 write 0x01337d thru 0x0163f1 (0x3075 bytes) 000022 1740485169.693138 fallocate 0x02de2b thru 0x036007 (0x81dd bytes) 000023 1740485169.694680 mapwrite 0x013372 thru 0x017c36 (0x48c5 bytes) 000024 1740485169.699313 mapwrite 0x013121 thru 0x014c0f (0x1aef bytes) 000025 1740485169.703530 write 0x0028c5 thru 0x003cd2 (0x140e bytes) 000028 1740485169.706860 fallocate 0x00995d thru 0x00f644 (0x5ce8 bytes) 000029 1740485169.708807 fallocate 0x01945f thru 0x02749c (0xe03e bytes) 000030 1740485169.710246 mapread 0x00a48a thru 0x00e075 (0x3bec bytes) 000031 1740485169.710830 write 0x029ee9 thru 0x030233 (0x634b bytes) 000033 1740485169.713271 punch from 0x023100 to 0x030de3 (0xdce4 bytes) 000035 1740485169.717246 write 0x0051eb thru 0x00e002 (0x8e18 bytes) 000037 1740485169.718609 read 0x004770 thru 0x008636 (0x3ec7 bytes) 000038 1740485169.719042 read 0x008a4c thru 0x017e6d (0xf422 bytes) 000039 1740485169.719468 mapread 0x00b9a4 thru 0x00d2db (0x1938 bytes) 000040 1740485169.721701 MIRROR_EXTEND to 2 mirrors 000041 1740485169.832704 write 0x014a31 thru 0x019ec8 (0x5498 bytes) 000042 1740485169.843839 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000046 1740485169.876617 write 0x03110b thru 0x0324b6 (0x13ac bytes) 000047 1740485169.880270 write 0x00c382 thru 0x013e04 (0x7a83 bytes) 000048 1740485169.883075 trunc from 0x036008 to 0x02fc72 (0x6395 bytes) 000049 1740485169.890149 fallocate 0x00d1bc thru 0x014261 (0x70a6 bytes) 000050 1740485169.891909 mapread 0x0181bd thru 0x025de9 (0xdc2d bytes) 000051 1740485169.897188 mapread 0x00853b thru 0x015398 (0xce5e bytes) 000052 1740485169.898961 read 0x00f65e thru 0x0179a3 (0x8346 bytes) 000054 1740485169.899994 read 0x02f94c thru 0x02fc72 (0x327 bytes) 000055 1740485169.902499 mapwrite 0x003bc7 thru 0x00db19 (0x9f53 bytes) 000056 1740485169.909143 mapwrite 0x0030fa thru 0x00a809 (0x7710 bytes) 000058 1740485169.914118 write 0x03bca5 thru 0x03ffff (0x435b bytes) 000059 1740485169.914910 read 0x03405d thru 0x03525a (0x11fe bytes) 000060 1740485169.916950 read 0x03e07b thru 0x03ffff (0x1f85 bytes) 000062 1740485169.917702 write 0x00ecc1 thru 0x01a119 (0xb459 bytes) 000063 1740485169.918359 read 0x01d735 thru 0x02694c (0x9218 bytes) 000064 1740485169.918727 mapwrite 0x03af8b thru 0x03ffff (0x5075 bytes) 000065 1740485169.923672 trunc from 0x040000 to 0x01c5f7 (0x23a08 bytes) 000066 1740485169.928109 fallocate 0x01b89d thru 0x01e92a (0x308e bytes) 000067 1740485169.929719 trunc from 0x01e92b to 0x024964 (0x603a bytes) 000068 1740485169.933405 fallocate 0x00ce8d thru 0x00e3ac (0x1520 bytes) 000069 1740485169.934959 fallocate 0x011da1 thru 0x01906f (0x72cf bytes) 000070 1740485169.936328 mapwrite 0x024861 thru 0x0332bd (0xea5d bytes) 000071 1740485169.949591 punch from 0x0160d3 to 0x01b409 (0x5337 bytes) 000073 1740485169.951811 write 0x033bf8 thru 0x03825f (0x4668 bytes) 000074 1740485169.952549 mapread 0x016fbc thru 0x0269d2 (0xfa17 bytes) 000075 1740485169.959389 punch from 0x00b72f to 0x00e11a (0x29ec bytes) 000077 1740485169.961141 read 0x0139ac thru 0x01c93f (0x8f94 bytes) 000078 1740485169.962445 MIRROR_EXTEND to 2 mirrors 000079 1740485170.082623 punch from 0x020c41 to 0x02e17e (0xd53e bytes) 000080 1740485170.090749 fallocate 0x018959 thru 0x023ffe (0xb6a6 bytes) 000082 1740485170.092685 read 0x00ba9c thru 0x01361d (0x7b82 bytes) 000083 1740485170.095165 mapwrite 0x021e4b thru 0x02873e (0x68f4 bytes) 000085 1740485170.105077 mapread 0x0035ca thru 0x00bee0 (0x8917 bytes) 000088 1740485170.108041 punch from 0x00be5e to 0x0132f3 (0x7496 bytes) 000090 1740485170.109883 read 0x020c65 thru 0x02adc7 (0xa163 bytes) 000091 1740485170.111665 MIRROR_RESYNC 2 mirrors 000093 1740485170.253515 mapread 0x00cd94 thru 0x01a5b3 (0xd820 bytes) 000094 1740485170.259200 mapwrite 0x0028f5 thru 0x0124b8 (0xfbc4 bytes) 000095 1740485170.293917 write 0x0182ce thru 0x0275f6 (0xf329 bytes) 000097 1740485170.296513 trunc from 0x038260 to 0x022070 (0x161ef bytes) 000098 1740485170.304961 fallocate 0x008ae7 thru 0x018642 (0xfb5c bytes) 000100 1740485170.306627 mapread 0x01429f thru 0x021283 (0xcfe5 bytes) 000101 1740485170.307149 mapwrite 0x03ebf8 thru 0x03ffff (0x1408 bytes) 000102 1740485170.317776 write 0x02c23f thru 0x02dcb8 (0x1a7a bytes) 000103 1740485170.320432 read 0x00aba7 thru 0x014faa (0xa404 bytes) 000104 1740485170.320853 write 0x008ec0 thru 0x014768 (0xb8a9 bytes) 000106 1740485170.321826 trunc from 0x040000 to 0x0244e0 (0x1bb1f bytes) 000107 1740485170.325824 mapread 0x021d8b thru 0x023454 (0x16ca bytes) 000109 1740485170.327805 mapwrite 0x02d512 thru 0x0309b3 (0x34a2 bytes) 000110 1740485170.338514 mapread 0x01e513 thru 0x020121 (0x1c0f bytes) 000111 1740485170.339047 read 0x01340f thru 0x01b62c (0x821e bytes) 000112 1740485170.340642 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000113 1740485170.368268 mapread 0x02f320 thru 0x0309b3 (0x1694 bytes) 000114 1740485170.371222 fallocate 0x03234c thru 0x039956 (0x760b bytes) 000117 1740485170.373642 mapread 0x01d613 thru 0x0234c9 (0x5eb7 bytes) 000118 1740485170.376043 punch from 0x01989a to 0x0229ba (0x9121 bytes) 000120 1740485170.378037 read 0x02a533 thru 0x0309b3 (0x6481 bytes) 000121 1740485170.379679 mapwrite 0x03359e thru 0x03ffff (0xca62 bytes) 000122 1740485170.394751 read 0x00ff20 thru 0x01a261 (0xa342 bytes) 000123 1740485170.397619 fallocate 0x01baf0 thru 0x01c47e (0x98f bytes) 000124 1740485170.399022 punch from 0x01e25c to 0x026e4f (0x8bf4 bytes) 000125 1740485170.400316 write 0x01bd67 thru 0x0293bd (0xd657 bytes) 000126 1740485170.401224 trunc from 0x040000 to 0x0315fb (0xea04 bytes) 000127 1740485170.405200 fallocate 0x03753c thru 0x03ffff (0x8ac4 bytes) 000128 1740485170.407183 punch from 0x038f13 to 0x03b92e (0x2a1c bytes) 000131 1740485170.409390 fallocate 0x01c763 thru 0x02425c (0x7afa bytes) 000132 1740485170.410831 mapwrite 0x01d402 thru 0x021a13 (0x4612 bytes) 000133 1740485170.417132 read 0x013477 thru 0x020693 (0xd21d bytes) 000134 1740485170.417495 mapwrite 0x024fb5 thru 0x0252d2 (0x31e bytes) 000136 1740485170.421468 punch from 0x00bff3 to 0x00d324 (0x1332 bytes) 000137 1740485170.423004 trunc from 0x040000 to 0x024e87 (0x1b178 bytes) 000139 1740485170.427530 mapwrite 0x033934 thru 0x03c174 (0x8841 bytes) 000141 1740485170.440361 write 0x039dc6 thru 0x03e8a9 (0x4ae4 bytes) 000142 1740485170.441068 trunc from 0x03e8aa to 0x02a621 (0x14288 bytes) 000143 1740485170.444508 fallocate 0x01409b thru 0x022a1d (0xe983 bytes) 000144 1740485170.446118 punch from 0x00affb to 0x0161f7 (0xb1fd bytes) 000146 1740485170.448094 mapwrite 0x006945 thru 0x00a98a (0x4046 bytes) 000147 1740485170.454067 punch from 0x00e897 to 0x014890 (0x5ffa bytes) 000148 1740485170.455381 mapwrite 0x036af3 thru 0x03ffff (0x950d bytes) 000149 1740485170.476196 write 0x00f0cc thru 0x019a11 (0xa946 bytes) 000151 1740485170.481510 mapread 0x034127 thru 0x03ffff (0xbed9 bytes) 000152 1740485170.484817 fallocate 0x030e95 thru 0x031eb0 (0x101c bytes) 000153 1740485170.486323 punch from 0x017e40 to 0x01fde8 (0x7fa9 bytes) 000155 1740485170.490497 mapread 0x005e85 thru 0x012c73 (0xcdef bytes) 000157 1740485170.496869 write 0x00a635 thru 0x017e66 (0xd832 bytes) 000158 1740485170.499410 mapread 0x020986 thru 0x02a07b (0x96f6 bytes) 000160 1740485170.500377 trunc from 0x040000 to 0x002e01 (0x3d1fe bytes) 000162 1740485170.505709 read 0x000d4b thru 0x002e01 (0x20b7 bytes) 000163 1740485170.506142 fallocate 0x012b6d thru 0x020962 (0xddf6 bytes) 000164 1740485170.508040 trunc from 0x020963 to 0x02821b (0x78b9 bytes) 000165 1740485170.511726 trunc from 0x02821c to 0x02f0ad (0x6e92 bytes) 000168 1740485170.516306 trunc from 0x02f0ae to 0x010ec6 (0x1e1e7 bytes) 000169 1740485170.520440 read 0x00204b thru 0x00b232 (0x91e8 bytes) 000170 1740485170.523019 mapread 0x00df04 thru 0x010ec6 (0x2fc3 bytes) 000174 1740485170.524470 mapread 0x0109d6 thru 0x010ec6 (0x4f1 bytes) 000175 1740485170.524883 mapread 0x006307 thru 0x010ec6 (0xabc0 bytes) 000176 1740485170.525432 mapread 0x006be2 thru 0x008235 (0x1654 bytes) 000178 1740485170.526072 fallocate 0x034742 thru 0x03b5fc (0x6ebb bytes) 000179 1740485170.527718 fallocate 0x02b529 thru 0x031905 (0x63dd bytes) 000180 1740485170.529293 trunc from 0x03b5fd to 0x0165dd (0x2501f bytes) 000182 1740485170.533988 mapwrite 0x03b7d3 thru 0x03ffff (0x482d bytes) 000183 1740485170.546355 punch from 0x03d10f to 0x03ffff (0x2ef1 bytes) 000184 1740485170.548020 punch from 0x00141f to 0x004417 (0x2ff9 bytes) 000185 1740485170.549570 trunc from 0x040000 to 0x01e6c5 (0x2193a bytes) 000187 1740485170.554359 mapwrite 0x0087ae thru 0x014f75 (0xc7c8 bytes) 000188 1740485170.561817 mapwrite 0x017b68 thru 0x020e64 (0x92fd bytes) 000189 1740485170.576551 punch from 0x01c796 to 0x020e64 (0x46cf bytes) 000190 1740485170.578830 read 0x005d17 thru 0x009863 (0x3b4d bytes) 000192 1740485170.579550 write 0x02f4f1 thru 0x035431 (0x5f41 bytes) 000194 1740485170.580988 punch from 0x00248a to 0x004dbc (0x2933 bytes) 000197 1740485170.583491 mapwrite 0x00b203 thru 0x00e8e4 (0x36e2 bytes) 000198 1740485170.589430 fallocate 0x02ec2e thru 0x0316d4 (0x2aa7 bytes) 000200 1740485170.591725 read 0x016141 thru 0x022b65 (0xca25 bytes) 000201 1740485170.594999 write 0x02e5f6 thru 0x036078 (0x7a83 bytes) 000202 1740485170.595721 fallocate 0x0160d9 thru 0x024005 (0xdf2d bytes) 000203 1740485170.597332 read 0x034577 thru 0x0351c6 (0xc50 bytes) 000204 1740485170.597744 trunc from 0x036079 to 0x024d16 (0x11362 bytes) 000205 1740485170.602463 mapread 0x022b83 thru 0x024d16 (0x2194 bytes) 000206 1740485170.603083 punch from 0x0067e9 to 0x00b65e (0x4e76 bytes) 000207 1740485170.604624 read 0x01605c thru 0x024aa2 (0xea47 bytes) 000208 1740485170.605065 punch from 0x024a58 to 0x024d16 (0x2bf bytes) 000210 1740485170.606726 punch from 0x0147bb to 0x01a7a1 (0x5fe7 bytes) 000211 1740485170.608094 trunc from 0x024d17 to 0x031bbc (0xcea6 bytes) 000213 1740485170.611761 write 0x032228 thru 0x039da7 (0x7b80 bytes) 000214 1740485170.612618 mapwrite 0x03e180 thru 0x03ffff (0x1e80 bytes) 000216 1740485170.624172 mapwrite 0x00f1c8 thru 0x017cc2 (0x8afb bytes) 000218 1740485170.629406 read 0x03f208 thru 0x03ffff (0xdf8 bytes) 000220 1740485170.630073 punch from 0x01da53 to 0x02b209 (0xd7b7 bytes) 000221 1740485170.631511 mapread 0x03c670 thru 0x03e116 (0x1aa7 bytes) 000223 1740485170.632468 mapread 0x0129ba thru 0x01736e (0x49b5 bytes) 000224 1740485170.632911 punch from 0x02bd93 to 0x03aadc (0xed4a bytes) 000226 1740485170.635802 MIRROR_EXTEND to 2 mirrors 000227 1740485170.762511 punch from 0x007e36 to 0x00d77d (0x5948 bytes) 000228 1740485170.772154 read 0x022325 thru 0x02eec7 (0xcba3 bytes) 000229 1740485170.775295 trunc from 0x040000 to 0x019f21 (0x260de bytes) 000230 1740485170.786078 trunc from 0x019f22 to 0x03d7fa (0x238d9 bytes) 000231 1740485170.790108 trunc from 0x03d7fb to 0x02790a (0x15ef0 bytes) 000232 1740485170.794517 punch from 0x01626b to 0x01692d (0x6c3 bytes) 000233 1740485170.796268 trunc from 0x02790b to 0x00f9d6 (0x17f34 bytes) 000234 1740485170.800676 fallocate 0x03e5e7 thru 0x03ffff (0x1a19 bytes) 000236 1740485170.802756 read 0x00472f thru 0x006cb0 (0x2582 bytes) 000237 1740485170.805430 mapwrite 0x013953 thru 0x019236 (0x58e4 bytes) 000238 1740485170.817553 read 0x0108b9 thru 0x013214 (0x295c bytes) 000239 1740485170.819295 MIRROR_EXTEND to 3 mirrors 000240 1740485170.912799 mapwrite 0x00ca1e thru 0x012fed (0x65d0 bytes) 000241 1740485170.944916 mapread 0x01073b thru 0x019236 (0x8afc bytes) 000242 1740485170.946979 write 0x025a4e thru 0x0261d1 (0x784 bytes) 000243 1740485170.949717 MIRROR_RESYNC 3 mirrors 000244 1740485171.096083 punch from 0x00c95d to 0x01ba19 (0xf0bd bytes) 000245 1740485171.104846 mapwrite 0x00f1d7 thru 0x00faec (0x916 bytes) 000246 1740485171.112514 MIRROR_RESYNC 3 mirrors 000248 1740485171.246410 punch from 0x018382 to 0x020833 (0x84b2 bytes) 000249 1740485171.253960 mapwrite 0x0181b0 thru 0x01d16e (0x4fbf bytes) 000250 1740485171.263737 punch from 0x00d35d to 0x0106db (0x337f bytes) 000251 1740485171.265009 mapwrite 0x022c63 thru 0x027dbd (0x515b bytes) 000252 1740485171.282021 MIRROR_EXTEND to 4 mirrors 000253 1740485171.373493 mapwrite 0x02a69c thru 0x02c9b0 (0x2315 bytes) 000254 1740485171.392021 write 0x03235a thru 0x03fd5e (0xda05 bytes) 000255 1740485171.392900 write 0x024dcc thru 0x02d217 (0x844c bytes) 000257 1740485171.396203 write 0x01d083 thru 0x0281b7 (0xb135 bytes) 000259 1740485171.397166 mapwrite 0x029469 thru 0x0388d9 (0xf471 bytes) 000260 1740485171.404211 mapwrite 0x037335 thru 0x03ffff (0x8ccb bytes) 000261 1740485171.414550 fallocate 0x02837b thru 0x02cbb4 (0x483a bytes) 000262 1740485171.416055 read 0x03e878 thru 0x03ffff (0x1788 bytes) 000263 1740485171.416358 fallocate 0x02540c thru 0x025987 (0x57c bytes) 000264 1740485171.417618 read 0x0079e2 thru 0x012006 (0xa625 bytes) 000265 1740485171.420241 write 0x02093b thru 0x02a1eb (0x98b1 bytes) 000266 1740485171.421111 mapread 0x0302c0 thru 0x03f6ed (0xf42e bytes) 000267 1740485171.421730 read 0x00e50b thru 0x015aa2 (0x7598 bytes) 000268 1740485171.422135 mapwrite 0x00f705 thru 0x014942 (0x523e bytes) 000269 1740485171.427364 mapread 0x003f4a thru 0x007e52 (0x3f09 bytes) 000270 1740485171.427958 mapread 0x0180c4 thru 0x025505 (0xd442 bytes) 000271 1740485171.428538 write 0x00dc06 thru 0x00eb76 (0xf71 bytes) 000273 1740485171.429472 write 0x01b336 thru 0x02b13d (0xfe08 bytes) 000275 1740485171.430827 punch from 0x000ac4 to 0x001391 (0x8ce bytes) 000276 1740485171.432793 read 0x03eff2 thru 0x03ffff (0x100e bytes) 000277 1740485171.433244 write 0x01d4f5 thru 0x022c25 (0x5731 bytes) 000278 1740485171.433836 punch from 0x021d13 to 0x02e4ad (0xc79b bytes) 000279 1740485171.438086 mapwrite 0x010d47 thru 0x014103 (0x33bd bytes) 000280 1740485171.442833 trunc from 0x040000 to 0x005678 (0x3a987 bytes) 000281 1740485171.446779 punch from 0x004a09 to 0x005678 (0xc70 bytes) 000282 1740485171.448164 punch from 0x000594 to 0x00123e (0xcab bytes) 000283 1740485171.449269 trunc from 0x005679 to 0x00ac48 (0x55d0 bytes) 000284 1740485171.452638 fallocate 0x002a75 thru 0x006cc0 (0x424c bytes) 000285 1740485171.454278 fallocate 0x03588a thru 0x03ffff (0xa776 bytes) 000286 1740485171.457062 MIRROR_RESYNC 4 mirrors 000287 1740485171.557122 mapread 0x0078c3 thru 0x00ac48 (0x3386 bytes) 000289 1740485171.560803 read 0x0097b1 thru 0x00ac48 (0x1498 bytes) 000290 1740485171.562046 MIRROR_EXTEND to 5 mirrors 000293 1740485171.645736 read 0x00685b thru 0x00ac48 (0x43ee bytes) 000294 1740485171.647322 mapwrite 0x01144d thru 0x017a0a (0x65be bytes) 000295 1740485171.671952 MIRROR_RESYNC 5 mirrors 000296 1740485171.821472 fallocate 0x00dd73 thru 0x01c607 (0xe895 bytes) 000297 1740485171.832478 fallocate 0x021a0d thru 0x023d05 (0x22f9 bytes) 000299 1740485171.835508 fallocate 0x006b3f thru 0x013ac9 (0xcf8b bytes) 000300 1740485171.837477 mapread 0x016da6 thru 0x01f8ce (0x8b29 bytes) 000302 1740485171.845443 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000304 1740485171.900091 fallocate 0x020a30 thru 0x02f3ad (0xe97e bytes) 000305 1740485171.901890 mapwrite 0x00bf4d thru 0x01ae3e (0xeef2 bytes) 000306 1740485171.916803 write 0x021539 thru 0x029fb8 (0x8a80 bytes) 000308 1740485171.922447 fallocate 0x009e56 thru 0x016faa (0xd155 bytes) 000309 1740485171.924684 write 0x03ee99 thru 0x03ffff (0x1167 bytes) 000311 1740485171.925615 mapread 0x0021f2 thru 0x00c6d1 (0xa4e0 bytes) 000312 1740485171.928136 write 0x019c14 thru 0x01fd16 (0x6103 bytes) 000315 1740485171.929392 punch from 0x018b12 to 0x0216db (0x8bca bytes) 000318 1740485171.934204 trunc from 0x040000 to 0x025fa2 (0x1a05d bytes) 000319 1740485171.943895 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000320 1740485171.992259 mapread 0x0012e7 thru 0x010a19 (0xf733 bytes) 000321 1740485172.001315 MIRROR_EXTEND to 4 mirrors 000322 1740485172.119230 punch from 0x00b529 to 0x012836 (0x730e bytes) 000323 1740485172.127551 mapread 0x0217e7 thru 0x025fa2 (0x47bc bytes) 000324 1740485172.132480 write 0x039a82 thru 0x03ffff (0x657e bytes) 000325 1740485172.134455 punch from 0x007372 to 0x00fa92 (0x8721 bytes) 000326 1740485172.135800 mapwrite 0x00ffe4 thru 0x01f316 (0xf333 bytes) 000327 1740485172.148644 MIRROR_RESYNC 4 mirrors 000328 1740485172.394365 read 0x036c01 thru 0x03ffff (0x93ff bytes) 000329 1740485172.396609 punch from 0x038a57 to 0x03eb48 (0x60f2 bytes) 000331 1740485172.404160 mapread 0x0265cb thru 0x032542 (0xbf78 bytes) 000332 1740485172.411134 mapread 0x00cf7e thru 0x00f04d (0x20d0 bytes) 000334 1740485172.413956 trunc from 0x040000 to 0x00ac6e (0x35391 bytes) 000335 1740485172.424304 read 0x000f97 thru 0x00ac6e (0x9cd8 bytes) 000336 1740485172.427154 fallocate 0x00269a thru 0x00b728 (0x908f bytes) 000337 1740485172.428963 mapread 0x00146c thru 0x00a8f1 (0x9486 bytes) 000338 1740485172.429544 mapwrite 0x02be77 thru 0x0311d7 (0x5361 bytes) 000340 1740485172.447334 write 0x02cf6a thru 0x03b077 (0xe10e bytes) 000341 1740485172.450157 MIRROR_EXTEND to 5 mirrors 000342 1740485172.573332 write 0x000cb8 thru 0x00eb0b (0xde54 bytes) 000343 1740485172.588736 punch from 0x033306 to 0x03b077 (0x7d72 bytes) 000344 1740485172.590377 write 0x038b24 thru 0x03ffff (0x74dc bytes) 000345 1740485172.592525 write 0x02d70a thru 0x039a95 (0xc38c bytes) 000346 1740485172.597530 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000347 1740485172.653252 mapwrite 0x004ed6 thru 0x01098d (0xbab8 bytes) 000348 1740485172.667240 read 0x021de6 thru 0x0294a1 (0x76bc bytes) 000349 1740485172.670151 fallocate 0x02e239 thru 0x03a627 (0xc3ef bytes) 000350 1740485172.671595 punch from 0x010971 to 0x01c580 (0xbc10 bytes) 000351 1740485172.673042 mapread 0x008d0e thru 0x01719f (0xe492 bytes) 000352 1740485172.676052 trunc from 0x040000 to 0x01f9ea (0x20615 bytes) 000353 1740485172.685086 mapwrite 0x020579 thru 0x0260a7 (0x5b2f bytes) 000354 1740485172.699809 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000355 1740485172.746680 punch from 0x01f19d to 0x0260a7 (0x6f0b bytes) 000356 1740485172.748706 punch from 0x0221e4 to 0x0259a5 (0x37c2 bytes) 000357 1740485172.751701 MIRROR_RESYNC 3 mirrors 000358 1740485172.861828 mapread 0x00e89b thru 0x016b3d (0x82a3 bytes) 000359 1740485172.867467 mapread 0x00e702 thru 0x017d34 (0x9633 bytes) 000360 1740485172.867827 punch from 0x0123dc to 0x016a16 (0x463b bytes) 000361 1740485172.877432 fallocate 0x0163f7 thru 0x01734d (0xf57 bytes) 000362 1740485172.879267 mapwrite 0x031e0e thru 0x034a6d (0x2c60 bytes) 000363 1740485172.895824 write 0x02d27e thru 0x037484 (0xa207 bytes) 000364 1740485172.896680 mapread 0x0353f3 thru 0x037484 (0x2092 bytes) 000365 1740485172.897117 write 0x0304fd thru 0x030783 (0x287 bytes) 000367 1740485172.897543 write 0x0082ce thru 0x018133 (0xfe66 bytes) 000368 1740485172.901929 trunc from 0x037485 to 0x031593 (0x5ef1 bytes) 000369 1740485172.906728 read 0x00c5ef thru 0x013b5e (0x7570 bytes) 000371 1740485172.907483 fallocate 0x00f281 thru 0x01ce16 (0xdb96 bytes) 000372 1740485172.911475 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000373 1740485172.967186 read 0x0015f3 thru 0x007a0b (0x6419 bytes) 000374 1740485172.971716 MIRROR_EXTEND to 3 mirrors 000375 1740485173.086188 mapread 0x026ce6 thru 0x031593 (0xa8ae bytes) 000376 1740485173.096146 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000377 1740485173.140681 mapwrite 0x03396a thru 0x03e910 (0xafa7 bytes) 000379 1740485173.159880 MIRROR_RESYNC 2 mirrors 000380 1740485173.265859 trunc from 0x03e911 to 0x0341b3 (0xa75d bytes) 000381 1740485173.280342 fallocate 0x009852 thru 0x015cea (0xc499 bytes) 000382 1740485173.281887 trunc from 0x0341b4 to 0x00a505 (0x29cae bytes) 000383 1740485173.287570 write 0x02fa31 thru 0x036f79 (0x7549 bytes) 000384 1740485173.288591 mapread 0x025cc0 thru 0x02a619 (0x495a bytes) 000386 1740485173.294266 mapread 0x028f55 thru 0x03387e (0xa92a bytes) 000387 1740485173.294939 mapwrite 0x02ad20 thru 0x02d711 (0x29f2 bytes) 000388 1740485173.302060 write 0x005c16 thru 0x00f050 (0x943b bytes) 000389 1740485173.306561 trunc from 0x036f7a to 0x03d7a7 (0x682e bytes) 000390 1740485173.311134 write 0x02f93b thru 0x036028 (0x66ee bytes) 000391 1740485173.311845 mapread 0x01ded6 thru 0x025ceb (0x7e16 bytes) 000392 1740485173.314290 write 0x023072 thru 0x02b2f3 (0x8282 bytes) 000393 1740485173.315048 mapwrite 0x03f221 thru 0x03ffff (0xddf bytes) 000395 1740485173.325823 mapread 0x02fac2 thru 0x0314e6 (0x1a25 bytes) 000396 1740485173.326463 fallocate 0x03b480 thru 0x03ffff (0x4b80 bytes) 000397 1740485173.328096 read 0x01ee04 thru 0x026c5a (0x7e57 bytes) 000399 1740485173.328819 write 0x0318f0 thru 0x038b6a (0x727b bytes) 000401 1740485173.329736 punch from 0x010d99 to 0x01f1c7 (0xe42f bytes) 000403 1740485173.331460 mapwrite 0x006874 thru 0x016686 (0xfe13 bytes) 000406 1740485173.340606 trunc from 0x040000 to 0x012b66 (0x2d499 bytes) 000407 1740485173.344944 mapread 0x005631 thru 0x01110d (0xbadd bytes) 000408 1740485173.345623 mapread 0x0119bb thru 0x012b66 (0x11ac bytes) 000410 1740485173.346488 write 0x02426b thru 0x0312b4 (0xd04a bytes) 000411 1740485173.347595 write 0x02ce54 thru 0x034834 (0x79e1 bytes) 000414 1740485173.348909 trunc from 0x034835 to 0x000a0c (0x33e28 bytes) 000415 1740485173.353573 trunc from 0x000a0d to 0x033865 (0x32e59 bytes) 000417 1740485173.357831 mapread 0x02c5d4 thru 0x033865 (0x7292 bytes) 000418 1740485173.360630 fallocate 0x01e0ec thru 0x02959f (0xb4b4 bytes) 000420 1740485173.362859 trunc from 0x033866 to 0x031217 (0x264e bytes) 000421 1740485173.368947 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000424 1740485173.404730 fallocate 0x0016d9 thru 0x007d13 (0x663b bytes) 000425 1740485173.406908 fallocate 0x005ae2 thru 0x00be8c (0x63ab bytes) 000426 1740485173.408590 mapwrite 0x034f13 thru 0x03ffff (0xb0ed bytes) 000427 1740485173.427413 mapwrite 0x014fae thru 0x021286 (0xc2d9 bytes) 000428 1740485173.437954 mapwrite 0x0293d8 thru 0x035c9e (0xc8c7 bytes) 000429 1740485173.444638 trunc from 0x040000 to 0x017bb5 (0x2844a bytes) 000430 1740485173.448744 trunc from 0x017bb6 to 0x0254fa (0xd945 bytes) 000432 1740485173.452699 mapread 0x01ae9d thru 0x0254fa (0xa65e bytes) 000434 1740485173.457715 mapread 0x01a03b thru 0x0254fa (0xb4c0 bytes) 000435 1740485173.458310 punch from 0x000c84 to 0x0067b1 (0x5b2e bytes) 000437 1740485173.460455 fallocate 0x0204bf thru 0x026ef6 (0x6a38 bytes) 000438 1740485173.461998 mapwrite 0x00df0e thru 0x01c89c (0xe98f bytes) 000439 1740485173.469289 write 0x006da6 thru 0x007165 (0x3c0 bytes) 000441 1740485173.472125 trunc from 0x026ef7 to 0x000d22 (0x261d4 bytes) 000443 1740485173.477404 read 0x000bdc thru 0x000d22 (0x147 bytes) 000444 1740485173.479516 write 0x0161f2 thru 0x019293 (0x30a2 bytes) 000446 1740485173.480573 read 0x00e7e2 thru 0x019293 (0xaab2 bytes) 000447 1740485173.483640 fallocate 0x005947 thru 0x014fb5 (0xf66f bytes) 000448 1740485173.485400 read 0x006320 thru 0x00ac44 (0x4925 bytes) 000451 1740485173.486462 punch from 0x00c7d6 to 0x019293 (0xcabe bytes) 000453 1740485173.490943 read 0x010774 thru 0x019293 (0x8b20 bytes) 000454 1740485173.493814 read 0x0183ba thru 0x019293 (0xeda bytes) 000457 1740485173.494893 punch from 0x01124c to 0x019293 (0x8048 bytes) 000458 1740485173.496471 punch from 0x003d4c to 0x00d581 (0x9836 bytes) 000461 1740485173.499131 mapwrite 0x02eeb3 thru 0x03e5cb (0xf719 bytes) 000462 1740485173.514746 mapwrite 0x00c648 thru 0x0149a5 (0x835e bytes) 000463 1740485173.521135 write 0x0166c0 thru 0x022357 (0xbc98 bytes) 000465 1740485173.522373 write 0x014dfd thru 0x022241 (0xd445 bytes) 000466 1740485173.523276 punch from 0x03b416 to 0x03e5cb (0x31b6 bytes) 000468 1740485173.525540 mapwrite 0x01b981 thru 0x022ef0 (0x7570 bytes) 000469 1740485173.531368 fallocate 0x008904 thru 0x00f0c9 (0x67c6 bytes) 000471 1740485173.533122 mapread 0x003537 thru 0x010c31 (0xd6fb bytes) 000472 1740485173.533751 punch from 0x0088ca to 0x00b618 (0x2d4f bytes) 000473 1740485173.535211 read 0x03e080 thru 0x03e5cb (0x54c bytes) 000474 1740485173.539062 fallocate 0x010edd thru 0x01951c (0x8640 bytes) 000476 1740485173.541015 mapwrite 0x02456a thru 0x0256a4 (0x113b bytes) 000477 1740485173.545761 trunc from 0x03e5cc to 0x017a8c (0x26b3f bytes) 000478 1740485173.550531 mapwrite 0x038a6b thru 0x03ffff (0x7595 bytes) 000479 1740485173.572922 mapread 0x0064df thru 0x007484 (0xfa6 bytes) 000480 1740485173.573365 read 0x0011f0 thru 0x011016 (0xfe27 bytes) 000481 1740485173.573735 write 0x01fc98 thru 0x02be8c (0xc1f5 bytes) 000482 1740485173.577105 write 0x00664a thru 0x009892 (0x3249 bytes) 000484 1740485173.578243 trunc from 0x040000 to 0x02dc08 (0x123f7 bytes) 000485 1740485173.582258 write 0x0117f2 thru 0x01a0d7 (0x88e6 bytes) 000486 1740485173.583140 write 0x0233a8 thru 0x0319be (0xe617 bytes) 000487 1740485173.584148 write 0x02eb22 thru 0x035bf5 (0x70d4 bytes) 000488 1740485173.584931 fallocate 0x01bf25 thru 0x01d54b (0x1627 bytes) 000489 1740485173.586840 trunc from 0x035bf6 to 0x037845 (0x1c50 bytes) 000491 1740485173.591961 trunc from 0x037846 to 0x019cd1 (0x1db74 bytes) 000492 1740485173.597060 read 0x00978e thru 0x0157d2 (0xc045 bytes) 000493 1740485173.598956 MIRROR_EXTEND to 2 mirrors 000494 1740485173.718144 write 0x02a7f0 thru 0x02d43c (0x2c4d bytes) 000495 1740485173.727999 trunc from 0x02d43d to 0x0146a3 (0x18d99 bytes) 000496 1740485173.738159 mapread 0x00f0c5 thru 0x00f97c (0x8b8 bytes) 000497 1740485173.740158 mapread 0x0057fd thru 0x006d2c (0x1530 bytes) 000498 1740485173.743692 read 0x00fea8 thru 0x010070 (0x1c9 bytes) 000499 1740485173.744107 fallocate 0x03385f thru 0x037f8c (0x472e bytes) 000501 1740485173.746162 punch from 0x0003e1 to 0x00804e (0x7c6e bytes) 000502 1740485173.747698 fallocate 0x0023d4 thru 0x0114a5 (0xf0d2 bytes) 000503 1740485173.749114 punch from 0x00cfd5 to 0x012771 (0x579d bytes) 000504 1740485173.751909 MIRROR_RESYNC 2 mirrors 000505 1740485173.846380 trunc from 0x0146a4 to 0x027c85 (0x135e2 bytes) 000506 1740485173.860465 trunc from 0x027c86 to 0x03c598 (0x14913 bytes) 000507 1740485173.864951 write 0x01f527 thru 0x02a5da (0xb0b4 bytes) 000508 1740485173.867562 fallocate 0x027c87 thru 0x02dbde (0x5f58 bytes) 000509 1740485173.869027 read 0x024e16 thru 0x03093e (0xbb29 bytes) 000512 1740485173.871705 read 0x027ba4 thru 0x03239a (0xa7f7 bytes) 000513 1740485173.874523 read 0x002139 thru 0x00b394 (0x925c bytes) 000515 1740485173.875223 mapwrite 0x00faea thru 0x010d26 (0x123d bytes) 000516 1740485173.880773 punch from 0x025378 to 0x0257d1 (0x45a bytes) 000517 1740485173.882242 mapread 0x01360b thru 0x01e16d (0xab63 bytes) 000518 1740485173.882896 punch from 0x016611 to 0x02653a (0xff2a bytes) 000519 1740485173.884471 read 0x014514 thru 0x0238b0 (0xf39d bytes) 000520 1740485173.887501 fallocate 0x02a73f thru 0x02eb3d (0x43ff bytes) 000521 1740485173.889246 fallocate 0x01741d thru 0x022c9a (0xb87e bytes) 000522 1740485173.890740 write 0x01257d thru 0x01f9b5 (0xd439 bytes) 000523 1740485173.891798 fallocate 0x0266d6 thru 0x02b86e (0x5199 bytes) 000525 1740485173.893842 mapwrite 0x0255c7 thru 0x034ca6 (0xf6e0 bytes) 000526 1740485173.900393 trunc from 0x03c599 to 0x02f37b (0xd21d bytes) 000527 1740485173.904264 write 0x02b420 thru 0x02e0d2 (0x2cb3 bytes) 000528 1740485173.904847 write 0x00c35e thru 0x0172bf (0xaf62 bytes) 000529 1740485173.905530 mapwrite 0x02b407 thru 0x02dee8 (0x2ae2 bytes) 000530 1740485173.910850 mapread 0x02ae9d thru 0x02f37b (0x44df bytes) 000531 1740485173.912927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000532 1740485173.944607 mapread 0x00544f thru 0x013f27 (0xead9 bytes) 000534 1740485173.950754 mapread 0x02bd35 thru 0x02f37b (0x3647 bytes) 000535 1740485173.953175 punch from 0x01d768 to 0x029ee0 (0xc779 bytes) 000536 1740485173.954588 trunc from 0x02f37c to 0x02b1a9 (0x41d2 bytes) 000538 1740485173.959328 punch from 0x020880 to 0x02b1a9 (0xa92a bytes) 000540 1740485173.962036 MIRROR_EXTEND to 2 mirrors 000541 1740485174.067245 MIRROR_RESYNC 2 mirrors 000542 1740485174.112807 punch from 0x0072cf to 0x00de16 (0x6b48 bytes) 000543 1740485174.123070 MIRROR_RESYNC 2 mirrors 000544 1740485174.240732 punch from 0x01f59a to 0x029808 (0xa26f bytes) 000545 1740485174.249266 MIRROR_EXTEND to 3 mirrors 000546 1740485174.392102 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000547 1740485174.470539 punch from 0x00a7b9 to 0x012580 (0x7dc8 bytes) 000548 1740485174.478371 read 0x008acb thru 0x017127 (0xe65d bytes) 000549 1740485174.480791 write 0x027fad thru 0x02bc35 (0x3c89 bytes) 000550 1740485174.484220 write 0x0137ca thru 0x018fc2 (0x57f9 bytes) 000551 1740485174.485843 mapwrite 0x032d2d thru 0x0377d1 (0x4aa5 bytes) 000552 1740485174.496762 write 0x002353 thru 0x00ac46 (0x88f4 bytes) 000554 1740485174.499274 fallocate 0x0202a3 thru 0x0296fa (0x9458 bytes) 000556 1740485174.501221 write 0x02fff9 thru 0x033c7b (0x3c83 bytes) 000557 1740485174.501775 trunc from 0x0377d2 to 0x017e11 (0x1f9c0 bytes) 000559 1740485174.505740 trunc from 0x017e12 to 0x036469 (0x1e658 bytes) 000560 1740485174.508912 punch from 0x00ed76 to 0x01335d (0x45e8 bytes) 000561 1740485174.510368 mapwrite 0x018a54 thru 0x0194f1 (0xa9e bytes) 000562 1740485174.517154 mapwrite 0x0288e8 thru 0x03797f (0xf098 bytes) 000563 1740485174.532579 write 0x006edb thru 0x0168e4 (0xfa0a bytes) 000565 1740485174.534124 mapwrite 0x02b4b0 thru 0x039f79 (0xeaca bytes) 000566 1740485174.549400 write 0x030f3a thru 0x039e1a (0x8ee1 bytes) 000568 1740485174.550469 trunc from 0x039f7a to 0x017fd3 (0x21fa6 bytes) 000569 1740485174.555127 mapwrite 0x0058c2 thru 0x01210f (0xc84e bytes) 000570 1740485174.564305 mapwrite 0x00bff8 thru 0x00fcb8 (0x3cc1 bytes) 000571 1740485174.571510 read 0x00d773 thru 0x00fbfe (0x248c bytes) 000574 1740485174.574302 MIRROR_EXTEND to 3 mirrors 000575 1740485174.673128 trunc from 0x017fd4 to 0x00ed50 (0x9283 bytes) 000576 1740485174.687255 write 0x001b87 thru 0x007781 (0x5bfb bytes) 000577 1740485174.689938 fallocate 0x00845d thru 0x013d5f (0xb903 bytes) 000579 1740485174.691698 read 0x00707a thru 0x011012 (0x9f99 bytes) 000580 1740485174.693933 write 0x01958d thru 0x01d873 (0x42e7 bytes) 000582 1740485174.694622 fallocate 0x016720 thru 0x0234f4 (0xcdd5 bytes) 000583 1740485174.696410 mapwrite 0x001def thru 0x00d4b1 (0xb6c3 bytes) 000584 1740485174.704459 mapread 0x01224b thru 0x017059 (0x4e0f bytes) 000585 1740485174.709571 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000586 1740485174.764767 fallocate 0x0273c9 thru 0x035257 (0xde8f bytes) 000588 1740485174.767088 fallocate 0x028ac0 thru 0x033150 (0xa691 bytes) 000590 1740485174.769540 punch from 0x00b3c6 to 0x019e00 (0xea3b bytes) 000591 1740485174.771683 punch from 0x0308d1 to 0x033150 (0x2880 bytes) 000592 1740485174.773473 write 0x021115 thru 0x029776 (0x8662 bytes) 000593 1740485174.778837 MIRROR_RESYNC 2 mirrors 000594 1740485174.925357 read 0x00b8ab thru 0x00f436 (0x3b8c bytes) 000595 1740485174.927875 fallocate 0x0083e6 thru 0x00b9ff (0x361a bytes) 000596 1740485174.936529 read 0x027c1a thru 0x02ddc8 (0x61af bytes) 000597 1740485174.939977 read 0x02d41b thru 0x02d5d4 (0x1ba bytes) 000598 1740485174.940386 mapread 0x0015d0 thru 0x003ef4 (0x2925 bytes) 000599 1740485174.940988 fallocate 0x004807 thru 0x012527 (0xdd21 bytes) 000600 1740485174.942738 trunc from 0x033151 to 0x026926 (0xc82a bytes) 000602 1740485174.952204 trunc from 0x026927 to 0x03f99c (0x19076 bytes) 000604 1740485174.955979 mapwrite 0x026d3c thru 0x028870 (0x1b35 bytes) 000605 1740485174.963506 MIRROR_EXTEND to 3 mirrors 000606 1740485175.081210 mapwrite 0x00f73a thru 0x013e23 (0x46ea bytes) 000607 1740485175.113710 punch from 0x010976 to 0x011c09 (0x1294 bytes) 000608 1740485175.115333 fallocate 0x02a411 thru 0x03584a (0xb43a bytes) 000610 1740485175.117161 fallocate 0x01d776 thru 0x0276aa (0x9f35 bytes) 000611 1740485175.120470 MIRROR_EXTEND to 4 mirrors 000613 1740485175.250781 trunc from 0x03f99d to 0x0188c9 (0x270d3 bytes) 000614 1740485175.261598 fallocate 0x01cf01 thru 0x01d0ff (0x1ff bytes) 000618 1740485175.264154 fallocate 0x008b85 thru 0x009f5b (0x13d7 bytes) 000619 1740485175.266008 read 0x00a869 thru 0x010887 (0x601f bytes) 000620 1740485175.268727 read 0x015d96 thru 0x0188c9 (0x2b34 bytes) 000621 1740485175.271561 fallocate 0x015dd1 thru 0x01e36a (0x859a bytes) 000622 1740485175.273378 mapwrite 0x00d0b0 thru 0x01c6b1 (0xf602 bytes) 000624 1740485175.284619 mapread 0x00c582 thru 0x0199a7 (0xd426 bytes) 000625 1740485175.285327 mapread 0x01a417 thru 0x01e36a (0x3f54 bytes) 000627 1740485175.286161 punch from 0x0160bd to 0x0187a0 (0x26e4 bytes) 000628 1740485175.288115 mapwrite 0x0373d3 thru 0x039401 (0x202f bytes) 000629 1740485175.300714 trunc from 0x039402 to 0x01b4c2 (0x1df3f bytes) 000630 1740485175.305849 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000632 1740485175.346873 MIRROR_RESYNC 3 mirrors 000633 1740485175.457579 trunc from 0x01b4c3 to 0x0322ae (0x16dec bytes) 000634 1740485175.471883 mapwrite 0x007ee4 thru 0x013635 (0xb752 bytes) 000635 1740485175.486089 trunc from 0x0322af to 0x025e21 (0xc48d bytes) 000636 1740485175.490469 fallocate 0x03034e thru 0x036c7e (0x6931 bytes) 000637 1740485175.492048 mapread 0x00de8c thru 0x01daaf (0xfc24 bytes) 000638 1740485175.494449 punch from 0x007a4c to 0x015dfd (0xe3b2 bytes) 000639 1740485175.495936 trunc from 0x025e22 to 0x01a120 (0xbd01 bytes) 000642 1740485175.499981 write 0x006575 thru 0x011688 (0xb114 bytes) 000643 1740485175.501950 mapwrite 0x013205 thru 0x01e065 (0xae61 bytes) 000644 1740485175.514242 trunc from 0x01e066 to 0x00033b (0x1dd2a bytes) 000645 1740485175.518189 punch from 0x0001b0 to 0x00033b (0x18c bytes) 000646 1740485175.519543 read 0x000149 thru 0x00033b (0x1f3 bytes) 000647 1740485175.521233 mapread 0x00020e thru 0x00033b (0x12e bytes) 000648 1740485175.521815 read 0x0001f9 thru 0x00033b (0x143 bytes) 000649 1740485175.522190 fallocate 0x01a176 thru 0x021f94 (0x7e1f bytes) 000651 1740485175.524216 mapwrite 0x0307fd thru 0x031a60 (0x1264 bytes) 000653 1740485175.535898 mapread 0x022cf2 thru 0x0269ef (0x3cfe bytes) 000654 1740485175.536392 trunc from 0x031a61 to 0x01ff9d (0x11ac3 bytes) 000655 1740485175.539719 punch from 0x01f833 to 0x01ff9d (0x76b bytes) 000656 1740485175.540944 mapread 0x00b117 thru 0x011210 (0x60fa bytes) 000657 1740485175.541543 mapwrite 0x01b83d thru 0x01be79 (0x63d bytes) 000658 1740485175.545958 punch from 0x019930 to 0x01ff9d (0x666e bytes) 000661 1740485175.548207 mapread 0x01543d thru 0x01ff9d (0xab61 bytes) 000662 1740485175.555800 punch from 0x01287a to 0x01d31e (0xaaa5 bytes) 000663 1740485175.557205 trunc from 0x01ff9e to 0x007b0f (0x1848e bytes) 000664 1740485175.561868 mapwrite 0x02dea8 thru 0x0350e5 (0x723e bytes) 000665 1740485175.579743 MIRROR_RESYNC 3 mirrors 000666 1740485175.681869 fallocate 0x032551 thru 0x03dc02 (0xb6b2 bytes) 000667 1740485175.689904 mapread 0x02f843 thru 0x033392 (0x3b50 bytes) 000668 1740485175.694287 fallocate 0x00474d thru 0x010f56 (0xc80a bytes) 000669 1740485175.695881 punch from 0x00b0cb to 0x016f9b (0xbed1 bytes) 000670 1740485175.697327 mapwrite 0x022156 thru 0x02ddb8 (0xbc63 bytes) 000672 1740485175.706123 write 0x03233b thru 0x032508 (0x1ce bytes) 000674 1740485175.707048 punch from 0x000362 to 0x00883b (0x84da bytes) 000675 1740485175.708996 read 0x01e63a thru 0x01e94e (0x315 bytes) 000676 1740485175.709384 write 0x02e6d0 thru 0x032ab5 (0x43e6 bytes) 000678 1740485175.710504 write 0x03545c thru 0x03ffff (0xaba4 bytes) 000681 1740485175.712085 write 0x0117e3 thru 0x01c448 (0xac66 bytes) 000682 1740485175.717017 MIRROR_RESYNC 3 mirrors 000685 1740485175.852015 MIRROR_RESYNC 3 mirrors 000686 1740485175.890930 MIRROR_EXTEND to 4 mirrors 000687 1740485175.996777 fallocate 0x02623e thru 0x02a4e4 (0x42a7 bytes) 000688 1740485176.005228 MIRROR_EXTEND to 5 mirrors 000689 1740485176.129545 mapwrite 0x02c940 thru 0x03485e (0x7f1f bytes) 000690 1740485176.161937 mapread 0x025385 thru 0x0289c1 (0x363d bytes) 000691 1740485176.164388 trunc from 0x040000 to 0x01e262 (0x21d9d bytes) 000692 1740485176.171213 mapread 0x00ee0c thru 0x016810 (0x7a05 bytes) 000693 1740485176.176322 mapread 0x016a22 thru 0x01db7d (0x715c bytes) 000695 1740485176.177306 write 0x00d7ed thru 0x012bd6 (0x53ea bytes) 000696 1740485176.177980 mapread 0x01c87e thru 0x01e262 (0x19e5 bytes) 000697 1740485176.178515 mapread 0x01de2d thru 0x01e262 (0x436 bytes) 000698 1740485176.179078 trunc from 0x01e263 to 0x0272b7 (0x9055 bytes) 000700 1740485176.183743 fallocate 0x006d2a thru 0x011296 (0xa56d bytes) 000701 1740485176.185580 read 0x009499 thru 0x012d08 (0x9870 bytes) 000702 1740485176.187197 MIRROR_RESYNC 5 mirrors 000703 1740485176.362866 mapwrite 0x02d59c thru 0x03b8c5 (0xe32a bytes) 000704 1740485176.386017 write 0x00b918 thru 0x013b40 (0x8229 bytes) 000705 1740485176.388457 punch from 0x035cea to 0x03b8c5 (0x5bdc bytes) 000706 1740485176.390028 read 0x00bf6a thru 0x01806e (0xc105 bytes) 000707 1740485176.392205 read 0x01a4a1 thru 0x01d4ce (0x302e bytes) 000708 1740485176.392501 read 0x01014d thru 0x01a560 (0xa414 bytes) 000709 1740485176.392899 fallocate 0x00b3f4 thru 0x01182b (0x6438 bytes) 000711 1740485176.394426 fallocate 0x017a67 thru 0x017a90 (0x2a bytes) 000713 1740485176.395890 write 0x004b35 thru 0x007f47 (0x3413 bytes) 000714 1740485176.396460 mapread 0x026723 thru 0x026a13 (0x2f1 bytes) 000716 1740485176.399411 mapwrite 0x037c8b thru 0x03ffff (0x8375 bytes) 000717 1740485176.411242 fallocate 0x039e1c thru 0x03ffff (0x61e4 bytes) 000718 1740485176.412857 mapwrite 0x00765c thru 0x00eea4 (0x7849 bytes) 000719 1740485176.418417 mapread 0x039def thru 0x03ffff (0x6211 bytes) 000720 1740485176.418976 trunc from 0x040000 to 0x029502 (0x16afd bytes) 000721 1740485176.423672 MIRROR_EXTEND to 6 mirrors 000722 1740485176.522916 trunc from 0x029503 to 0x00ee97 (0x1a66b bytes) 000724 1740485176.535191 fallocate 0x003a6f thru 0x00a39b (0x692d bytes) 000725 1740485176.537463 MIRROR_EXTEND to 7 mirrors 000726 1740485176.627710 write 0x001cbb thru 0x009ced (0x8033 bytes) 000728 1740485176.638546 punch from 0x0072ca to 0x00e0d2 (0x6e09 bytes) 000729 1740485176.642717 mapread 0x00108b thru 0x00cfbb (0xbf31 bytes) 000730 1740485176.645561 fallocate 0x02a237 thru 0x039500 (0xf2ca bytes) 000731 1740485176.647276 trunc from 0x039501 to 0x009e58 (0x2f6a8 bytes) 000732 1740485176.655708 punch from 0x001ed2 to 0x0083b0 (0x64df bytes) 000733 1740485176.658561 MIRROR_EXTEND to 8 mirrors 000734 1740485176.760077 punch from 0x006363 to 0x009e58 (0x3af6 bytes) 000735 1740485176.768699 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000736 1740485176.818757 mapread 0x003a22 thru 0x009e58 (0x6437 bytes) 000737 1740485176.823810 fallocate 0x01b2b5 thru 0x021c0e (0x695a bytes) 000738 1740485176.825508 fallocate 0x014148 thru 0x019afb (0x59b4 bytes) 000739 1740485176.827245 trunc from 0x021c0f to 0x028dc1 (0x71b3 bytes) 000740 1740485176.834644 punch from 0x01d23e to 0x020a18 (0x37db bytes) 000741 1740485176.837500 MIRROR_RESYNC 7 mirrors 000743 1740485177.141477 mapwrite 0x035aaa thru 0x03bf22 (0x6479 bytes) 000744 1740485177.163193 trunc from 0x03bf23 to 0x011772 (0x2a7b0 bytes) 000745 1740485177.168828 mapwrite 0x03261e thru 0x033eaa (0x188d bytes) 000747 1740485177.182612 fallocate 0x021c8a thru 0x02902b (0x73a2 bytes) 000748 1740485177.184616 write 0x0205cd thru 0x0212a5 (0xcd9 bytes) 000749 1740485177.187466 read 0x001140 thru 0x0089b4 (0x7875 bytes) 000750 1740485177.189627 mapwrite 0x01af37 thru 0x01f776 (0x4840 bytes) 000751 1740485177.195198 read 0x0157fe thru 0x019e09 (0x460c bytes) 000752 1740485177.197933 mapwrite 0x0003b7 thru 0x00d789 (0xd3d3 bytes) 000753 1740485177.204348 MIRROR_EXTEND to 8 mirrors 000754 1740485177.305567 write 0x004a30 thru 0x009358 (0x4929 bytes) 000755 1740485177.316890 fallocate 0x002efd thru 0x00e9bf (0xbac3 bytes) 000756 1740485177.318556 trunc from 0x033eab to 0x023bf1 (0x102b9 bytes) 000757 1740485177.327721 punch from 0x0006af to 0x00bfea (0xb93c bytes) 000758 1740485177.331313 fallocate 0x01afdd thru 0x01ba73 (0xa97 bytes) 000759 1740485177.333716 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000760 1740485177.377553 fallocate 0x00e286 thru 0x01abf3 (0xc96e bytes) 000761 1740485177.379061 mapwrite 0x016de4 thru 0x022fbf (0xc1dc bytes) 000763 1740485177.390182 trunc from 0x023bf2 to 0x000725 (0x234cc bytes) 000764 1740485177.394500 fallocate 0x03158d thru 0x03ea7e (0xd4f2 bytes) 000766 1740485177.396097 trunc from 0x000726 to 0x03ff1b (0x3f7f6 bytes) 000767 1740485177.399954 trunc from 0x03ff1c to 0x03e13e (0x1ddd bytes) 000768 1740485177.403950 write 0x02ec79 thru 0x035322 (0x66aa bytes) 000769 1740485177.406928 mapread 0x0213a4 thru 0x02d415 (0xc072 bytes) 000770 1740485177.410142 mapread 0x023929 thru 0x029507 (0x5bdf bytes) 000771 1740485177.410717 fallocate 0x029716 thru 0x02a516 (0xe01 bytes) 000772 1740485177.412605 trunc from 0x03e13f to 0x0016e7 (0x3ca57 bytes) 000773 1740485177.418311 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000774 1740485177.470280 mapwrite 0x01f4c4 thru 0x02a899 (0xb3d6 bytes) 000775 1740485177.486770 punch from 0x0022d3 to 0x00468f (0x23bd bytes) 000776 1740485177.488339 mapread 0x02008e thru 0x02a899 (0xa80c bytes) 000777 1740485177.490479 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000779 1740485177.537622 punch from 0x027a01 to 0x02a899 (0x2e99 bytes) 000780 1740485177.539060 mapread 0x00ddf3 thru 0x0138ad (0x5abb bytes) 000781 1740485177.543761 mapwrite 0x01ee7f thru 0x027793 (0x8915 bytes) 000782 1740485177.554556 MIRROR_RESYNC 5 mirrors 000783 1740485177.716479 write 0x03ad06 thru 0x03ffff (0x52fa bytes) 000784 1740485177.724117 mapread 0x029dae thru 0x02f5b9 (0x580c bytes) 000785 1740485177.729931 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000786 1740485177.777159 mapwrite 0x016fe5 thru 0x026636 (0xf652 bytes) 000787 1740485177.789643 trunc from 0x040000 to 0x01d940 (0x226bf bytes) 000789 1740485177.797333 trunc from 0x01d941 to 0x00e932 (0xf00e bytes) 000790 1740485177.800233 punch from 0x00cbc0 to 0x00e932 (0x1d73 bytes) 000792 1740485177.801677 mapwrite 0x028cfc thru 0x0303d3 (0x76d8 bytes) 000794 1740485177.812559 MIRROR_EXTEND to 5 mirrors 000796 1740485177.906996 read 0x00230b thru 0x00f520 (0xd216 bytes) 000798 1740485177.912467 write 0x0167a7 thru 0x02090e (0xa168 bytes) 000799 1740485177.923288 punch from 0x018019 to 0x01e627 (0x660f bytes) 000801 1740485177.927011 mapwrite 0x01f942 thru 0x027519 (0x7bd8 bytes) 000802 1740485177.935563 punch from 0x02c668 to 0x0303d3 (0x3d6c bytes) 000803 1740485177.937126 trunc from 0x0303d4 to 0x0155b2 (0x1ae21 bytes) 000804 1740485177.945555 mapread 0x00af79 thru 0x0155b2 (0xa63a bytes) 000805 1740485177.949422 punch from 0x00a792 to 0x011af0 (0x735f bytes) 000806 1740485177.950954 mapwrite 0x0292ad thru 0x02af08 (0x1c5c bytes) 000807 1740485177.961633 write 0x010910 thru 0x016d87 (0x6478 bytes) 000808 1740485177.964866 fallocate 0x02a536 thru 0x034aa0 (0xa56b bytes) 000809 1740485177.966469 mapwrite 0x03734a thru 0x03ffff (0x8cb6 bytes) 000810 1740485177.979957 mapread 0x0091db thru 0x0141fb (0xb021 bytes) 000812 1740485177.980937 write 0x03dc21 thru 0x03ffff (0x23df bytes) 000813 1740485177.982878 MIRROR_RESYNC 5 mirrors 000815 1740485178.190601 fallocate 0x0121b0 thru 0x016032 (0x3e83 bytes) 000816 1740485178.197111 write 0x034dc1 thru 0x03ffff (0xb23f bytes) 000817 1740485178.200079 MIRROR_RESYNC 5 mirrors 000818 1740485178.395766 mapread 0x0194b8 thru 0x021ac7 (0x8610 bytes) 000819 1740485178.401570 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000820 1740485178.474739 write 0x018719 thru 0x01e1e5 (0x5acd bytes) 000821 1740485178.488605 read 0x03d5a8 thru 0x03ffff (0x2a58 bytes) 000822 1740485178.491095 read 0x01ece7 thru 0x0229ad (0x3cc7 bytes) 000823 1740485178.496100 MIRROR_RESYNC 4 mirrors 000825 1740485178.714998 mapwrite 0x02b033 thru 0x037a0a (0xc9d8 bytes) 000826 1740485178.759475 MIRROR_EXTEND to 5 mirrors 000827 1740485178.883193 read 0x030a70 thru 0x036cac (0x623d bytes) 000828 1740485178.887678 punch from 0x015052 to 0x01b967 (0x6916 bytes) 000830 1740485178.898630 fallocate 0x034053 thru 0x039fc2 (0x5f70 bytes) 000831 1740485178.900372 write 0x0362b6 thru 0x038b58 (0x28a3 bytes) 000832 1740485178.905202 punch from 0x00551c to 0x0131f4 (0xdcd9 bytes) 000833 1740485178.907022 mapwrite 0x0049e0 thru 0x00da2a (0x904b bytes) 000834 1740485178.917687 write 0x01a875 thru 0x01ae8c (0x618 bytes) 000836 1740485178.920618 mapread 0x022245 thru 0x02fe37 (0xdbf3 bytes) 000837 1740485178.924752 trunc from 0x040000 to 0x02148e (0x1eb71 bytes) 000838 1740485178.935168 punch from 0x0014a0 to 0x002c06 (0x1767 bytes) 000839 1740485178.936998 write 0x01062e thru 0x01e89b (0xe26e bytes) 000840 1740485178.937939 mapwrite 0x014ed3 thru 0x017689 (0x27b7 bytes) 000841 1740485178.944838 mapwrite 0x016294 thru 0x01c49d (0x620a bytes) 000842 1740485178.951463 MIRROR_EXTEND to 6 mirrors 000843 1740485179.075440 mapread 0x003e5e thru 0x00de76 (0xa019 bytes) 000844 1740485179.084180 trunc from 0x02148f to 0x03271c (0x1128e bytes) 000846 1740485179.094927 read 0x028831 thru 0x0320d5 (0x98a5 bytes) 000848 1740485179.097586 trunc from 0x03271d to 0x007200 (0x2b51c bytes) 000850 1740485179.103460 mapread 0x003e7e thru 0x007200 (0x3383 bytes) 000852 1740485179.106353 fallocate 0x01d807 thru 0x0280d4 (0xa8ce bytes) 000853 1740485179.107921 read 0x0020ff thru 0x00463c (0x253e bytes) 000854 1740485179.108168 fallocate 0x003e07 thru 0x00c2c1 (0x84bb bytes) 000855 1740485179.110609 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000856 1740485179.151628 read 0x001017 thru 0x004c0b (0x3bf5 bytes) 000857 1740485179.153278 write 0x032313 thru 0x03ffff (0xdced bytes) 000859 1740485179.155544 punch from 0x034b2d to 0x037266 (0x273a bytes) 000860 1740485179.159081 fallocate 0x0377e2 thru 0x03ffff (0x881e bytes) 000861 1740485179.160408 mapread 0x007c18 thru 0x011e6a (0xa253 bytes) 000862 1740485179.165242 mapread 0x00fd1d thru 0x01f316 (0xf5fa bytes) 000863 1740485179.167747 fallocate 0x00e57d thru 0x01783c (0x92c0 bytes) 000864 1740485179.169267 read 0x00891a thru 0x00cc52 (0x4339 bytes) 000865 1740485179.169581 write 0x01bb77 thru 0x01dbf2 (0x207c bytes) 000866 1740485179.170155 fallocate 0x00c059 thru 0x0185c9 (0xc571 bytes) 000867 1740485179.171747 trunc from 0x040000 to 0x013a7f (0x2c580 bytes) 000868 1740485179.175223 mapread 0x00a2e0 thru 0x0117ea (0x750b bytes) 000869 1740485179.175602 fallocate 0x002b13 thru 0x01239c (0xf88a bytes) 000870 1740485179.177149 punch from 0x00c085 to 0x012826 (0x67a2 bytes) 000872 1740485179.178764 write 0x012938 thru 0x022562 (0xfc2b bytes) 000873 1740485179.181889 fallocate 0x0011ab thru 0x00e873 (0xd6c9 bytes) 000874 1740485179.183790 mapwrite 0x01636c thru 0x01e32a (0x7fbf bytes) 000875 1740485179.189989 trunc from 0x022563 to 0x02ea24 (0xc4c2 bytes) 000877 1740485179.194127 mapread 0x02e172 thru 0x02ea24 (0x8b3 bytes) 000879 1740485179.197191 trunc from 0x02ea25 to 0x005962 (0x290c2 bytes) 000880 1740485179.201110 trunc from 0x005963 to 0x03230c (0x2c9aa bytes) 000881 1740485179.204424 fallocate 0x00e3a2 thru 0x01a187 (0xbde6 bytes) 000882 1740485179.205902 mapwrite 0x03c91b thru 0x03ffff (0x36e5 bytes) 000883 1740485179.215351 mapread 0x00c937 thru 0x01297e (0x6048 bytes) 000884 1740485179.222264 write 0x028c20 thru 0x02aaae (0x1e8f bytes) 000885 1740485179.224704 trunc from 0x040000 to 0x039112 (0x6eed bytes) 000886 1740485179.228455 mapread 0x019780 thru 0x02084b (0x70cc bytes) 000887 1740485179.236431 punch from 0x00c671 to 0x015ae4 (0x9474 bytes) 000888 1740485179.237816 fallocate 0x03fd33 thru 0x03ffff (0x2cd bytes) 000889 1740485179.239172 trunc from 0x039113 to 0x02b0aa (0xe068 bytes) 000892 1740485179.243549 mapread 0x01d3e0 thru 0x01fddc (0x29fd bytes) 000894 1740485179.244473 write 0x018ccb thru 0x01cd3e (0x4074 bytes) 000895 1740485179.246845 punch from 0x020a4d to 0x0216a3 (0xc57 bytes) 000896 1740485179.248542 mapread 0x015f6d thru 0x01803d (0x20d1 bytes) 000898 1740485179.253050 fallocate 0x0356a1 thru 0x03e432 (0x8d92 bytes) 000899 1740485179.254867 write 0x01bee2 thru 0x0226b0 (0x67cf bytes) 000900 1740485179.256810 mapwrite 0x037cee thru 0x03af81 (0x3294 bytes) 000903 1740485179.271375 read 0x017a88 thru 0x0234bd (0xba36 bytes) 000904 1740485179.274943 punch from 0x00aca2 to 0x00d881 (0x2be0 bytes) 000905 1740485179.277186 write 0x0391f1 thru 0x0399c2 (0x7d2 bytes) 000906 1740485179.277572 fallocate 0x0078ea thru 0x01317b (0xb892 bytes) 000907 1740485179.279197 read 0x0223df thru 0x02e81d (0xc43f bytes) 000908 1740485179.279629 mapread 0x01c8ca thru 0x027004 (0xa73b bytes) 000909 1740485179.280240 read 0x00304e thru 0x003930 (0x8e3 bytes) 000910 1740485179.280636 write 0x039101 thru 0x03ffff (0x6eff bytes) 000912 1740485179.283313 MIRROR_EXTEND to 6 mirrors 000913 1740485179.408688 read 0x00f5da thru 0x0189f3 (0x941a bytes) 000914 1740485179.412712 mapread 0x018efd thru 0x02581c (0xc920 bytes) 000915 1740485179.422847 MIRROR_RESYNC 6 mirrors 000916 1740485179.665181 fallocate 0x0076a1 thru 0x008544 (0xea4 bytes) 000917 1740485179.674112 mapwrite 0x020b30 thru 0x023bb7 (0x3088 bytes) 000918 1740485179.683405 write 0x02f1ad thru 0x038706 (0x955a bytes) 000919 1740485179.688813 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000920 1740485179.743328 write 0x016df3 thru 0x01916c (0x237a bytes) 000921 1740485179.747896 mapread 0x02f1c8 thru 0x0333fc (0x4235 bytes) 000923 1740485179.750898 read 0x033e48 thru 0x036a61 (0x2c1a bytes) 000924 1740485179.751212 read 0x032fed thru 0x03ffff (0xd013 bytes) 000926 1740485179.754520 write 0x027f7e thru 0x02c7f1 (0x4874 bytes) 000927 1740485179.756518 MIRROR_EXTEND to 6 mirrors 000928 1740485179.870715 mapwrite 0x018c94 thru 0x022709 (0x9a76 bytes) 000929 1740485179.904258 write 0x0223f0 thru 0x02c597 (0xa1a8 bytes) 000930 1740485179.907845 trunc from 0x040000 to 0x027273 (0x18d8c bytes) 000934 1740485179.918239 write 0x02b0e7 thru 0x0372f4 (0xc20e bytes) 000935 1740485179.919339 write 0x025e3c thru 0x0316da (0xb89f bytes) 000937 1740485179.920288 punch from 0x033e20 to 0x0372f4 (0x34d5 bytes) 000938 1740485179.925155 read 0x01ac45 thru 0x02a4d9 (0xf895 bytes) 000939 1740485179.927297 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000940 1740485179.985345 punch from 0x024f6e to 0x03114e (0xc1e1 bytes) 000941 1740485179.987074 fallocate 0x00c7be thru 0x017ba9 (0xb3ec bytes) 000942 1740485179.988524 read 0x008ec1 thru 0x00df11 (0x5051 bytes) 000945 1740485179.991065 read 0x00e6f2 thru 0x015c6f (0x757e bytes) 000946 1740485179.993979 fallocate 0x01ad00 thru 0x01d086 (0x2387 bytes) 000947 1740485179.996186 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000948 1740485180.042344 trunc from 0x0372f5 to 0x0370a6 (0x24e bytes) 000949 1740485180.047187 mapread 0x02c699 thru 0x0370a6 (0xaa0e bytes) 000950 1740485180.054687 trunc from 0x0370a7 to 0x02654d (0x10b59 bytes) 000951 1740485180.059214 read 0x00a2d9 thru 0x011b5d (0x7885 bytes) 000952 1740485180.061951 punch from 0x004a3f to 0x00fa8f (0xb051 bytes) 000953 1740485180.063461 fallocate 0x02280e thru 0x0272ac (0x4a9f bytes) 000955 1740485180.065318 write 0x024a17 thru 0x027e91 (0x347b bytes) 000956 1740485180.069703 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000957 1740485180.118409 fallocate 0x02592e thru 0x03067d (0xad50 bytes) 000958 1740485180.119918 trunc from 0x027e92 to 0x013206 (0x14c8b bytes) 000959 1740485180.124317 trunc from 0x013207 to 0x036761 (0x2355b bytes) 000961 1740485180.127792 mapwrite 0x006309 thru 0x00982a (0x3522 bytes) 000962 1740485180.136420 fallocate 0x03d267 thru 0x03ffff (0x2d99 bytes) 000963 1740485180.137919 trunc from 0x036762 to 0x01ae18 (0x1b949 bytes) 000964 1740485180.141759 punch from 0x0092eb to 0x0135ae (0xa2c4 bytes) 000965 1740485180.145053 MIRROR_RESYNC 3 mirrors 000967 1740485180.308595 punch from 0x00c74b to 0x011e4a (0x5700 bytes) 000968 1740485180.316874 mapread 0x00df96 thru 0x013bfd (0x5c68 bytes) 000970 1740485180.322868 MIRROR_EXTEND to 4 mirrors 000971 1740485180.426996 MIRROR_EXTEND to 5 mirrors 000972 1740485180.532544 punch from 0x010b3d to 0x017f68 (0x742c bytes) 000973 1740485180.540361 read 0x01762b thru 0x01ae18 (0x37ee bytes) 000974 1740485180.542316 write 0x0226cf thru 0x025e14 (0x3746 bytes) 000976 1740485180.544302 trunc from 0x025e15 to 0x03bb7f (0x15d6b bytes) 000977 1740485180.553768 MIRROR_RESYNC 5 mirrors 000978 1740485180.778154 mapread 0x019d78 thru 0x0278e3 (0xdb6c bytes) 000980 1740485180.784266 mapwrite 0x01e6d0 thru 0x029b67 (0xb498 bytes) 000983 1740485180.811917 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xa06:0x0]: Invalid argument (22) 000984 1740485180.856147 trunc from 0x03bb80 to 0x03e655 (0x2ad6 bytes) 000985 1740485180.863630 trunc from 0x03e656 to 0x0251c9 (0x1948c bytes) 000986 1740485180.867128 punch from 0x01f6d0 to 0x0251c9 (0x5afa bytes) 000987 1740485180.868554 fallocate 0x03d330 thru 0x03f081 (0x1d52 bytes) 000988 1740485180.870045 trunc from 0x0251ca to 0x00002d (0x2519c bytes) 000989 1740485180.874304 write 0x0276fc thru 0x03539c (0xdca1 bytes) 000990 1740485180.875476 mapread 0x017a44 thru 0x018535 (0xaf2 bytes) 000992 1740485180.878765 fallocate 0x012cd8 thru 0x0171a3 (0x44cc bytes) 000993 1740485180.880462 trunc from 0x03539d to 0x02ee24 (0x6578 bytes) 000994 1740485180.884184 mapread 0x02129b thru 0x0264b2 (0x5218 bytes) 000995 1740485180.886610 write 0x000bc3 thru 0x00350a (0x2948 bytes) 000996 1740485180.889202 trunc from 0x02ee25 to 0x00ffcf (0x1ee55 bytes) 000997 1740485180.892884 punch from 0x007a93 to 0x00b25a (0x37c8 bytes) 000998 1740485180.894196 trunc from 0x00ffd0 to 0x01f644 (0xf675 bytes) 000999 1740485180.897903 write 0x0372a3 thru 0x03b4e2 (0x4240 bytes) 001000 1740485180.898640 mapread 0x00fd51 thru 0x01ce4b (0xd0fb bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 71: check mirror extend parent fid ==== 07:06:21 (1740485181) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000405:0xa3f:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 07:06:24 (1740485184) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre2 Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6395936 .. Extending file size to 2168928 .. done Extending file size to 7591360 .. Extending file size to 2389632 .. Extending file size to 7901184 .. Extending file size to 3325824 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5036832 .. Extending file size to 7352192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8457824 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8853920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8865408 .. Extending file size to 9102240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9204032 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9226656 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9259904 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 3990 3991 3993 3994 3997 failed 192.168.202.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre2 (opts:) 192.168.202.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:07:28 (1740485248) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre2 Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4966688 .. Extending file size to 1079424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8059808 .. Extending file size to 2337504 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6695552 .. Extending file size to 8540608 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6870624 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9021152 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8351648 .. failed Extending file size to 9306528 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9168608 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9238080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 8608 8609 8611 8613 8615 192.168.202.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre2 (opts:) 192.168.202.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:08:31 (1740485311) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2772 1284916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3372 1284316 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15860 3591160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21916 3585104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37776 7176264 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2772 1284916 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3372 1284316 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15860 3591160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21916 3585104 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37776 7176264 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0327563 s, 64.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.054214 s, 38.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:08:36 (1740485316) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0326363 s, 64.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 07:08:37 (1740485317) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:08:39 (1740485319) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0245157 s, 85.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xebe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xebd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xebe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:08:41 (1740485321) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0260792 s, 80.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:08:43 (1740485323) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.021288 s, 98.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xebf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xebf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xebf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xebf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xec0:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:09:05 (1740485345) Waiting for MDT destroys to complete oleg252-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg252-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg252-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136439 s, 76.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xec0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xec2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xec1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xec0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xec2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xec1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xec1:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:09:38 (1740485378) Waiting for MDT destroys to complete oleg252-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg252-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg252-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0216089 s, 97.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:10:10 (1740485410) Waiting for MDT destroys to complete oleg252-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg252-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg252-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg252-server: Pool lustre.test_204f created Adding targets to pool oleg252-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg252-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0215354 s, 97.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg252-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg252-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg252-server: Pool lustre.test_204f destroyed PASS 204f (45s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:10:55 (1740485455) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:10:57 (1740485457) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:10:58 (1740485458) oleg252-server: Pool lustre.test_206 created Adding targets to pool oleg252-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg252-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg252-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg252-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg252-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:11:12 (1740485472) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204221 s, 51.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105508 s, 99.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf83bd1e2 Mirror 3: 0xf83bd1e2 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:11:14 (1740485474) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:11:15 (1740485475) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:11:15 (1740485475) Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00799062 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2410000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d60000 Read this (4 bytes): BBBB 192.168.202.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg252-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:11:29 (1740485489) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00236324 s, 3.5 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:11:30 (1740485490) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898583 s, 117 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9a:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-02-25 07:11:31.000000000 -0500 Modify: 2025-02-25 07:11:31.000000000 -0500 Change: 2025-02-25 07:11:31.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:11:32 (1740485492) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00834418 s, 126 MB/s oleg252-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg252-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg252-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg252-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 oleg252-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg252-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg252-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:11:46 (1740485506) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0206323 s, 2.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00418336 s, 979 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 25 07:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 25 07:11 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 689 sec ============ 07:11:48 (1740485508) === sanity-flr: start cleanup 07:11:48 (1740485508) === === sanity-flr: finish cleanup 07:11:50 (1740485510) ===