-----============= acceptance-small: sanity-flr ============----- Mon Apr 7 05:56:26 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg212-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:56:30 (1744019790) === oleg212-client.virtnet: executing check_config_client /mnt/lustre oleg212-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg212-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012e65a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012e65a000.idle_timeout=debug disable quota as required oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:56:39 (1744019799) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanity-flr test 0a: lfs mirror create with -N option == 05:56:40 (1744019800) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:56:43 (1744019803) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:56:44 (1744019804) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:56:45 (1744019805) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:56:49 (1744019809) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:56:50 (1744019810) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:56:51 (1744019811) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012e65a000.stats=clear osc.lustre-OST0001-osc-ffff88012e65a000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:56:53 (1744019813) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:56:55 (1744019815) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:56:57 (1744019817) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:56:58 (1744019818) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 05:57:00 (1744019820) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:57:02 (1744019822) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:57:03 (1744019823) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:57:05 (1744019825) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:57:08 (1744019828) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:57:09 (1744019829) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.232168 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200112 s, 52.4 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:57:11 (1744019831) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.135211 s, 132 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:57:13 (1744019833) Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:57:39 (1744019859) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.145706 s, 64.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:57:48 (1744019868) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:58:08 (1744019888) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.73428 s, 71.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 7 05:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.245s user 0m0.001s sys 0m0.107s Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.245s user 0m0.000s sys 0m0.112s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.260s user 0m0.001s sys 0m0.104s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg212-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e65a000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:58:29 (1744019909) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:58:30 (1744019910) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:58:31 (1744019911) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 05:58:31 (1744019911) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 05:58:33 (1744019913) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.129419 s, 97.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:58:35 (1744019915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00881046 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00805219 s, 130 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:58:37 (1744019917) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:58:40 (1744019920) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000475298 s, 2.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000471197 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000603303 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000580745 s, 1.8 MB/s flv=10 olv=10 PASS 36d (1s) == sanity-flr test 37: mirror I/O API verification ======= 05:58:41 (1744019921) create test files with size 15754670 .. 0c9ef131b800c1bc100db3180f79b4cc - 05f4a062254a0bb46c3b8761c551f9b0 - 11b73ab0d98e443bf013471b8107e489 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 05:58:46 (1744019926) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:58:48 (1744019928) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:58:50 (1744019930) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155691 s, 67.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00769742 s, 136 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 05:58:52 (1744019932) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0208428 s, 101 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0786192 s, 53.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.032748 s, 64.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 05:58:54 (1744019934) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 05:58:54 (1744019934) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:58:55 (1744019935) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00842555 s, 124 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 05:58:57 (1744019937) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0677138 s, 46.5 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:58:59 (1744019939) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:59:01 (1744019941) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0731169 s, 143 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:59:24 (1744019964) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a3db6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0817259 s, 128 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aa656800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:59:27 (1744019967) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:59:33 (1744019973) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:59:39 (1744019979) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 05:59:40 (1744019980) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0651236 s, 48.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0465142 s, 67.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 8fc29b481278c7aab9a58e5fd12fd665 - mirror 1 checksum: 5977213c7e181e547f5712eda93c0641 - mirror 2 checksum: 8fc29b481278c7aab9a58e5fd12fd665 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 7 05:59:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 7 05:59:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8fc29b481278c7aab9a58e5fd12fd665 - mirror 1 checksum: 5977213c7e181e547f5712eda93c0641 - mirror 2 checksum: 5977213c7e181e547f5712eda93c0641 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 7 05:59:40 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 7 05:59:40 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:59:42 (1744019982) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00972221 s, 108 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:59:44 (1744019984) Failing mds1 on oleg212-server Stopping /mnt/lustre-mds1 (opts:) on oleg212-server 05:59:50 (1744019990) shut down facet: mds1 facet_host: oleg212-server facet_failover_host: oleg212-server Failover mds1 to oleg212-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 06:00:03 (1744020003) targets are mounted 06:00:03 (1744020003) facet_failover done oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00735378 s, 143 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:00:11 (1744020011) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00682588 s, 3.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00387693 s, 5.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:00:13 (1744020013) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00789487 s, 2.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00277164 s, 7.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:00:14 (1744020014) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00214059 s, 7.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:00:16 (1744020016) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00899573 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:00:18 (1744020018) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00139972 s, 2.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00634129 s, 646 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:00:19 (1744020019) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:00:22 (1744020022) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:00:40 (1744020040) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4d:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:00:57 (1744020057) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:01:13 (1744020073) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 06:01:15 (1744020075) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26279 mirror split pid 26280 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26279 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26280 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 06:02:16 (1744020136) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 137 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x2773b 000001 1744020137.226964 fallocate 0x01a3b2 thru 0x02773a (0xd389 bytes) fallocating to largest ever: 0x28d6d 000003 1744020137.230947 fallocate 0x01a738 thru 0x028d6c (0xe635 bytes) 000004 1744020137.232304 write 0x013a7a thru 0x0165e9 (0x2b70 bytes) 000006 1744020137.235702 mapread 0x01fd20 thru 0x023c3d (0x3f1e bytes) 000007 1744020137.239281 mapwrite 0x03d7a3 thru 0x03ffff (0x285d bytes) 000008 1744020137.252439 mapwrite 0x03220b thru 0x03ffff (0xddf5 bytes) 000009 1744020137.261482 mapread 0x033889 thru 0x033d30 (0x4a8 bytes) 000010 1744020137.261901 fallocate 0x00d80f thru 0x01513f (0x7931 bytes) 000011 1744020137.263154 trunc from 0x040000 to 0x009f0d (0x360f2 bytes) fallocating to largest ever: 0x3e22d 000012 1744020137.267341 fallocate 0x035a65 thru 0x03e22c (0x87c8 bytes) 000013 1744020137.269101 fallocate 0x02506b thru 0x02f558 (0xa4ee bytes) 000014 1744020137.270621 mapwrite 0x0393e1 thru 0x03c306 (0x2f26 bytes) 000015 1744020137.282255 mapwrite 0x0166c7 thru 0x02487b (0xe1b5 bytes) 000016 1744020137.292853 mapread 0x00d04e thru 0x019c0c (0xcbbf bytes) 000017 1744020137.294967 mapread 0x0171aa thru 0x020bf5 (0x9a4c bytes) 000018 1744020137.295335 read 0x00b1af thru 0x017e7a (0xcccc bytes) 000020 1744020137.295817 write 0x007a4e thru 0x00975a (0x1d0d bytes) 000021 1744020137.296217 write 0x026f63 thru 0x028c49 (0x1ce7 bytes) 000022 1744020137.298223 read 0x033f10 thru 0x039bc7 (0x5cb8 bytes) 000023 1744020137.298666 punch from 0x02a1c7 to 0x038d82 (0xebbc bytes) 000024 1744020137.300130 write 0x02ad38 thru 0x034f71 (0xa23a bytes) 000025 1744020137.303853 MIRROR_EXTEND to 2 mirrors 000026 1744020137.403604 trunc from 0x03c307 to 0x01a309 (0x21ffd bytes) 000027 1744020137.417293 write 0x011bc4 thru 0x016526 (0x4963 bytes) 000028 1744020137.419948 punch from 0x00c0d3 to 0x011b0a (0x5a38 bytes) 000029 1744020137.423250 fallocate 0x02e3b0 thru 0x03d0be (0xed0f bytes) 000031 1744020137.424837 write 0x0079c2 thru 0x011995 (0x9fd4 bytes) 000033 1744020137.428673 read 0x017f73 thru 0x01a309 (0x2397 bytes) 000034 1744020137.430098 mapwrite 0x0332a7 thru 0x03cff0 (0x9d4a bytes) 000035 1744020137.443348 trunc from 0x03cff1 to 0x00737a (0x35c76 bytes) 000036 1744020137.447024 read 0x004d08 thru 0x00737a (0x2673 bytes) 000039 1744020137.447798 read 0x00305d thru 0x0046e7 (0x168b bytes) 000041 1744020137.448436 mapread 0x0020f5 thru 0x00737a (0x5286 bytes) 000042 1744020137.448937 write 0x02c1f3 thru 0x02ec04 (0x2a12 bytes) 000045 1744020137.449815 fallocate 0x00a379 thru 0x00cafe (0x2786 bytes) 000046 1744020137.452355 MIRROR_RESYNC 2 mirrors 000047 1744020137.531933 write 0x03ffc7 thru 0x03ffff (0x39 bytes) 000048 1744020137.538769 fallocate 0x006517 thru 0x00c70c (0x61f6 bytes) 000049 1744020137.540388 mapread 0x0075b5 thru 0x00ad62 (0x37ae bytes) 000051 1744020137.544830 fallocate 0x02e614 thru 0x03450c (0x5ef9 bytes) 000052 1744020137.546332 trunc from 0x040000 to 0x036296 (0x9d69 bytes) 000053 1744020137.554245 punch from 0x01d7c0 to 0x02976b (0xbfac bytes) fallocating to largest ever: 0x40000 000054 1744020137.555798 fallocate 0x03c571 thru 0x03ffff (0x3a8f bytes) 000056 1744020137.557549 mapwrite 0x009bf9 thru 0x00e8c4 (0x4ccc bytes) 000057 1744020137.564597 punch from 0x00f0a9 to 0x016967 (0x78bf bytes) 000058 1744020137.566137 punch from 0x00fdba to 0x012e65 (0x30ac bytes) 000059 1744020137.567511 mapwrite 0x0345cc thru 0x03c3eb (0x7e20 bytes) 000061 1744020137.581341 MIRROR_EXTEND to 3 mirrors 000062 1744020137.682166 read 0x007b22 thru 0x0168bb (0xed9a bytes) 000065 1744020137.684905 fallocate 0x01e7eb thru 0x02c3cb (0xdbe1 bytes) 000066 1744020137.693231 fallocate 0x032da4 thru 0x03ffff (0xd25c bytes) 000068 1744020137.694860 punch from 0x027715 to 0x036b77 (0xf463 bytes) 000069 1744020137.696416 read 0x020629 thru 0x0299a0 (0x9378 bytes) 000070 1744020137.701686 MIRROR_RESYNC 3 mirrors 000072 1744020137.823805 read 0x014c87 thru 0x023f57 (0xf2d1 bytes) 000073 1744020137.826790 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000074 1744020137.874480 write 0x02e28c thru 0x035879 (0x75ee bytes) 000075 1744020137.883520 read 0x03a9d7 thru 0x03c3eb (0x1a15 bytes) 000076 1744020137.885134 write 0x002379 thru 0x00a2d4 (0x7f5c bytes) 000077 1744020137.887525 write 0x0123f0 thru 0x01d2e2 (0xaef3 bytes) 000078 1744020137.890641 trunc from 0x03c3ec to 0x025727 (0x16cc4 bytes) 000079 1744020137.899168 trunc from 0x025728 to 0x02f788 (0xa061 bytes) 000080 1744020137.902577 mapread 0x004e60 thru 0x00baf8 (0x6c99 bytes) 000081 1744020137.904163 MIRROR_RESYNC 2 mirrors 000082 1744020137.998103 write 0x02ea49 thru 0x035d03 (0x72bb bytes) 000083 1744020138.005519 read 0x0111bc thru 0x013356 (0x219b bytes) 000085 1744020138.007235 fallocate 0x031969 thru 0x0319f7 (0x8f bytes) 000086 1744020138.008834 read 0x00ac3e thru 0x011c70 (0x7033 bytes) 000087 1744020138.011771 mapwrite 0x033ae4 thru 0x0388bc (0x4dd9 bytes) 000088 1744020138.026350 mapread 0x02f246 thru 0x034cf1 (0x5aac bytes) 000089 1744020138.026785 fallocate 0x01f3f3 thru 0x0253e2 (0x5ff0 bytes) 000090 1744020138.028049 mapread 0x02de15 thru 0x0388bc (0xaaa8 bytes) 000091 1744020138.028610 write 0x0325e7 thru 0x03d2c3 (0xacdd bytes) 000093 1744020138.029884 read 0x02886f thru 0x033f4f (0xb6e1 bytes) 000094 1744020138.030347 trunc from 0x03d2c4 to 0x01106d (0x2c256 bytes) 000095 1744020138.034211 write 0x004a08 thru 0x010da3 (0xc39c bytes) 000097 1744020138.035212 write 0x019b89 thru 0x0237e7 (0x9c5f bytes) 000098 1744020138.035957 write 0x0040cb thru 0x013af7 (0xfa2d bytes) 000099 1744020138.038162 trunc from 0x0237e8 to 0x0155ce (0xe219 bytes) 000100 1744020138.042508 trunc from 0x0155cf to 0x039f2d (0x2495f bytes) 000101 1744020138.045563 write 0x015d6e thru 0x0188b8 (0x2b4b bytes) 000102 1744020138.049411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000103 1744020138.079387 fallocate 0x0257c2 thru 0x032bfb (0xd43a bytes) 000104 1744020138.080907 mapread 0x0023de thru 0x004386 (0x1fa9 bytes) 000106 1744020138.086202 write 0x02158d thru 0x026c0b (0x567f bytes) 000110 1744020138.090827 write 0x01ee7f thru 0x02c6ff (0xd881 bytes) 000111 1744020138.092655 trunc from 0x039f2e to 0x03c2f3 (0x23c6 bytes) 000112 1744020138.095914 punch from 0x00f2df to 0x019a07 (0xa729 bytes) 000114 1744020138.097600 read 0x033aec thru 0x037df0 (0x4305 bytes) 000115 1744020138.099521 mapwrite 0x002f98 thru 0x00d688 (0xa6f1 bytes) 000116 1744020138.105693 fallocate 0x01c4c9 thru 0x02b8b7 (0xf3ef bytes) 000117 1744020138.107082 read 0x00cf3b thru 0x01c340 (0xf406 bytes) 000118 1744020138.109360 punch from 0x02503a to 0x02997a (0x4941 bytes) 000119 1744020138.110598 mapread 0x020026 thru 0x0271a0 (0x717b bytes) 000120 1744020138.114196 write 0x021f68 thru 0x0225e2 (0x67b bytes) 000122 1744020138.115063 write 0x0218f9 thru 0x02fa5d (0xe165 bytes) 000124 1744020138.116162 read 0x02673e thru 0x02d488 (0x6d4b bytes) 000125 1744020138.116406 write 0x002893 thru 0x0078c1 (0x502f bytes) 000126 1744020138.117019 trunc from 0x03c2f4 to 0x02b5d5 (0x10d1e bytes) 000128 1744020138.120878 write 0x0059d0 thru 0x00c65c (0x6c8d bytes) 000129 1744020138.121565 trunc from 0x02b5d6 to 0x007785 (0x23e50 bytes) 000130 1744020138.125301 trunc from 0x007786 to 0x00656f (0x1216 bytes) 000131 1744020138.128781 read 0x0013f4 thru 0x003313 (0x1f20 bytes) 000132 1744020138.129051 read 0x004c76 thru 0x00656f (0x18fa bytes) 000133 1744020138.129291 fallocate 0x015a68 thru 0x0201a7 (0xa740 bytes) 000135 1744020138.130849 trunc from 0x006570 to 0x02daa1 (0x27532 bytes) 000136 1744020138.134077 write 0x003466 thru 0x00bca9 (0x8844 bytes) 000137 1744020138.135608 fallocate 0x03fc09 thru 0x03ffff (0x3f7 bytes) 000138 1744020138.136896 trunc from 0x040000 to 0x0228f1 (0x1d70e bytes) 000140 1744020138.140265 trunc from 0x0228f2 to 0x02516f (0x287e bytes) 000141 1744020138.143408 fallocate 0x02123e thru 0x02c5c9 (0xb38c bytes) 000142 1744020138.146918 read 0x00cb1e thru 0x0173bf (0xa8a2 bytes) 000143 1744020138.149685 mapwrite 0x0162b9 thru 0x02181b (0xb563 bytes) 000144 1744020138.156572 mapwrite 0x00fecb thru 0x016d38 (0x6e6e bytes) 000146 1744020138.161387 read 0x0135de thru 0x01aa1d (0x7440 bytes) 000147 1744020138.161855 fallocate 0x00a4e1 thru 0x019443 (0xef63 bytes) 000149 1744020138.163784 mapwrite 0x000a91 thru 0x006790 (0x5d00 bytes) 000150 1744020138.170120 mapwrite 0x010731 thru 0x01bfa0 (0xb870 bytes) 000152 1744020138.175019 read 0x00ec61 thru 0x01bb49 (0xcee9 bytes) 000153 1744020138.175393 read 0x00b807 thru 0x00e68c (0x2e86 bytes) 000156 1744020138.176642 read 0x02a0be thru 0x02c5c9 (0x250c bytes) 000158 1744020138.177449 mapread 0x026b5c thru 0x02a620 (0x3ac5 bytes) 000159 1744020138.177973 mapread 0x001904 thru 0x0063d5 (0x4ad2 bytes) 000160 1744020138.178319 fallocate 0x00ed11 thru 0x0129ca (0x3cba bytes) 000161 1744020138.179832 trunc from 0x02c5ca to 0x0224bf (0xa10a bytes) 000162 1744020138.183637 mapwrite 0x01d496 thru 0x01f336 (0x1ea1 bytes) 000163 1744020138.187983 write 0x0065e7 thru 0x00be5b (0x5875 bytes) 000164 1744020138.188569 mapwrite 0x036e94 thru 0x03ffff (0x916c bytes) 000165 1744020138.199754 mapwrite 0x00feb9 thru 0x018870 (0x89b8 bytes) 000168 1744020138.204463 punch from 0x031f8d to 0x03d8e9 (0xb95d bytes) 000170 1744020138.206128 punch from 0x039705 to 0x03ffff (0x68fb bytes) 000171 1744020138.207187 fallocate 0x006482 thru 0x007793 (0x1312 bytes) 000172 1744020138.208286 fallocate 0x037ab1 thru 0x03c5ff (0x4b4f bytes) 000173 1744020138.209398 trunc from 0x040000 to 0x00493e (0x3b6c1 bytes) 000174 1744020138.213150 write 0x024273 thru 0x02e8c1 (0xa64f bytes) 000175 1744020138.214040 punch from 0x023cdf to 0x0291f5 (0x5517 bytes) 000176 1744020138.217081 write 0x00beff thru 0x01997c (0xda7e bytes) 000177 1744020138.220878 mapread 0x01fd45 thru 0x01fea8 (0x164 bytes) 000178 1744020138.222475 fallocate 0x02d4bc thru 0x0343c0 (0x6f05 bytes) 000179 1744020138.223986 read 0x01eb34 thru 0x02a4b2 (0xb97f bytes) 000181 1744020138.226667 mapread 0x0265d4 thru 0x027a4a (0x1477 bytes) 000182 1744020138.227189 read 0x017fad thru 0x019782 (0x17d6 bytes) 000184 1744020138.227774 write 0x039b82 thru 0x03bb87 (0x2006 bytes) 000185 1744020138.228377 mapwrite 0x031ae8 thru 0x03ffff (0xe518 bytes) 000186 1744020138.241607 trunc from 0x040000 to 0x0063ad (0x39c52 bytes) 000187 1744020138.245568 trunc from 0x0063ae to 0x03a52f (0x34182 bytes) 000188 1744020138.248778 punch from 0x029566 to 0x02d226 (0x3cc1 bytes) 000189 1744020138.250231 trunc from 0x03a530 to 0x013517 (0x27018 bytes) 000191 1744020138.254021 fallocate 0x009684 thru 0x017cce (0xe64b bytes) 000192 1744020138.255653 read 0x012f29 thru 0x017b13 (0x4beb bytes) 000193 1744020138.258668 trunc from 0x017ccf to 0x021e54 (0xa186 bytes) 000194 1744020138.261765 write 0x01365d thru 0x01577b (0x211f bytes) 000195 1744020138.262236 punch from 0x01115e to 0x01c8c4 (0xb767 bytes) 000196 1744020138.265371 fallocate 0x0392b7 thru 0x03ffff (0x6d49 bytes) 000197 1744020138.266684 mapread 0x0086d6 thru 0x01259d (0x9ec8 bytes) 000198 1744020138.270859 read 0x00ecb8 thru 0x01a20a (0xb553 bytes) 000200 1744020138.273319 read 0x016a01 thru 0x01ed57 (0x8357 bytes) 000201 1744020138.273590 mapwrite 0x02cd04 thru 0x03100a (0x4307 bytes) 000202 1744020138.279080 fallocate 0x02f141 thru 0x03ea99 (0xf959 bytes) 000203 1744020138.280521 read 0x00aea8 thru 0x01109c (0x61f5 bytes) 000204 1744020138.280813 read 0x02a443 thru 0x02df13 (0x3ad1 bytes) 000205 1744020138.281063 mapwrite 0x0227e1 thru 0x023ff9 (0x1819 bytes) 000206 1744020138.284538 mapread 0x00ed3a thru 0x019718 (0xa9df bytes) 000208 1744020138.285315 mapread 0x007537 thru 0x0153b9 (0xde83 bytes) 000211 1744020138.286251 mapread 0x00071e thru 0x00876e (0x8051 bytes) 000213 1744020138.286941 trunc from 0x040000 to 0x021b0a (0x1e4f5 bytes) 000214 1744020138.290615 write 0x03c944 thru 0x03d322 (0x9df bytes) 000215 1744020138.291181 read 0x02810c thru 0x035e6c (0xdd61 bytes) 000216 1744020138.293347 mapwrite 0x000720 thru 0x00da1b (0xd2fc bytes) 000217 1744020138.299515 punch from 0x01903f to 0x0206bd (0x767f bytes) 000220 1744020138.301588 mapwrite 0x0355aa thru 0x03960b (0x4062 bytes) 000221 1744020138.305894 fallocate 0x01cdb2 thru 0x01e81c (0x1a6b bytes) 000222 1744020138.307317 read 0x004ade thru 0x009d3d (0x5260 bytes) 000224 1744020138.308016 write 0x021ca7 thru 0x02a7d3 (0x8b2d bytes) 000225 1744020138.308604 fallocate 0x0054ec thru 0x00968b (0x41a0 bytes) 000227 1744020138.310234 punch from 0x03b070 to 0x03d322 (0x22b3 bytes) 000228 1744020138.312750 mapread 0x028f09 thru 0x029686 (0x77e bytes) 000229 1744020138.313126 mapread 0x0254e2 thru 0x02f6e5 (0xa204 bytes) 000230 1744020138.313554 mapwrite 0x02c539 thru 0x035465 (0x8f2d bytes) 000232 1744020138.320824 mapread 0x03ad28 thru 0x03d322 (0x25fb bytes) 000233 1744020138.324031 mapwrite 0x031dc4 thru 0x03935d (0x759a bytes) 000234 1744020138.329046 punch from 0x019e15 to 0x028c99 (0xee85 bytes) 000236 1744020138.330610 fallocate 0x03cea7 thru 0x03ffff (0x3159 bytes) 000238 1744020138.333265 punch from 0x02f4ba to 0x032037 (0x2b7e bytes) 000239 1744020138.334620 mapread 0x00090f thru 0x0014e5 (0xbd7 bytes) 000240 1744020138.336265 punch from 0x00af5f to 0x00ff2d (0x4fcf bytes) 000241 1744020138.337582 write 0x019cda thru 0x020ea2 (0x71c9 bytes) 000242 1744020138.341898 read 0x01340f thru 0x023346 (0xff38 bytes) 000244 1744020138.344550 write 0x030a54 thru 0x039a91 (0x903e bytes) 000246 1744020138.345624 write 0x010122 thru 0x01b18d (0xb06c bytes) 000248 1744020138.346536 mapwrite 0x018611 thru 0x021d59 (0x9749 bytes) 000250 1744020138.352682 trunc from 0x040000 to 0x03ff89 (0x76 bytes) 000251 1744020138.356386 fallocate 0x03db2b thru 0x03ffff (0x24d5 bytes) 000252 1744020138.357624 punch from 0x01dfc1 to 0x0202fb (0x233b bytes) 000254 1744020138.359226 punch from 0x002d92 to 0x0079e7 (0x4c56 bytes) 000255 1744020138.360544 write 0x03a036 thru 0x03d930 (0x38fb bytes) 000256 1744020138.362100 write 0x020e81 thru 0x02c530 (0xb6b0 bytes) 000259 1744020138.364222 read 0x03264e thru 0x03879b (0x614e bytes) 000260 1744020138.364478 read 0x00e319 thru 0x01d8d9 (0xf5c1 bytes) 000261 1744020138.366430 read 0x0395ce thru 0x03ffff (0x6a32 bytes) 000262 1744020138.366817 fallocate 0x02ec5a thru 0x03a324 (0xb6cb bytes) 000263 1744020138.368236 mapread 0x0003b2 thru 0x00ce41 (0xca90 bytes) 000264 1744020138.368816 trunc from 0x040000 to 0x03e6cb (0x1934 bytes) 000265 1744020138.372276 mapwrite 0x01ecda thru 0x0291db (0xa502 bytes) 000266 1744020138.378238 mapwrite 0x0102dc thru 0x019b35 (0x985a bytes) 000267 1744020138.382250 mapwrite 0x02172a thru 0x0242d9 (0x2bb0 bytes) 000268 1744020138.385547 fallocate 0x032a0a thru 0x03c304 (0x98fb bytes) 000271 1744020138.387603 read 0x00f044 thru 0x01a2d7 (0xb294 bytes) 000272 1744020138.387927 mapwrite 0x0000ee thru 0x008a4e (0x8961 bytes) 000273 1744020138.392847 punch from 0x02d42a to 0x02e672 (0x1249 bytes) 000274 1744020138.394215 read 0x00ca35 thru 0x014ba3 (0x816f bytes) 000275 1744020138.394644 mapread 0x002ef9 thru 0x00db68 (0xac70 bytes) 000276 1744020138.396295 trunc from 0x03e6cc to 0x00e2e3 (0x303e8 bytes) 000278 1744020138.400786 mapwrite 0x00dccd thru 0x01d71b (0xfa4f bytes) 000279 1744020138.412976 mapwrite 0x0078a8 thru 0x013c2b (0xc384 bytes) 000281 1744020138.418399 write 0x013435 thru 0x01f343 (0xbf0f bytes) 000282 1744020138.419109 punch from 0x010448 to 0x01549b (0x5054 bytes) 000283 1744020138.422269 mapwrite 0x00567f thru 0x0120e0 (0xca62 bytes) 000284 1744020138.432417 mapwrite 0x020183 thru 0x02ce4e (0xcccc bytes) 000285 1744020138.442864 mapread 0x00bb0f thru 0x014c74 (0x9166 bytes) 000287 1744020138.445868 fallocate 0x022872 thru 0x0245a0 (0x1d2f bytes) 000289 1744020138.448630 MIRROR_EXTEND to 2 mirrors 000290 1744020138.542017 read 0x02af0a thru 0x02ce4e (0x1f45 bytes) 000291 1744020138.544966 trunc from 0x02ce4f to 0x02310f (0x9d3f bytes) 000292 1744020138.558177 trunc from 0x023110 to 0x00f6b9 (0x13a56 bytes) 000293 1744020138.562447 write 0x002f8d thru 0x00acbb (0x7d2f bytes) 000294 1744020138.565013 fallocate 0x02339b thru 0x02cfaa (0x9c10 bytes) 000295 1744020138.566436 trunc from 0x00f6ba to 0x037161 (0x27aa8 bytes) 000296 1744020138.569847 read 0x02947f thru 0x0301b8 (0x6d3a bytes) 000297 1744020138.573200 mapwrite 0x039552 thru 0x03ffff (0x6aae bytes) 000298 1744020138.585316 trunc from 0x040000 to 0x03afba (0x5045 bytes) 000300 1744020138.588904 mapread 0x008ba9 thru 0x00a8b2 (0x1d0a bytes) 000301 1744020138.589317 mapwrite 0x00d0dc thru 0x013dad (0x6cd2 bytes) 000302 1744020138.594481 trunc from 0x03afbb to 0x003fdd (0x36fdd bytes) 000303 1744020138.598248 mapwrite 0x011072 thru 0x01c5b4 (0xb543 bytes) 000305 1744020138.620257 MIRROR_EXTEND to 3 mirrors 000306 1744020138.704417 mapwrite 0x011bf5 thru 0x015236 (0x3642 bytes) 000307 1744020138.733121 MIRROR_RESYNC 3 mirrors 000309 1744020138.840412 MIRROR_RESYNC 3 mirrors 000312 1744020138.882270 MIRROR_RESYNC 3 mirrors 000313 1744020138.915896 mapread 0x0144fc thru 0x01c5b4 (0x80b9 bytes) 000315 1744020138.919813 write 0x01fbfb thru 0x02651d (0x6923 bytes) 000316 1744020138.928563 write 0x022eff thru 0x02ec23 (0xbd25 bytes) 000317 1744020138.929336 mapwrite 0x03dc5f thru 0x03ffff (0x23a1 bytes) 000318 1744020138.945122 mapread 0x00a3dc thru 0x010d64 (0x6989 bytes) 000319 1744020138.947680 punch from 0x03855b to 0x03baf4 (0x359a bytes) 000320 1744020138.949055 punch from 0x00d571 to 0x010270 (0x2d00 bytes) 000321 1744020138.950256 mapread 0x035ec6 thru 0x039470 (0x35ab bytes) 000322 1744020138.952332 punch from 0x00a8d0 to 0x018f3f (0xe670 bytes) 000323 1744020138.953654 punch from 0x00dd03 to 0x017b92 (0x9e90 bytes) 000324 1744020138.954820 fallocate 0x031b32 thru 0x0373c2 (0x5891 bytes) 000326 1744020138.956346 mapread 0x0274e2 thru 0x02e84b (0x736a bytes) 000327 1744020138.957848 punch from 0x0386a0 to 0x03ffff (0x7960 bytes) 000328 1744020138.958999 punch from 0x017208 to 0x01b26a (0x4063 bytes) 000329 1744020138.960224 mapread 0x030389 thru 0x03de18 (0xda90 bytes) 000330 1744020138.963327 punch from 0x028b6d to 0x02a767 (0x1bfb bytes) 000331 1744020138.966807 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000332 1744020139.010576 punch from 0x03ce5d to 0x03ffff (0x31a3 bytes) 000333 1744020139.011947 read 0x022f68 thru 0x02de1a (0xaeb3 bytes) 000334 1744020139.014205 punch from 0x02c049 to 0x036f52 (0xaf0a bytes) 000337 1744020139.016093 read 0x02c70d thru 0x038037 (0xb92b bytes) 000339 1744020139.019293 fallocate 0x029daa thru 0x0310db (0x7332 bytes) 000340 1744020139.021661 MIRROR_RESYNC 2 mirrors 000341 1744020139.125152 MIRROR_EXTEND to 3 mirrors 000342 1744020139.237930 read 0x02bc01 thru 0x033bea (0x7fea bytes) 000343 1744020139.239957 fallocate 0x015ba7 thru 0x01e601 (0x8a5b bytes) 000344 1744020139.248128 trunc from 0x040000 to 0x035d8f (0xa270 bytes) 000345 1744020139.252654 punch from 0x0325b5 to 0x0332ea (0xd36 bytes) 000346 1744020139.253807 punch from 0x0323d8 to 0x035d8f (0x39b8 bytes) 000348 1744020139.255413 mapwrite 0x02363c thru 0x02ba39 (0x83fe bytes) 000349 1744020139.265182 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000351 1744020139.306586 punch from 0x01637f to 0x01b947 (0x55c9 bytes) 000352 1744020139.308107 fallocate 0x03a35d thru 0x03ffff (0x5ca3 bytes) 000354 1744020139.309566 mapwrite 0x01c474 thru 0x020c26 (0x47b3 bytes) 000355 1744020139.319907 MIRROR_EXTEND to 3 mirrors 000356 1744020139.419088 fallocate 0x020e31 thru 0x02df26 (0xd0f6 bytes) 000358 1744020139.426454 read 0x01e158 thru 0x02629d (0x8146 bytes) 000360 1744020139.428559 read 0x017600 thru 0x01e7f1 (0x71f2 bytes) 000361 1744020139.431246 write 0x013b1e thru 0x0164c9 (0x29ac bytes) 000362 1744020139.432687 mapread 0x02f311 thru 0x02f47a (0x16a bytes) 000364 1744020139.433447 read 0x024d66 thru 0x02bbf5 (0x6e90 bytes) 000366 1744020139.434027 mapwrite 0x01c233 thru 0x022d80 (0x6b4e bytes) 000367 1744020139.439150 mapread 0x0163a7 thru 0x01d7cd (0x7427 bytes) 000368 1744020139.439541 trunc from 0x035d90 to 0x03dbdd (0x7e4e bytes) 000369 1744020139.446611 punch from 0x00b85e to 0x01a61a (0xedbd bytes) 000370 1744020139.448040 read 0x035921 thru 0x03dbdd (0x82bd bytes) 000371 1744020139.450513 mapwrite 0x02615e thru 0x02ee85 (0x8d28 bytes) 000373 1744020139.456220 write 0x006dfc thru 0x016347 (0xf54c bytes) 000374 1744020139.457084 read 0x00ed86 thru 0x015492 (0x670d bytes) 000375 1744020139.457371 mapwrite 0x00a7dc thru 0x019ad2 (0xf2f7 bytes) 000376 1744020139.462580 trunc from 0x03dbde to 0x01f3b2 (0x1e82b bytes) 000377 1744020139.465982 read 0x01e59b thru 0x01f3b2 (0xe18 bytes) 000378 1744020139.466239 write 0x01997f thru 0x024cd8 (0xb35a bytes) 000379 1744020139.468229 MIRROR_RESYNC 3 mirrors 000381 1744020139.570453 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000383 1744020139.620681 mapread 0x020745 thru 0x024cd8 (0x4594 bytes) 000384 1744020139.624480 mapwrite 0x008787 thru 0x00ad28 (0x25a2 bytes) 000385 1744020139.650267 mapread 0x01e62e thru 0x024cd8 (0x66ab bytes) 000386 1744020139.653093 punch from 0x00f18b to 0x013935 (0x47ab bytes) 000387 1744020139.656203 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000389 1744020139.684452 read 0x01c2a0 thru 0x01c714 (0x475 bytes) 000390 1744020139.685916 write 0x03a7fb thru 0x03c005 (0x180b bytes) 000391 1744020139.687364 trunc from 0x03c006 to 0x01d65e (0x1e9a7 bytes) 000392 1744020139.693827 mapread 0x002eda thru 0x0094ed (0x6614 bytes) 000393 1744020139.697821 mapwrite 0x02b15b thru 0x02f62b (0x44d1 bytes) 000395 1744020139.710323 mapread 0x0030e1 thru 0x007aff (0x4a1f bytes) 000396 1744020139.711783 mapread 0x009995 thru 0x019498 (0xfb04 bytes) 000397 1744020139.716937 punch from 0x028a39 to 0x02de7d (0x5445 bytes) 000398 1744020139.718389 read 0x02efe4 thru 0x02f62b (0x648 bytes) 000399 1744020139.718636 write 0x01e8ff thru 0x02e004 (0xf706 bytes) 000400 1744020139.720127 trunc from 0x02f62c to 0x00856e (0x270bd bytes) 000401 1744020139.724073 fallocate 0x03c3c4 thru 0x03ffff (0x3c3c bytes) 000403 1744020139.725977 punch from 0x0028ed to 0x0055e7 (0x2cfb bytes) 000404 1744020139.727196 mapwrite 0x0038bc thru 0x00f54e (0xbc93 bytes) 000405 1744020139.738797 write 0x0380e3 thru 0x03ffff (0x7f1d bytes) 000406 1744020139.739492 write 0x00126e thru 0x00ece7 (0xda7a bytes) 000407 1744020139.740179 punch from 0x00fd16 to 0x01240b (0x26f6 bytes) 000408 1744020139.741403 trunc from 0x040000 to 0x01da3a (0x225c5 bytes) 000409 1744020139.744520 mapwrite 0x005f72 thru 0x014713 (0xe7a2 bytes) 000411 1744020139.754088 read 0x003bac thru 0x006061 (0x24b6 bytes) 000412 1744020139.754449 trunc from 0x01da3b to 0x026d95 (0x935b bytes) 000413 1744020139.758582 MIRROR_EXTEND to 2 mirrors 000415 1744020139.851794 mapwrite 0x03bac4 thru 0x03ffff (0x453c bytes) 000416 1744020139.870855 trunc from 0x040000 to 0x03f60e (0x9f1 bytes) 000420 1744020139.875580 mapread 0x028c6c thru 0x036d15 (0xe0aa bytes) 000422 1744020139.879314 fallocate 0x018529 thru 0x01fdf1 (0x78c9 bytes) 000423 1744020139.881805 MIRROR_EXTEND to 3 mirrors 000424 1744020139.974319 read 0x037d01 thru 0x03f60e (0x790e bytes) 000425 1744020139.976701 write 0x02430a thru 0x02becb (0x7bc2 bytes) 000427 1744020139.987880 MIRROR_EXTEND to 4 mirrors 000428 1744020140.104463 trunc from 0x03f60f to 0x005067 (0x3a5a7 bytes) 000429 1744020140.114748 mapwrite 0x004fea thru 0x01004b (0xb062 bytes) 000430 1744020140.129828 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000432 1744020140.171227 MIRROR_RESYNC 3 mirrors 000434 1744020140.260636 punch from 0x0010d0 to 0x002405 (0x1336 bytes) 000436 1744020140.268316 trunc from 0x01004c to 0x00967d (0x69ce bytes) 000437 1744020140.273062 write 0x00a840 thru 0x01991e (0xf0df bytes) 000438 1744020140.274365 read 0x011080 thru 0x01991e (0x889f bytes) 000439 1744020140.275150 trunc from 0x01991f to 0x03b666 (0x21d48 bytes) 000440 1744020140.280007 MIRROR_RESYNC 3 mirrors 000441 1744020140.376820 read 0x030911 thru 0x038598 (0x7c88 bytes) 000443 1744020140.379373 punch from 0x015d6a to 0x023329 (0xd5c0 bytes) 000444 1744020140.387086 fallocate 0x02f7d2 thru 0x02fbec (0x41b bytes) 000445 1744020140.388518 write 0x013f8f thru 0x01ec5c (0xacce bytes) 000446 1744020140.391039 mapwrite 0x034ab0 thru 0x03ffff (0xb550 bytes) 000447 1744020140.405289 fallocate 0x02300f thru 0x02504b (0x203d bytes) 000448 1744020140.407784 write 0x01200c thru 0x0155ef (0x35e4 bytes) 000450 1744020140.410097 fallocate 0x02fdf3 thru 0x039a99 (0x9ca7 bytes) 000451 1744020140.411455 punch from 0x034f67 to 0x03cfeb (0x8085 bytes) 000452 1744020140.412998 mapread 0x024031 thru 0x02a8c4 (0x6894 bytes) 000453 1744020140.415528 trunc from 0x040000 to 0x0243fd (0x1bc02 bytes) 000454 1744020140.420179 MIRROR_RESYNC 3 mirrors 000455 1744020140.524553 mapwrite 0x00c561 thru 0x01aadd (0xe57d bytes) 000456 1744020140.555559 punch from 0x01ccad to 0x023819 (0x6b6d bytes) 000457 1744020140.557264 mapread 0x00e400 thru 0x018f30 (0xab31 bytes) 000458 1744020140.558992 MIRROR_EXTEND to 4 mirrors 000460 1744020140.657437 read 0x0233aa thru 0x0243fd (0x1054 bytes) 000461 1744020140.660490 fallocate 0x005acf thru 0x01569a (0xfbcc bytes) 000462 1744020140.668608 mapread 0x01a257 thru 0x0215a8 (0x7352 bytes) 000463 1744020140.672684 read 0x0174c5 thru 0x01aadb (0x3617 bytes) 000466 1744020140.675407 punch from 0x008f42 to 0x00c20f (0x32ce bytes) 000467 1744020140.676825 mapwrite 0x036f98 thru 0x03a5f3 (0x365c bytes) 000468 1744020140.692064 punch from 0x00b31a to 0x014330 (0x9017 bytes) 000469 1744020140.693675 mapwrite 0x007d9d thru 0x00e895 (0x6af9 bytes) 000470 1744020140.706477 read 0x001d69 thru 0x004cc3 (0x2f5b bytes) 000471 1744020140.706964 mapread 0x01b3cd thru 0x0278d3 (0xc507 bytes) 000472 1744020140.707465 fallocate 0x00a009 thru 0x00c037 (0x202f bytes) 000473 1744020140.709007 read 0x02f208 thru 0x03a5f3 (0xb3ec bytes) 000474 1744020140.709324 mapwrite 0x03595b thru 0x03c4c9 (0x6b6f bytes) 000475 1744020140.721668 write 0x03904e thru 0x03f609 (0x65bc bytes) 000476 1744020140.722310 mapwrite 0x001bd8 thru 0x00efbd (0xd3e6 bytes) 000477 1744020140.729547 read 0x023051 thru 0x031dbf (0xed6f bytes) 000478 1744020140.730136 fallocate 0x03e777 thru 0x03ffff (0x1889 bytes) 000479 1744020140.731539 read 0x000e85 thru 0x00f7f7 (0xe973 bytes) 000480 1744020140.733207 MIRROR_RESYNC 4 mirrors 000481 1744020140.899754 write 0x02be4e thru 0x034138 (0x82eb bytes) 000482 1744020140.909315 fallocate 0x00814e thru 0x00e56d (0x6420 bytes) 000483 1744020140.910941 read 0x019151 thru 0x027943 (0xe7f3 bytes) 000485 1744020140.913439 read 0x01bb00 thru 0x022d07 (0x7208 bytes) 000486 1744020140.913753 trunc from 0x03f60a to 0x022cc1 (0x1c948 bytes) 000487 1744020140.922237 mapread 0x01d240 thru 0x022cc1 (0x5a82 bytes) 000488 1744020140.922780 mapwrite 0x010f5a thru 0x014593 (0x363a bytes) 000489 1744020140.930026 write 0x008c3d thru 0x0178c6 (0xec8a bytes) 000490 1744020140.930853 read 0x0135f1 thru 0x0159de (0x23ee bytes) 000491 1744020140.931116 mapread 0x00d9a7 thru 0x0154c0 (0x7b1a bytes) 000492 1744020140.931490 fallocate 0x031f45 thru 0x036c60 (0x4d1c bytes) 000493 1744020140.932823 punch from 0x016d27 to 0x0182db (0x15b5 bytes) 000494 1744020140.937876 MIRROR_EXTEND to 5 mirrors 000495 1744020141.031906 mapwrite 0x00f4dc thru 0x01715c (0x7c81 bytes) 000496 1744020141.062974 trunc from 0x022cc2 to 0x0126b6 (0x1060b bytes) 000497 1744020141.071599 punch from 0x0082f5 to 0x0095d8 (0x12e4 bytes) 000498 1744020141.072933 write 0x010482 thru 0x011810 (0x138f bytes) 000499 1744020141.074787 MIRROR_RESYNC 5 mirrors 000500 1744020141.174938 mapwrite 0x025ba4 thru 0x03550c (0xf969 bytes) 000501 1744020141.196512 fallocate 0x0039d3 thru 0x00d725 (0x9d53 bytes) 000502 1744020141.197884 mapread 0x01cc7e thru 0x01f1a6 (0x2529 bytes) 000503 1744020141.201112 MIRROR_RESYNC 5 mirrors 000504 1744020141.325309 punch from 0x01ac43 to 0x0277b5 (0xcb73 bytes) 000505 1744020141.333151 trunc from 0x03550d to 0x02ca09 (0x8b03 bytes) 000506 1744020141.341277 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000507 1744020141.381884 mapread 0x02a8d9 thru 0x02ca09 (0x2131 bytes) 000508 1744020141.386393 fallocate 0x034e92 thru 0x03aaf2 (0x5c61 bytes) 000509 1744020141.387903 punch from 0x02125b to 0x02ca09 (0xb7af bytes) 000511 1744020141.389515 write 0x00487a thru 0x005d30 (0x14b7 bytes) 000512 1744020141.393099 punch from 0x02485a to 0x02b56f (0x6d16 bytes) 000513 1744020141.395401 MIRROR_EXTEND to 5 mirrors 000516 1744020141.490245 trunc from 0x02ca0a to 0x00aafd (0x21f0c bytes) 000517 1744020141.503767 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000518 1744020141.545454 mapwrite 0x01e3a5 thru 0x022a59 (0x46b5 bytes) 000519 1744020141.558252 mapwrite 0x018024 thru 0x01a975 (0x2952 bytes) 000520 1744020141.563491 punch from 0x01693c to 0x01fe87 (0x954c bytes) 000521 1744020141.564667 mapwrite 0x02d00a thru 0x0392a5 (0xc29c bytes) 000523 1744020141.577820 trunc from 0x0392a6 to 0x01e06a (0x1b23b bytes) 000524 1744020141.581118 punch from 0x019b4c to 0x01e06a (0x451f bytes) 000526 1744020141.582681 trunc from 0x01e06b to 0x0049a8 (0x196c2 bytes) 000527 1744020141.586045 fallocate 0x0326ac thru 0x037e56 (0x57ab bytes) 000528 1744020141.587378 read 0x00045e thru 0x0049a8 (0x454b bytes) 000531 1744020141.589706 write 0x00e389 thru 0x016326 (0x7f9e bytes) 000534 1744020141.591096 read 0x015b05 thru 0x016326 (0x822 bytes) 000535 1744020141.591481 trunc from 0x016327 to 0x02efe5 (0x18cbf bytes) 000536 1744020141.596390 MIRROR_RESYNC 4 mirrors 000540 1744020141.733460 write 0x02e013 thru 0x032d3a (0x4d28 bytes) 000541 1744020141.740496 punch from 0x01cfb1 to 0x020117 (0x3167 bytes) 000542 1744020141.741531 write 0x00452b thru 0x01168e (0xd164 bytes) 000543 1744020141.744063 punch from 0x030521 to 0x032d3a (0x281a bytes) 000545 1744020141.747429 mapread 0x02e489 thru 0x02febc (0x1a34 bytes) 000546 1744020141.747926 mapwrite 0x02c7de thru 0x036c16 (0xa439 bytes) 000547 1744020141.763025 mapwrite 0x01d06f thru 0x02149a (0x442c bytes) 000548 1744020141.768289 fallocate 0x01a538 thru 0x020f69 (0x6a32 bytes) 000549 1744020141.769578 punch from 0x00af1e to 0x019f85 (0xf068 bytes) 000552 1744020141.771435 mapread 0x001793 thru 0x004166 (0x29d4 bytes) 000553 1744020141.774482 MIRROR_RESYNC 4 mirrors 000554 1744020141.907677 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000555 1744020141.957061 read 0x004535 thru 0x0091b9 (0x4c85 bytes) 000556 1744020141.958891 mapread 0x01e6ac thru 0x02b206 (0xcb5b bytes) 000557 1744020141.964207 fallocate 0x016847 thru 0x020942 (0xa0fc bytes) 000558 1744020141.971563 read 0x03642f thru 0x036c16 (0x7e8 bytes) 000559 1744020141.974272 mapread 0x00fcb6 thru 0x00fe03 (0x14e bytes) 000560 1744020141.974681 punch from 0x02c18e to 0x0315f2 (0x5465 bytes) 000562 1744020141.976200 write 0x0044e2 thru 0x013ffa (0xfb19 bytes) 000563 1744020141.978099 mapread 0x0168b8 thru 0x0215ea (0xad33 bytes) 000565 1744020141.978901 trunc from 0x036c17 to 0x03d8aa (0x6c94 bytes) 000566 1744020141.986371 mapread 0x00ee9c thru 0x01ccdc (0xde41 bytes) 000567 1744020141.986969 mapread 0x01def5 thru 0x02be21 (0xdf2d bytes) 000568 1744020141.988439 MIRROR_EXTEND to 4 mirrors 000569 1744020142.100464 fallocate 0x02dc57 thru 0x0392cf (0xb679 bytes) 000572 1744020142.108497 write 0x011db2 thru 0x01aa83 (0x8cd2 bytes) 000573 1744020142.112006 mapwrite 0x0046f6 thru 0x00de4b (0x9756 bytes) 000574 1744020142.120268 punch from 0x01c878 to 0x024fd1 (0x875a bytes) 000575 1744020142.122830 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000576 1744020142.165310 fallocate 0x0268b0 thru 0x02c91d (0x606e bytes) 000579 1744020142.167505 mapwrite 0x0359d0 thru 0x03ffff (0xa630 bytes) 000582 1744020142.182639 mapread 0x006337 thru 0x0153b0 (0xf07a bytes) 000587 1744020142.188483 mapwrite 0x03a91e thru 0x03ffff (0x56e2 bytes) 000588 1744020142.193214 mapread 0x0375f7 thru 0x03ff71 (0x897b bytes) 000590 1744020142.194200 write 0x02a289 thru 0x032502 (0x827a bytes) 000591 1744020142.196853 write 0x00c870 thru 0x010bcd (0x435e bytes) 000592 1744020142.197450 mapwrite 0x031fa3 thru 0x03bd88 (0x9de6 bytes) 000593 1744020142.204136 MIRROR_RESYNC 3 mirrors 000595 1744020142.297235 fallocate 0x031ea2 thru 0x03641b (0x457a bytes) 000596 1744020142.304730 write 0x029b53 thru 0x038261 (0xe70f bytes) 000597 1744020142.308784 MIRROR_EXTEND to 4 mirrors 000598 1744020142.408610 mapread 0x028950 thru 0x031724 (0x8dd5 bytes) 000600 1744020142.417762 fallocate 0x023892 thru 0x025fa7 (0x2716 bytes) 000601 1744020142.425937 punch from 0x02306e to 0x02e2ee (0xb281 bytes) 000602 1744020142.427301 fallocate 0x026fd3 thru 0x031d21 (0xad4f bytes) 000605 1744020142.429061 trunc from 0x040000 to 0x03ecda (0x1325 bytes) 000606 1744020142.437215 trunc from 0x03ecdb to 0x0122fb (0x2c9df bytes) 000608 1744020142.440930 write 0x02d8c2 thru 0x03c3a8 (0xeae7 bytes) 000609 1744020142.441948 read 0x00c58f thru 0x01184e (0x52c0 bytes) 000610 1744020142.445130 MIRROR_EXTEND to 5 mirrors 000611 1744020142.541470 write 0x010e8e thru 0x0158f5 (0x4a68 bytes) 000612 1744020142.552286 MIRROR_EXTEND to 6 mirrors 000613 1744020142.646245 mapread 0x01d669 thru 0x025395 (0x7d2d bytes) 000614 1744020142.652865 trunc from 0x03c3a9 to 0x037187 (0x5221 bytes) 000616 1744020142.665593 punch from 0x0072f8 to 0x00d133 (0x5e3c bytes) 000617 1744020142.666939 read 0x02fad5 thru 0x035101 (0x562d bytes) 000618 1744020142.668735 mapread 0x035f50 thru 0x037187 (0x1238 bytes) 000619 1744020142.670384 write 0x005f82 thru 0x0063ec (0x46b bytes) 000620 1744020142.673751 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000621 1744020142.717942 punch from 0x02bb5f to 0x031cc4 (0x6166 bytes) 000623 1744020142.719534 mapwrite 0x00f438 thru 0x0107ca (0x1393 bytes) 000624 1744020142.726089 mapwrite 0x03f79f thru 0x03ffff (0x861 bytes) 000626 1744020142.736193 trunc from 0x040000 to 0x02272a (0x1d8d5 bytes) 000627 1744020142.740236 punch from 0x01c1d1 to 0x01d737 (0x1567 bytes) 000628 1744020142.741302 mapread 0x01f89a thru 0x02272a (0x2e91 bytes) 000630 1744020142.743448 mapwrite 0x0142a7 thru 0x0231e6 (0xef40 bytes) 000631 1744020142.754068 mapwrite 0x000992 thru 0x002b69 (0x21d8 bytes) 000634 1744020142.761277 write 0x02dec3 thru 0x02e42f (0x56d bytes) 000635 1744020142.761749 read 0x0064a1 thru 0x012491 (0xbff1 bytes) 000637 1744020142.762249 mapwrite 0x03716e thru 0x03d249 (0x60dc bytes) 000638 1744020142.772272 mapwrite 0x03765d thru 0x03ffff (0x89a3 bytes) 000639 1744020142.781211 mapread 0x01a444 thru 0x01fe5e (0x5a1b bytes) 000642 1744020142.782090 fallocate 0x006af0 thru 0x00f3a9 (0x88ba bytes) 000643 1744020142.783352 write 0x006dd4 thru 0x015bff (0xee2c bytes) 000644 1744020142.784357 mapwrite 0x000aa0 thru 0x00af80 (0xa4e1 bytes) 000645 1744020142.789392 fallocate 0x024a26 thru 0x02dccf (0x92aa bytes) 000646 1744020142.790630 trunc from 0x040000 to 0x022eee (0x1d111 bytes) 000647 1744020142.795018 write 0x011862 thru 0x01f2d2 (0xda71 bytes) 000649 1744020142.797173 MIRROR_EXTEND to 6 mirrors 000650 1744020142.881988 punch from 0x0080d1 to 0x0130a9 (0xafd9 bytes) 000652 1744020142.891185 read 0x0009dd thru 0x00dc23 (0xd247 bytes) 000653 1744020142.893418 trunc from 0x022eef to 0x028f73 (0x6085 bytes) 000654 1744020142.901528 fallocate 0x00b5d7 thru 0x016f3a (0xb964 bytes) 000655 1744020142.904035 MIRROR_EXTEND to 7 mirrors 000656 1744020143.005294 trunc from 0x028f74 to 0x00ba83 (0x1d4f0 bytes) 000657 1744020143.017601 mapwrite 0x03408e thru 0x03fd5b (0xbcce bytes) 000658 1744020143.032646 write 0x00d95d thru 0x0172d9 (0x997d bytes) 000661 1744020143.037077 read 0x029083 thru 0x0332a8 (0xa226 bytes) 000662 1744020143.039256 fallocate 0x0392ac thru 0x03a3fa (0x114f bytes) 000663 1744020143.041857 MIRROR_EXTEND to 8 mirrors 000664 1744020143.150095 mapread 0x0241c8 thru 0x0272e6 (0x311f bytes) 000665 1744020143.156767 punch from 0x019523 to 0x0200a1 (0x6b7f bytes) 000668 1744020143.165298 write 0x03eeef thru 0x03ffff (0x1111 bytes) 000670 1744020143.168374 fallocate 0x0154a4 thru 0x0234c7 (0xe024 bytes) 000671 1744020143.169798 read 0x02c17e thru 0x034cad (0x8b30 bytes) 000674 1744020143.172365 read 0x02feb9 thru 0x03ae76 (0xafbe bytes) 000675 1744020143.175165 trunc from 0x040000 to 0x022101 (0x1defe bytes) 000676 1744020143.183438 write 0x001fad thru 0x00fda4 (0xddf8 bytes) 000678 1744020143.184510 read 0x021bff thru 0x022101 (0x503 bytes) 000679 1744020143.184845 write 0x0044c4 thru 0x00f48b (0xafc8 bytes) 000680 1744020143.185576 read 0x00ce84 thru 0x0197e5 (0xc962 bytes) 000681 1744020143.186131 trunc from 0x022102 to 0x0324a4 (0x103a3 bytes) 000682 1744020143.189261 read 0x01ce1b thru 0x0242e2 (0x74c8 bytes) 000684 1744020143.191646 mapread 0x0086dd thru 0x011d93 (0x96b7 bytes) 000686 1744020143.192501 mapwrite 0x00192b thru 0x008f91 (0x7667 bytes) 000687 1744020143.199109 punch from 0x00a4f7 to 0x0135fd (0x9107 bytes) 000688 1744020143.200603 mapread 0x02050c thru 0x023519 (0x300e bytes) 000689 1744020143.201056 read 0x027e21 thru 0x0324a4 (0xa684 bytes) 000690 1744020143.201421 write 0x0246d3 thru 0x02600e (0x193c bytes) 000691 1744020143.201961 punch from 0x015148 to 0x020735 (0xb5ee bytes) 000692 1744020143.203627 write 0x029f23 thru 0x033373 (0x9451 bytes) 000693 1744020143.205250 read 0x005078 thru 0x006048 (0xfd1 bytes) 000694 1744020143.205503 punch from 0x0329d4 to 0x033373 (0x9a0 bytes) 000695 1744020143.210184 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000697 1744020143.254996 MIRROR_RESYNC 7 mirrors 000698 1744020143.572785 trunc from 0x033374 to 0x0024cf (0x30ea4 bytes) 000699 1744020143.585029 mapwrite 0x01fd4d thru 0x0289c2 (0x8c76 bytes) 000700 1744020143.600249 MIRROR_RESYNC 7 mirrors 000701 1744020143.761282 mapwrite 0x03bb11 thru 0x03bcfd (0x1ed bytes) 000702 1744020143.778112 fallocate 0x001857 thru 0x0093ba (0x7b64 bytes) 000703 1744020143.779448 mapwrite 0x00eeec thru 0x0151d1 (0x62e6 bytes) 000704 1744020143.787311 punch from 0x0355da to 0x03bcfd (0x6724 bytes) 000705 1744020143.788959 mapwrite 0x0241ea thru 0x02fde9 (0xbc00 bytes) 000706 1744020143.799141 MIRROR_RESYNC 7 mirrors 000707 1744020144.074247 trunc from 0x03bcfe to 0x035bc3 (0x613a bytes) 000708 1744020144.085416 MIRROR_EXTEND to 8 mirrors 000710 1744020144.210985 mapwrite 0x0193d6 thru 0x021892 (0x84bd bytes) 000711 1744020144.245218 fallocate 0x003015 thru 0x0096db (0x66c7 bytes) 000712 1744020144.248098 MIRROR_EXTEND to 9 mirrors 000713 1744020144.368105 mapwrite 0x012192 thru 0x018fc8 (0x6e37 bytes) 000714 1744020144.400479 fallocate 0x03073d thru 0x034185 (0x3a49 bytes) 000715 1744020144.402845 punch from 0x020904 to 0x02f854 (0xef51 bytes) 000716 1744020144.404220 read 0x024fd6 thru 0x02acfb (0x5d26 bytes) 000717 1744020144.406167 mapread 0x0182bf thru 0x01cf9e (0x4ce0 bytes) 000718 1744020144.408412 trunc from 0x035bc4 to 0x023d53 (0x11e70 bytes) 000719 1744020144.413601 read 0x01b255 thru 0x0207f3 (0x559f bytes) 000720 1744020144.413987 trunc from 0x023d54 to 0x038745 (0x149f2 bytes) 000721 1744020144.417299 punch from 0x0102c1 to 0x0182af (0x7fef bytes) 000722 1744020144.418792 trunc from 0x038746 to 0x0350be (0x3687 bytes) 000723 1744020144.422316 trunc from 0x0350bf to 0x03c10e (0x7050 bytes) 000725 1744020144.426043 trunc from 0x03c10f to 0x03b637 (0xad7 bytes) 000726 1744020144.429333 write 0x03844a thru 0x03ffff (0x7bb6 bytes) 000727 1744020144.432436 write 0x0241b3 thru 0x0255ac (0x13fa bytes) 000728 1744020144.432889 read 0x004d51 thru 0x00faea (0xad9a bytes) 000730 1744020144.433509 fallocate 0x02d562 thru 0x034e8a (0x7929 bytes) 000731 1744020144.435047 punch from 0x01873d to 0x025710 (0xcfd4 bytes) 000733 1744020144.440503 MIRROR_RESYNC 9 mirrors 000734 1744020144.827914 fallocate 0x03e732 thru 0x03ffff (0x18ce bytes) 000735 1744020144.835176 mapwrite 0x00e40f thru 0x01651d (0x810f bytes) 000736 1744020144.844862 punch from 0x0160f9 to 0x01b767 (0x566f bytes) 000737 1744020144.846332 mapwrite 0x021b97 thru 0x02697c (0x4de6 bytes) 000738 1744020144.852795 punch from 0x03a4fc to 0x03ffff (0x5b04 bytes) 000739 1744020144.853977 mapwrite 0x01a2c2 thru 0x028015 (0xdd54 bytes) 000741 1744020144.861003 mapread 0x0155fc thru 0x01c9bc (0x73c1 bytes) 000742 1744020144.862843 trunc from 0x040000 to 0x02b88c (0x14773 bytes) 000745 1744020144.871798 fallocate 0x010fdd thru 0x01f5d6 (0xe5fa bytes) 000747 1744020144.873523 read 0x0068d0 thru 0x00b173 (0x48a4 bytes) 000749 1744020144.875150 mapwrite 0x005902 thru 0x014ffb (0xf6fa bytes) 000750 1744020144.883700 fallocate 0x03fd5a thru 0x03ffff (0x2a6 bytes) 000751 1744020144.885023 mapread 0x014b5f thru 0x01defd (0x939f bytes) 000752 1744020144.885496 mapwrite 0x039c86 thru 0x03ffff (0x637a bytes) 000753 1744020144.896590 punch from 0x002062 to 0x00a745 (0x86e4 bytes) 000754 1744020144.898102 fallocate 0x02d302 thru 0x02ff20 (0x2c1f bytes) 000755 1744020144.900812 MIRROR_RESYNC 9 mirrors 000756 1744020145.182744 fallocate 0x01f8ca thru 0x02c2a0 (0xc9d7 bytes) 000757 1744020145.190934 trunc from 0x040000 to 0x03db1b (0x24e4 bytes) 000759 1744020145.195538 mapwrite 0x00a50b thru 0x011fce (0x7ac4 bytes) 000760 1744020145.205096 write 0x03aac2 thru 0x03e7b0 (0x3cef bytes) 000761 1744020145.207315 fallocate 0x03b1fa thru 0x03cb67 (0x196e bytes) 000762 1744020145.208629 trunc from 0x03e7b1 to 0x03aff6 (0x37ba bytes) 000763 1744020145.213108 read 0x00b70c thru 0x01afb3 (0xf8a8 bytes) 000764 1744020145.215357 fallocate 0x030502 thru 0x037522 (0x7021 bytes) 000765 1744020145.217117 mapwrite 0x0078fb thru 0x010114 (0x881a bytes) 000766 1744020145.224994 mapread 0x0111f8 thru 0x020186 (0xef8f bytes) 000767 1744020145.227501 write 0x00d8a8 thru 0x01cdc1 (0xf51a bytes) 000768 1744020145.228550 mapread 0x00189b thru 0x00f0a1 (0xd807 bytes) 000769 1744020145.230507 punch from 0x00592e to 0x012025 (0xc6f8 bytes) 000770 1744020145.234887 mapwrite 0x01690e thru 0x0233a3 (0xca96 bytes) 000771 1744020145.240976 punch from 0x007e5e to 0x00cf9a (0x513d bytes) 000772 1744020145.242361 punch from 0x0008b1 to 0x005de8 (0x5538 bytes) 000773 1744020145.243651 fallocate 0x00b4ba thru 0x00eb72 (0x36b9 bytes) 000774 1744020145.245156 mapread 0x006570 thru 0x008da9 (0x283a bytes) 000777 1744020145.249320 write 0x02bfb5 thru 0x039f79 (0xdfc5 bytes) 000778 1744020145.252292 mapwrite 0x0132ac thru 0x013e58 (0xbad bytes) 000779 1744020145.257066 mapwrite 0x01b190 thru 0x02701c (0xbe8d bytes) 000781 1744020145.263487 mapread 0x0284b3 thru 0x02e7a5 (0x62f3 bytes) 000783 1744020145.264389 punch from 0x00ff97 to 0x01a496 (0xa500 bytes) 000785 1744020145.267825 MIRROR_RESYNC 9 mirrors 000787 1744020145.492228 mapwrite 0x014340 thru 0x015175 (0xe36 bytes) 000788 1744020145.519634 trunc from 0x03aff7 to 0x002de5 (0x38211 bytes) 000789 1744020145.532671 mapread 0x0007ac thru 0x002de5 (0x263a bytes) 000790 1744020145.535593 mapwrite 0x019600 thru 0x023ddf (0xa7e0 bytes) 000791 1744020145.547415 mapread 0x00b416 thru 0x010de4 (0x59cf bytes) 000793 1744020145.549841 mapwrite 0x011e77 thru 0x01aff9 (0x9183 bytes) 000795 1744020145.554786 punch from 0x004f49 to 0x00fdf2 (0xaeaa bytes) 000796 1744020145.556173 trunc from 0x023de0 to 0x00696c (0x1d473 bytes) 000797 1744020145.559939 mapread 0x00637a thru 0x00696c (0x5f3 bytes) 000798 1744020145.562689 write 0x026377 thru 0x02a6a4 (0x432e bytes) 000799 1744020145.563361 read 0x002326 thru 0x00aa8a (0x8765 bytes) 000800 1744020145.565399 mapread 0x02a35b thru 0x02a6a4 (0x34a bytes) 000801 1744020145.565845 punch from 0x00141d to 0x002d58 (0x193c bytes) 000803 1744020145.567463 punch from 0x01981c to 0x01ebd9 (0x53be bytes) 000804 1744020145.568515 fallocate 0x037c1a thru 0x03ffff (0x83e6 bytes) 000805 1744020145.569908 write 0x02353e thru 0x02f690 (0xc153 bytes) 000806 1744020145.573405 mapread 0x03b136 thru 0x03ffff (0x4eca bytes) 000807 1744020145.575504 fallocate 0x03281b thru 0x03ff52 (0xd738 bytes) 000808 1744020145.576902 fallocate 0x022e4d thru 0x026761 (0x3915 bytes) 000809 1744020145.578212 mapread 0x03d0ba thru 0x03ffff (0x2f46 bytes) 000811 1744020145.578818 punch from 0x01280a to 0x0188df (0x60d6 bytes) 000812 1744020145.579995 punch from 0x019e88 to 0x026b60 (0xccd9 bytes) 000813 1744020145.583332 write 0x03bf39 thru 0x03ffff (0x40c7 bytes) 000814 1744020145.583858 mapread 0x013310 thru 0x02170f (0xe400 bytes) 000815 1744020145.587786 write 0x025435 thru 0x028e06 (0x39d2 bytes) 000816 1744020145.588253 read 0x025bae thru 0x02af2b (0x537e bytes) 000817 1744020145.588522 read 0x0020de thru 0x0048f5 (0x2818 bytes) 000818 1744020145.590567 fallocate 0x020071 thru 0x022aa9 (0x2a39 bytes) 000820 1744020145.592085 read 0x0273be thru 0x02871b (0x135e bytes) 000821 1744020145.592430 punch from 0x00e484 to 0x017262 (0x8ddf bytes) 000822 1744020145.593636 write 0x00692a thru 0x012e36 (0xc50d bytes) 000823 1744020145.595282 fallocate 0x01849e thru 0x01bd8a (0x38ed bytes) 000827 1744020145.597301 mapwrite 0x006573 thru 0x00edb3 (0x8841 bytes) 000829 1744020145.603177 punch from 0x039f8a to 0x03aeca (0xf41 bytes) 000830 1744020145.604436 punch from 0x01019c to 0x01dd6d (0xdbd2 bytes) 000831 1744020145.610143 MIRROR_RESYNC 9 mirrors 000833 1744020145.937350 mapread 0x020b89 thru 0x022f56 (0x23ce bytes) 000834 1744020145.941935 write 0x004139 thru 0x00a9cc (0x6894 bytes) 000835 1744020145.951628 read 0x03521b thru 0x03ffff (0xade5 bytes) 000836 1744020145.953897 mapwrite 0x024576 thru 0x02f827 (0xb2b2 bytes) 000837 1744020145.963896 read 0x033e93 thru 0x03cafe (0x8c6c bytes) 000838 1744020145.964237 write 0x03482c thru 0x03ffff (0xb7d4 bytes) 000839 1744020145.965184 fallocate 0x01e873 thru 0x02674d (0x7edb bytes) 000840 1744020145.966806 mapread 0x023680 thru 0x028813 (0x5194 bytes) 000841 1744020145.967423 punch from 0x03e756 to 0x03f352 (0xbfd bytes) 000842 1744020145.971571 read 0x021a18 thru 0x031312 (0xf8fb bytes) 000843 1744020145.971982 write 0x0361d6 thru 0x03cf11 (0x6d3c bytes) 000844 1744020145.972592 mapwrite 0x02df22 thru 0x03cb9c (0xec7b bytes) 000845 1744020145.980598 punch from 0x0252cf to 0x02ca59 (0x778b bytes) 000847 1744020145.982909 punch from 0x00321d to 0x0073c0 (0x41a4 bytes) 000848 1744020145.984241 read 0x02c4da thru 0x02c887 (0x3ae bytes) 000850 1744020145.988447 MIRROR_EXTEND to 10 mirrors 000853 1744020146.106897 punch from 0x01453a to 0x01571e (0x11e5 bytes) 000854 1744020146.115332 fallocate 0x03e588 thru 0x03f034 (0xaad bytes) 000856 1744020146.117255 mapread 0x02f50a thru 0x0396ee (0xa1e5 bytes) 000857 1744020146.123883 mapread 0x025be3 thru 0x02ff88 (0xa3a6 bytes) 000858 1744020146.126250 punch from 0x00968c to 0x00e660 (0x4fd5 bytes) 000859 1744020146.127815 read 0x032ff1 thru 0x03fa79 (0xca89 bytes) 000860 1744020146.128124 mapwrite 0x038161 thru 0x03dec4 (0x5d64 bytes) 000861 1744020146.134132 mapwrite 0x031541 thru 0x03dd92 (0xc852 bytes) 000862 1744020146.138953 trunc from 0x040000 to 0x0070c6 (0x38f39 bytes) 000863 1744020146.145076 mapread 0x006d0a thru 0x0070c6 (0x3bd bytes) 000864 1744020146.148355 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000865 1744020146.200129 MIRROR_RESYNC 9 mirrors 000866 1744020146.400563 punch from 0x0011c0 to 0x0070c6 (0x5f07 bytes) 000867 1744020146.409194 punch from 0x005cab to 0x0070c6 (0x141c bytes) 000868 1744020146.410455 fallocate 0x01afee thru 0x01ed1f (0x3d32 bytes) 000869 1744020146.413151 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000870 1744020146.456168 mapwrite 0x022d24 thru 0x02ac59 (0x7f36 bytes) 000872 1744020146.470375 read 0x01705d thru 0x01f845 (0x87e9 bytes) 000873 1744020146.472462 trunc from 0x02ac5a to 0x0135c0 (0x17699 bytes) 000874 1744020146.476335 mapread 0x00016e thru 0x00292c (0x27bf bytes) 000875 1744020146.479872 mapread 0x0076bc thru 0x008a7c (0x13c1 bytes) 000876 1744020146.480230 fallocate 0x006e01 thru 0x01079d (0x999d bytes) 000877 1744020146.481740 fallocate 0x02ede2 thru 0x03bd72 (0xcf91 bytes) 000878 1744020146.483074 fallocate 0x010e6e thru 0x020e07 (0xff9a bytes) 000879 1744020146.484428 fallocate 0x03b542 thru 0x03ffff (0x4abe bytes) 000882 1744020146.486390 fallocate 0x03ef32 thru 0x03ffff (0x10ce bytes) 000883 1744020146.487828 trunc from 0x040000 to 0x018f9b (0x27064 bytes) 000884 1744020146.491511 mapread 0x00e838 thru 0x010756 (0x1f1f bytes) 000885 1744020146.491989 trunc from 0x018f9c to 0x018033 (0xf68 bytes) 000886 1744020146.495116 fallocate 0x030088 thru 0x0380d9 (0x8052 bytes) 000887 1744020146.496565 read 0x00848d thru 0x0165af (0xe123 bytes) 000889 1744020146.499677 read 0x00d644 thru 0x01a5c8 (0xcf85 bytes) 000890 1744020146.500100 mapread 0x0326c9 thru 0x0380d9 (0x5a11 bytes) 000891 1744020146.500504 read 0x034b2a thru 0x0380d9 (0x35b0 bytes) 000892 1744020146.502030 MIRROR_EXTEND to 9 mirrors 000893 1744020146.607709 write 0x03c1c6 thru 0x03ffff (0x3e3a bytes) 000895 1744020146.617259 mapread 0x01a156 thru 0x01c9ad (0x2858 bytes) 000896 1744020146.621752 punch from 0x007af0 to 0x013a25 (0xbf36 bytes) 000897 1744020146.623314 mapwrite 0x0200c9 thru 0x02da04 (0xd93c bytes) 000898 1744020146.633244 trunc from 0x040000 to 0x013de7 (0x2c218 bytes) 000899 1744020146.642641 trunc from 0x013de8 to 0x03b71f (0x27938 bytes) 000900 1744020146.645907 mapread 0x020961 thru 0x02b2b1 (0xa951 bytes) 000901 1744020146.651152 write 0x018488 thru 0x01b61b (0x3194 bytes) 000902 1744020146.654450 write 0x03dc14 thru 0x03ffff (0x23ec bytes) 000903 1744020146.655071 mapread 0x00bd06 thru 0x013702 (0x79fd bytes) 000904 1744020146.658140 write 0x013141 thru 0x0155ee (0x24ae bytes) 000905 1744020146.660175 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000906 1744020146.715338 write 0x006f9b thru 0x00b889 (0x48ef bytes) 000908 1744020146.719659 mapread 0x02ef53 thru 0x02f4e1 (0x58f bytes) 000909 1744020146.722329 mapwrite 0x03237b thru 0x03b1a0 (0x8e26 bytes) 000910 1744020146.730275 trunc from 0x040000 to 0x01db0b (0x224f4 bytes) 000911 1744020146.734117 fallocate 0x0212ad thru 0x0279b9 (0x670d bytes) 000912 1744020146.736426 mapwrite 0x0202bf thru 0x02cde3 (0xcb25 bytes) 000913 1744020146.749411 mapread 0x029e68 thru 0x02cde3 (0x2f7c bytes) 000914 1744020146.750025 read 0x001950 thru 0x00d16b (0xb81c bytes) 000915 1744020146.753184 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000916 1744020146.804356 punch from 0x029df8 to 0x02cde3 (0x2fec bytes) 000917 1744020146.806093 punch from 0x00d5b8 to 0x013f0c (0x6955 bytes) 000918 1744020146.807926 punch from 0x01ec18 to 0x0229e6 (0x3dcf bytes) 000919 1744020146.810443 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000920 1744020146.859491 read 0x003005 thru 0x010b82 (0xdb7e bytes) 000921 1744020146.862898 mapwrite 0x01def5 thru 0x0236e1 (0x57ed bytes) 000922 1744020146.873159 read 0x000227 thru 0x009348 (0x9122 bytes) 000923 1744020146.876502 MIRROR_EXTEND to 7 mirrors 000925 1744020146.995443 punch from 0x00da46 to 0x012d5e (0x5319 bytes) 000926 1744020147.004339 trunc from 0x02cde4 to 0x015cf4 (0x170ef bytes) 000927 1744020147.014335 mapread 0x010a0d thru 0x013531 (0x2b25 bytes) 000928 1744020147.019346 mapread 0x0134dc thru 0x015cf4 (0x2819 bytes) 000929 1744020147.019892 mapread 0x014b41 thru 0x015cf4 (0x11b4 bytes) 000930 1744020147.020467 mapread 0x0055a5 thru 0x00e1f5 (0x8c51 bytes) 000931 1744020147.023032 punch from 0x00c735 to 0x015cf4 (0x95c0 bytes) 000932 1744020147.024542 mapread 0x00cafc thru 0x015cf4 (0x91f9 bytes) 000933 1744020147.028189 trunc from 0x015cf5 to 0x01c66a (0x6976 bytes) 000934 1744020147.032132 mapread 0x01592c thru 0x01c66a (0x6d3f bytes) 000935 1744020147.035402 mapwrite 0x034e03 thru 0x03b932 (0x6b30 bytes) 000936 1744020147.047384 mapread 0x00b961 thru 0x0163d4 (0xaa74 bytes) 000937 1744020147.048072 read 0x02b4cd thru 0x02e091 (0x2bc5 bytes) 000938 1744020147.049777 write 0x00f73e thru 0x01e0f4 (0xe9b7 bytes) 000939 1744020147.051707 read 0x022fc5 thru 0x02b4db (0x8517 bytes) 000940 1744020147.055671 MIRROR_RESYNC 7 mirrors 000941 1744020147.336966 mapread 0x01c76c thru 0x027c58 (0xb4ed bytes) 000942 1744020147.343709 MIRROR_EXTEND to 8 mirrors 000943 1744020147.459008 mapread 0x0080a5 thru 0x01476e (0xc6ca bytes) 000944 1744020147.465609 trunc from 0x03b933 to 0x003912 (0x38020 bytes) 000945 1744020147.481029 mapread 0x002ba7 thru 0x003912 (0xd6c bytes) 000948 1744020147.483340 punch from 0x0016a4 to 0x003912 (0x226f bytes) 000949 1744020147.484631 mapread 0x001ff4 thru 0x003912 (0x191f bytes) 000950 1744020147.487033 fallocate 0x02bfe0 thru 0x039964 (0xd985 bytes) 000951 1744020147.488940 read 0x00ff47 thru 0x01258e (0x2648 bytes) 000952 1744020147.490684 trunc from 0x039965 to 0x02a7b8 (0xf1ac bytes) 000953 1744020147.495781 read 0x01fc34 thru 0x0203e5 (0x7b2 bytes) 000956 1744020147.499424 trunc from 0x02a7b9 to 0x00e60e (0x1c1aa bytes) 000957 1744020147.504074 mapread 0x0004a9 thru 0x00d2d6 (0xce2e bytes) 000959 1744020147.504967 read 0x002381 thru 0x00df3d (0xbbbd bytes) 000960 1744020147.505317 punch from 0x00218d to 0x00ab3f (0x89b3 bytes) 000961 1744020147.506517 fallocate 0x037c36 thru 0x03ffff (0x83ca bytes) 000962 1744020147.507886 mapwrite 0x030606 thru 0x03e2a5 (0xdca0 bytes) 000964 1744020147.521246 write 0x001d78 thru 0x008f74 (0x71fd bytes) 000965 1744020147.522154 write 0x03231b thru 0x03416b (0x1e51 bytes) 000966 1744020147.522562 read 0x01a81b thru 0x01ab12 (0x2f8 bytes) 000967 1744020147.522868 fallocate 0x02d3b5 thru 0x02f65d (0x22a9 bytes) 000968 1744020147.524277 fallocate 0x00dd01 thru 0x01a7e1 (0xcae1 bytes) 000969 1744020147.525613 trunc from 0x03e2a6 to 0x00cee6 (0x313bf bytes) 000972 1744020147.531654 punch from 0x005ad2 to 0x00cee6 (0x7415 bytes) 000973 1744020147.535419 write 0x02d876 thru 0x032ade (0x5269 bytes) 000974 1744020147.536183 trunc from 0x032adf to 0x02ea48 (0x4096 bytes) 000975 1744020147.539743 mapwrite 0x023f26 thru 0x029757 (0x5832 bytes) 000976 1744020147.548784 fallocate 0x027f6c thru 0x02c24c (0x42e1 bytes) 000977 1744020147.550316 write 0x022b19 thru 0x029239 (0x6721 bytes) 000979 1744020147.551100 trunc from 0x02ea49 to 0x01c600 (0x12448 bytes) 000980 1744020147.554903 trunc from 0x01c601 to 0x00a870 (0x11d90 bytes) 000981 1744020147.558354 fallocate 0x01decc thru 0x02b7df (0xd914 bytes) 000982 1744020147.560951 MIRROR_EXTEND to 9 mirrors 000983 1744020147.655697 mapwrite 0x037e55 thru 0x03ffff (0x81ab bytes) 000984 1744020147.678655 write 0x037624 thru 0x03ffff (0x89dc bytes) 000985 1744020147.679335 mapwrite 0x02844b thru 0x035710 (0xd2c6 bytes) 000986 1744020147.689499 read 0x004bef thru 0x013ebb (0xf2cd bytes) 000987 1744020147.691927 trunc from 0x040000 to 0x03ea78 (0x1587 bytes) 000988 1744020147.696626 write 0x02597e thru 0x035677 (0xfcfa bytes) 000992 1744020147.698022 mapwrite 0x012fe8 thru 0x01352a (0x543 bytes) 000993 1744020147.703033 punch from 0x00fb03 to 0x01b250 (0xb74e bytes) 000994 1744020147.704343 mapwrite 0x038a2e thru 0x03ae4d (0x2420 bytes) 000996 1744020147.707961 punch from 0x030dd9 to 0x03c2f0 (0xb518 bytes) 000997 1744020147.709335 write 0x01c570 thru 0x028dfa (0xc88b bytes) 000998 1744020147.711937 read 0x03074d thru 0x0392ff (0x8bb3 bytes) 001000 1744020147.714556 write 0x0128b1 thru 0x01e276 (0xb9c6 bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 71: check mirror extend parent fid ==== 06:02:28 (1744020148) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0xaa1:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 06:02:31 (1744020151) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2382560 .. Extending file size to 4681568 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7539328 .. Extending file size to 5754368 .. Extending file size to 7724704 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6177440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8088224 .. Extending file size to 7816160 .. Extending file size to 8967712 .. Extending file size to 8387616 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8679712 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9257600 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9401824 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8772000 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8839840 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9287264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 4634 4635 4637 4639 4641 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) 192.168.202.112@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:03:43 (1744020223) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3810944 .. Extending file size to 4447616 .. failed Extending file size to 7006752 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 4418944 .. Extending file size to 6104768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8395328 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8574176 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9045088 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8673408 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8727488 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8853952 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9009024 .. Extending file size to 9067840 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 8900 8901 8903 8904 8907 failed 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) 192.168.202.112@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:04:55 (1744020295) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4128 1283560 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14676 3592344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22460 3584560 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37136 7176904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4128 1283560 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1285624 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14676 3592344 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22460 3584560 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37136 7176904 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0327083 s, 64.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0470559 s, 44.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:05:00 (1744020300) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0258342 s, 81.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 06:05:02 (1744020302) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xaad:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xffc:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:05:03 (1744020303) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202725 s, 103 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x43:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x43:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:05:05 (1744020305) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0217364 s, 96.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xffd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xffd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xffe:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:05:07 (1744020307) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0212605 s, 98.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x44:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x45:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x44:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x45:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:05:29 (1744020329) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116541 s, 90.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x45:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x48:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x45:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x48:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x47:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x46:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:06:03 (1744020363) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xfff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0212108 s, 98.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xfff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xffe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xfff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:06:38 (1744020398) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg212-server: Pool lustre.test_204f created Adding targets to pool oleg212-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg212-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0194782 s, 108 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1000:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1000:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1000:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1000:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg212-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg212-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg212-server: Pool lustre.test_204f destroyed PASS 204f (46s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:07:24 (1744020444) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:07:26 (1744020446) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:07:27 (1744020447) oleg212-server: Pool lustre.test_206 created Adding targets to pool oleg212-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg212-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg212-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg212-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg212-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:07:40 (1744020460) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178442 s, 58.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895527 s, 117 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf71cb5d8 Mirror 3: 0xf71cb5d8 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:07:42 (1744020462) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:07:43 (1744020463) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:07:44 (1744020464) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00867229 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x10a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe10000 Read this (4 bytes): BBBB 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:07:57 (1744020477) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00275298 s, 3.0 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:07:58 (1744020478) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843641 s, 124 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xacc lmm_fid: [0x200000405:0xacc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xacc lmm_fid: [0x200000405:0xacc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x100b:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613900 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-07 06:07:58.000000000 -0400 Modify: 2025-04-07 06:07:58.000000000 -0400 Change: 2025-04-07 06:07:59.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:08:00 (1744020480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00728574 s, 144 MB/s oleg212-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg212-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg212-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg212-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg212-server: O/280000401/d10/100a: File not found by ext2_lookup PASS 210b (9s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:08:09 (1744020489) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0203776 s, 2.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0039132 s, 1.0 MB/s size after second write -rw-r--r-- 1 root root 4096 Apr 7 06:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xad1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 7 06:08 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 705 sec ============ 06:08:11 (1744020491) === sanity-flr: start cleanup 06:08:11 (1744020491) === === sanity-flr: finish cleanup 06:08:13 (1744020493) ===