-----============= acceptance-small: sanity-flr ============----- Tue Dec 24 23:04:23 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg342-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:04:31 (1735099471) === oleg342-client.virtnet: executing check_config_client /mnt/lustre oleg342-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg342-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ab15b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ab15b000.idle_timeout=debug disable quota as required oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:04:42 (1735099482) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7601] == sanity-flr test 0a: lfs mirror create with -N option == 23:04:43 (1735099483) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:04:47 (1735099487) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:04:49 (1735099489) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:04:50 (1735099490) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:04:56 (1735099496) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:04:57 (1735099497) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 23:04:59 (1735099499) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800ab15b000.stats=clear osc.lustre-OST0001-osc-ffff8800ab15b000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:05:01 (1735099501) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:05:04 (1735099504) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:05:06 (1735099506) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:05:07 (1735099507) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 23:05:10 (1735099510) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:05:12 (1735099512) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:05:14 (1735099514) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:05:17 (1735099517) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:05:19 (1735099519) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:05:21 (1735099521) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.301829 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0334156 s, 31.4 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:05:24 (1735099524) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0641639 s, 131 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:05:26 (1735099526) Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:05:51 (1735099551) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0711096 s, 59.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:06:03 (1735099563) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:06:28 (1735099588) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.826072 s, 63.5 MB/s -rw-r--r-- 1 root root 52428800 Dec 24 23:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.219s user 0m0.000s sys 0m0.130s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... real 0m0.225s user 0m0.000s sys 0m0.143s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.251s user 0m0.000s sys 0m0.133s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ab15b000.ost_server_uuid in FULL state after 0 sec PASS 33b (26s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:06:54 (1735099614) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:06:54 (1735099614) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:06:55 (1735099615) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:06:56 (1735099616) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 23:06:58 (1735099618) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.137737 s, 91.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:07:01 (1735099621) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965544 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100125 s, 105 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:07:03 (1735099623) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:07:06 (1735099626) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000608963 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000601687 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000676608 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000854239 s, 1.2 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:07:08 (1735099628) create test files with size 15733229 .. 55af028e910d975a4e8ff952dd956651 - ff07f3bf7e35ba0bb637fcab059a164c - eb9b6d886ac15ef9b0a0f03f9116a343 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 23:07:14 (1735099634) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:07:18 (1735099638) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:07:19 (1735099639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309807 s, 33.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121506 s, 86.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x40:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 23:07:22 (1735099642) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.02585 s, 81.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.101564 s, 41.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0535797 s, 39.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:07:25 (1735099645) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:07:26 (1735099646) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:07:27 (1735099647) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122925 s, 85.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:07:29 (1735099649) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0834933 s, 37.7 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:07:31 (1735099651) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:07:33 (1735099653) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0808273 s, 130 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:07:55 (1735099675) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b5870000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0852433 s, 123 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aa9d4800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:07:58 (1735099678) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:08:06 (1735099686) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:08:12 (1735099692) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:08:13 (1735099693) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0691493 s, 45.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0506458 s, 62.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 5c08066e6ab2f79f09d37c3608b9012b - mirror 1 checksum: fae65a4336f1e8db68b905ba0fa2aabc - mirror 2 checksum: 5c08066e6ab2f79f09d37c3608b9012b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 23:08:14 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 23:08:14 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5c08066e6ab2f79f09d37c3608b9012b - mirror 1 checksum: fae65a4336f1e8db68b905ba0fa2aabc - mirror 2 checksum: fae65a4336f1e8db68b905ba0fa2aabc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 23:08:14 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 23:08:14 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:08:15 (1735099695) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898734 s, 117 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:08:17 (1735099697) Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 23:08:24 (1735099704) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 23:08:37 (1735099717) targets are mounted 23:08:37 (1735099717) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00990557 s, 106 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:08:45 (1735099725) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00860784 s, 2.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00444391 s, 4.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:08:47 (1735099727) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0116284 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00409118 s, 5.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:08:49 (1735099729) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00239551 s, 6.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:08:51 (1735099731) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00749481 s, 2.7 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:08:53 (1735099733) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00163521 s, 2.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00915586 s, 447 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:08:54 (1735099734) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:08:58 (1735099738) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:09:16 (1735099756) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:09:32 (1735099772) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:09:49 (1735099789) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 23:09:51 (1735099791) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26328 mirror split pid 26329 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26328 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26329 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 23:10:53 (1735099853) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9853 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1735099853.571790 MIRROR_EXTEND to 2 mirrors 000002 1735099853.651700 write 0x02fab0 thru 0x03528d (0x57de bytes) 000004 1735099853.665192 mapwrite 0x013512 thru 0x021570 (0xe05f bytes) 000005 1735099853.686321 write 0x03051b thru 0x031d46 (0x182c bytes) 000006 1735099853.686820 write 0x0181f5 thru 0x01f41a (0x7226 bytes) 000007 1735099853.687395 mapread 0x0197c0 thru 0x023e84 (0xa6c5 bytes) fallocating to largest ever: 0x40000 000008 1735099853.688042 fallocate 0x03c6d9 thru 0x03ffff (0x3927 bytes) 000009 1735099853.690233 write 0x010d8a thru 0x01e8d0 (0xdb47 bytes) 000010 1735099853.691037 mapwrite 0x016964 thru 0x021add (0xb17a bytes) 000011 1735099853.698050 read 0x01af72 thru 0x01cb6f (0x1bfe bytes) 000012 1735099853.698481 mapread 0x002082 thru 0x00d3e6 (0xb365 bytes) 000013 1735099853.703806 fallocate 0x01efd5 thru 0x0250b5 (0x60e1 bytes) 000014 1735099853.705547 write 0x033bb7 thru 0x039184 (0x55ce bytes) 000015 1735099853.708417 punch from 0x000406 to 0x009c87 (0x9882 bytes) 000016 1735099853.710859 mapread 0x02594e thru 0x02741d (0x1ad0 bytes) 000017 1735099853.714857 mapread 0x00a62e thru 0x00e937 (0x430a bytes) 000018 1735099853.717824 MIRROR_RESYNC 2 mirrors 000019 1735099853.841699 mapread 0x001a3d thru 0x011626 (0xfbea bytes) 000020 1735099853.849057 fallocate 0x03a9c1 thru 0x03dcfd (0x333d bytes) 000021 1735099853.859806 read 0x01af7c thru 0x02943a (0xe4bf bytes) 000022 1735099853.863581 fallocate 0x00db2a thru 0x01350b (0x59e2 bytes) 000024 1735099853.866240 fallocate 0x003b6d thru 0x0055f5 (0x1a89 bytes) 000025 1735099853.868798 fallocate 0x025ad6 thru 0x02ee5f (0x938a bytes) 000027 1735099853.870968 fallocate 0x02594a thru 0x035541 (0xfbf8 bytes) 000028 1735099853.873348 read 0x01967a thru 0x0276b3 (0xe03a bytes) 000030 1735099853.878287 trunc from 0x040000 to 0x020ca9 (0x1f356 bytes) 000031 1735099853.890661 mapwrite 0x0285c6 thru 0x02ed95 (0x67d0 bytes) 000032 1735099853.912762 fallocate 0x0134bf thru 0x015fa8 (0x2aea bytes) 000033 1735099853.915370 mapread 0x005e14 thru 0x011ed5 (0xc0c2 bytes) 000034 1735099853.916003 mapread 0x019b9e thru 0x0278ea (0xdd4d bytes) 000035 1735099853.928022 MIRROR_EXTEND to 3 mirrors 000037 1735099854.111160 mapread 0x020e1a thru 0x025d29 (0x4f10 bytes) 000039 1735099854.126095 read 0x02bb0a thru 0x02ed95 (0x328c bytes) 000041 1735099854.132374 mapwrite 0x01301f thru 0x01b39f (0x8381 bytes) 000043 1735099854.171873 read 0x00ca15 thru 0x00dad8 (0x10c4 bytes) 000044 1735099854.172320 fallocate 0x027ab4 thru 0x033bf9 (0xc146 bytes) 000046 1735099854.175205 fallocate 0x02a6ea thru 0x039839 (0xf150 bytes) 000047 1735099854.177821 fallocate 0x01b645 thru 0x01f4f8 (0x3eb4 bytes) 000048 1735099854.179982 read 0x02b031 thru 0x02ed95 (0x3d65 bytes) 000050 1735099854.184098 punch from 0x0270d1 to 0x02c8bb (0x57eb bytes) 000051 1735099854.186630 mapread 0x00f6c3 thru 0x00fa12 (0x350 bytes) 000052 1735099854.189531 MIRROR_RESYNC 3 mirrors 000053 1735099854.404523 read 0x0218b8 thru 0x02ed95 (0xd4de bytes) 000054 1735099854.408735 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000055 1735099854.483536 write 0x02020f thru 0x027f6a (0x7d5c bytes) 000056 1735099854.498548 write 0x03fb1f thru 0x03ffff (0x4e1 bytes) 000057 1735099854.499163 read 0x00e091 thru 0x01c030 (0xdfa0 bytes) 000058 1735099854.502209 fallocate 0x0302f9 thru 0x03640a (0x6112 bytes) 000059 1735099854.503822 read 0x03f5fb thru 0x03ffff (0xa05 bytes) 000060 1735099854.504216 punch from 0x00ce2b to 0x010ffa (0x41d0 bytes) 000061 1735099854.506125 trunc from 0x040000 to 0x02c828 (0x137d7 bytes) 000062 1735099854.518029 mapread 0x01e673 thru 0x0287e4 (0xa172 bytes) 000063 1735099854.522245 punch from 0x00fbdc to 0x01bbec (0xc011 bytes) 000064 1735099854.524803 trunc from 0x02c829 to 0x018d07 (0x13b21 bytes) 000065 1735099854.530742 mapread 0x011d53 thru 0x018d07 (0x6fb5 bytes) 000066 1735099854.539747 fallocate 0x03f5b8 thru 0x03ffff (0xa48 bytes) 000067 1735099854.541405 mapread 0x00e00b thru 0x012c4e (0x4c44 bytes) 000068 1735099854.544280 fallocate 0x0309d5 thru 0x03e55d (0xdb89 bytes) 000069 1735099854.545794 read 0x00f844 thru 0x015817 (0x5fd4 bytes) 000070 1735099854.546154 mapread 0x01fe17 thru 0x028af5 (0x8cdf bytes) 000071 1735099854.549072 mapread 0x00881c thru 0x010bcb (0x83b0 bytes) 000072 1735099854.549567 fallocate 0x017223 thru 0x01a59a (0x3378 bytes) 000074 1735099854.551644 punch from 0x00bb55 to 0x014bee (0x909a bytes) 000075 1735099854.553412 punch from 0x000d2a to 0x00ba26 (0xacfd bytes) 000076 1735099854.555214 write 0x020cf2 thru 0x02c532 (0xb841 bytes) 000079 1735099854.556699 mapwrite 0x02e9cf thru 0x0377aa (0x8ddc bytes) 000080 1735099854.565259 read 0x0245f2 thru 0x02c76c (0x817b bytes) 000082 1735099854.566039 write 0x021dcd thru 0x02c77f (0xa9b3 bytes) 000083 1735099854.566939 mapread 0x02675b thru 0x028458 (0x1cfe bytes) 000084 1735099854.570197 MIRROR_EXTEND to 3 mirrors 000085 1735099854.737105 MIRROR_EXTEND to 4 mirrors 000086 1735099854.912011 mapread 0x008671 thru 0x009508 (0xe98 bytes) 000087 1735099854.923800 MIRROR_RESYNC 4 mirrors 000088 1735099855.085237 write 0x03c3c3 thru 0x03ffff (0x3c3d bytes) 000089 1735099855.097155 mapread 0x03fc02 thru 0x03ffff (0x3fe bytes) 000090 1735099855.097695 mapwrite 0x01e1a0 thru 0x02c6f5 (0xe556 bytes) 000091 1735099855.114322 write 0x027b6f thru 0x034514 (0xc9a6 bytes) 000092 1735099855.117474 punch from 0x00bfcc to 0x011bf5 (0x5c2a bytes) 000093 1735099855.119544 trunc from 0x040000 to 0x0349b0 (0xb64f bytes) 000094 1735099855.132240 mapwrite 0x0114d2 thru 0x016da8 (0x58d7 bytes) 000095 1735099855.146415 punch from 0x02ad3c to 0x0349b0 (0x9c75 bytes) 000097 1735099855.150135 write 0x01f799 thru 0x0238ca (0x4132 bytes) 000099 1735099855.151367 write 0x02ca5e thru 0x039680 (0xcc23 bytes) 000100 1735099855.155679 punch from 0x0395c6 to 0x039680 (0xbb bytes) 000101 1735099855.163074 mapwrite 0x01d86d thru 0x02b315 (0xdaa9 bytes) 000102 1735099855.174654 fallocate 0x01075e thru 0x0133eb (0x2c8e bytes) 000103 1735099855.177807 read 0x010da5 thru 0x012ea3 (0x20ff bytes) 000104 1735099855.178924 trunc from 0x039681 to 0x019b5d (0x1fb23 bytes) 000105 1735099855.185298 fallocate 0x02cfa4 thru 0x03940a (0xc467 bytes) 000106 1735099855.187527 mapread 0x02915d thru 0x029265 (0x109 bytes) 000108 1735099855.192108 fallocate 0x018635 thru 0x01e87b (0x6247 bytes) 000109 1735099855.194447 read 0x0041fa thru 0x00df4e (0x9d55 bytes) 000110 1735099855.198461 trunc from 0x03940b to 0x03c0d4 (0x2cca bytes) 000111 1735099855.205625 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000112 1735099855.278347 read 0x01c485 thru 0x02a42b (0xdfa7 bytes) 000113 1735099855.282038 mapwrite 0x019683 thru 0x01f2a1 (0x5c1f bytes) 000115 1735099855.295874 read 0x007868 thru 0x01194d (0xa0e6 bytes) 000116 1735099855.300533 trunc from 0x03c0d5 to 0x03b68e (0xa46 bytes) 000119 1735099855.308052 read 0x039d62 thru 0x03b68e (0x192d bytes) 000120 1735099855.308468 mapwrite 0x01a600 thru 0x023e78 (0x9879 bytes) 000121 1735099855.316497 fallocate 0x025d2f thru 0x02dadf (0x7db1 bytes) 000122 1735099855.318426 punch from 0x00d48a to 0x01b3d4 (0xdf4b bytes) 000123 1735099855.321006 read 0x00d14c thru 0x013087 (0x5f3c bytes) 000124 1735099855.324399 read 0x00f1da thru 0x012b12 (0x3939 bytes) 000125 1735099855.324825 mapread 0x025054 thru 0x034310 (0xf2bd bytes) 000126 1735099855.328021 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000127 1735099855.384888 mapwrite 0x03957d thru 0x03ffff (0x6a83 bytes) 000129 1735099855.405132 fallocate 0x034a4e thru 0x03ffff (0xb5b2 bytes) 000130 1735099855.407029 fallocate 0x0072cb thru 0x00eea6 (0x7bdc bytes) 000131 1735099855.409245 read 0x031ae2 thru 0x03d1f3 (0xb712 bytes) 000132 1735099855.412759 punch from 0x03e66e to 0x03ffff (0x1992 bytes) 000133 1735099855.416685 MIRROR_RESYNC 2 mirrors 000134 1735099855.603658 MIRROR_RESYNC 2 mirrors 000136 1735099855.653937 mapwrite 0x01e317 thru 0x022674 (0x435e bytes) 000137 1735099855.700213 fallocate 0x00da6e thru 0x00f448 (0x19db bytes) 000138 1735099855.703965 punch from 0x0051c1 to 0x00d0cd (0x7f0d bytes) 000140 1735099855.708353 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000141 1735099855.752546 punch from 0x02a7c4 to 0x02eb45 (0x4382 bytes) 000142 1735099855.755164 trunc from 0x040000 to 0x008f55 (0x370aa bytes) 000143 1735099855.769526 punch from 0x0049dc to 0x005f7c (0x15a1 bytes) 000144 1735099855.771900 fallocate 0x03fda7 thru 0x03ffff (0x259 bytes) 000145 1735099855.774070 mapwrite 0x0396d4 thru 0x03ac47 (0x1574 bytes) 000147 1735099855.787068 trunc from 0x040000 to 0x009445 (0x36bba bytes) 000148 1735099855.795626 write 0x00f5dc thru 0x01682a (0x724f bytes) 000150 1735099855.797120 read 0x00bba5 thru 0x00f454 (0x38b0 bytes) 000151 1735099855.800065 read 0x010d23 thru 0x01682a (0x5b08 bytes) 000152 1735099855.800476 mapwrite 0x02d770 thru 0x03506f (0x7900 bytes) 000153 1735099855.824167 read 0x021591 thru 0x025861 (0x42d1 bytes) 000155 1735099855.825258 trunc from 0x035070 to 0x01db4f (0x17520 bytes) 000156 1735099855.831428 mapwrite 0x025b73 thru 0x0355c5 (0xfa53 bytes) 000158 1735099855.857646 mapread 0x01dff8 thru 0x02dbce (0xfbd7 bytes) 000159 1735099855.858385 fallocate 0x012b9a thru 0x017a29 (0x4e90 bytes) 000160 1735099855.860463 mapwrite 0x0189ec thru 0x020978 (0x7f8d bytes) 000161 1735099855.867391 read 0x02a17b thru 0x02afac (0xe32 bytes) 000162 1735099855.867638 trunc from 0x0355c6 to 0x037ce5 (0x2720 bytes) 000163 1735099855.871788 punch from 0x01261d to 0x01e30a (0xbcee bytes) 000164 1735099855.873864 read 0x02720f thru 0x02fe55 (0x8c47 bytes) 000166 1735099855.874516 trunc from 0x037ce6 to 0x000552 (0x37793 bytes) 000167 1735099855.881334 MIRROR_EXTEND to 2 mirrors 000168 1735099855.993471 punch from 0x0002ee to 0x000552 (0x265 bytes) 000169 1735099856.007627 fallocate 0x0177af thru 0x022015 (0xa867 bytes) 000170 1735099856.009706 read 0x00f34f thru 0x0151b7 (0x5e69 bytes) 000171 1735099856.012909 fallocate 0x033fa5 thru 0x03a9f0 (0x6a4c bytes) 000172 1735099856.015898 trunc from 0x022016 to 0x00d4e4 (0x14b31 bytes) 000173 1735099856.035408 MIRROR_RESYNC 2 mirrors 000174 1735099856.147320 read 0x003ace thru 0x00894f (0x4e82 bytes) 000175 1735099856.151157 MIRROR_EXTEND to 3 mirrors 000176 1735099856.302862 mapread 0x001712 thru 0x00a41f (0x8d0e bytes) 000178 1735099856.308598 punch from 0x00ac8f to 0x00d4e4 (0x2856 bytes) 000180 1735099856.325633 read 0x0042b0 thru 0x00877b (0x44cc bytes) 000181 1735099856.329922 read 0x003739 thru 0x00ce58 (0x9720 bytes) 000182 1735099856.335616 trunc from 0x00d4e5 to 0x0116cd (0x41e9 bytes) 000183 1735099856.355923 mapread 0x008dde thru 0x00bf48 (0x316b bytes) 000185 1735099856.356990 read 0x00201f thru 0x00e5b8 (0xc59a bytes) 000188 1735099856.360264 fallocate 0x00baef thru 0x01118b (0x569d bytes) 000189 1735099856.362455 fallocate 0x023cb7 thru 0x030900 (0xcc4a bytes) 000190 1735099856.364747 mapwrite 0x0331c5 thru 0x03b561 (0x839d bytes) 000191 1735099856.383891 mapread 0x01db6f thru 0x02307c (0x550e bytes) 000192 1735099856.384669 punch from 0x025a81 to 0x030a18 (0xaf98 bytes) 000193 1735099856.387034 mapwrite 0x020839 thru 0x02ae73 (0xa63b bytes) 000194 1735099856.408839 punch from 0x01fe68 to 0x02168f (0x1828 bytes) 000195 1735099856.411696 mapread 0x016043 thru 0x01f0f4 (0x90b2 bytes) 000196 1735099856.414949 fallocate 0x02a615 thru 0x02d784 (0x3170 bytes) 000197 1735099856.417363 mapwrite 0x01d8ac thru 0x024cf2 (0x7447 bytes) 000198 1735099856.428903 fallocate 0x034529 thru 0x03ffff (0xbad7 bytes) 000199 1735099856.431036 mapread 0x00a9f8 thru 0x00d228 (0x2831 bytes) 000200 1735099856.431560 mapread 0x006bcc thru 0x007b15 (0xf4a bytes) 000201 1735099856.432043 mapwrite 0x025703 thru 0x02d567 (0x7e65 bytes) 000202 1735099856.445808 mapread 0x00495f thru 0x00cd84 (0x8426 bytes) 000203 1735099856.446433 mapwrite 0x0340b9 thru 0x03ffff (0xbf47 bytes) 000206 1735099856.465072 fallocate 0x00d8df thru 0x018d8e (0xb4b0 bytes) 000208 1735099856.468009 trunc from 0x040000 to 0x0246a3 (0x1b95c bytes) 000209 1735099856.472963 mapwrite 0x00b888 thru 0x012890 (0x7009 bytes) 000210 1735099856.481525 punch from 0x00ca10 to 0x0107bd (0x3dae bytes) 000211 1735099856.483713 punch from 0x0219e2 to 0x0246a3 (0x2cc2 bytes) 000212 1735099856.485679 mapwrite 0x03e976 thru 0x03ffff (0x168a bytes) 000213 1735099856.506288 read 0x03dd71 thru 0x03ffff (0x228f bytes) 000214 1735099856.512574 MIRROR_EXTEND to 4 mirrors 000215 1735099856.726105 write 0x03d1d2 thru 0x03df56 (0xd85 bytes) 000216 1735099856.738304 fallocate 0x012e98 thru 0x01b1c9 (0x8332 bytes) 000217 1735099856.740908 fallocate 0x008b52 thru 0x015255 (0xc704 bytes) 000218 1735099856.743039 write 0x0097df thru 0x0151ef (0xba11 bytes) 000219 1735099856.746462 mapread 0x00cde6 thru 0x017a9d (0xacb8 bytes) 000220 1735099856.750068 mapwrite 0x020076 thru 0x0256d2 (0x565d bytes) 000221 1735099856.765354 read 0x03c03d thru 0x03ffff (0x3fc3 bytes) 000222 1735099856.769571 MIRROR_RESYNC 4 mirrors 000223 1735099857.067886 MIRROR_RESYNC 4 mirrors 000224 1735099857.121415 mapwrite 0x002d01 thru 0x011dc8 (0xf0c8 bytes) 000225 1735099857.174826 mapwrite 0x00e89a thru 0x0105b8 (0x1d1f bytes) 000226 1735099857.188657 read 0x019ae8 thru 0x01b886 (0x1d9f bytes) 000228 1735099857.194363 read 0x00aac3 thru 0x00db1c (0x305a bytes) 000229 1735099857.197808 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000231 1735099857.260222 MIRROR_RESYNC 3 mirrors 000232 1735099857.472228 punch from 0x02bf6f to 0x036e38 (0xaeca bytes) 000234 1735099857.487412 mapread 0x0060d3 thru 0x00782e (0x175c bytes) 000235 1735099857.492149 mapwrite 0x029fc6 thru 0x02d735 (0x3770 bytes) 000236 1735099857.505378 MIRROR_EXTEND to 4 mirrors 000237 1735099857.697849 fallocate 0x01fe06 thru 0x02ee87 (0xf082 bytes) 000238 1735099857.712524 write 0x03f9ef thru 0x03ffff (0x611 bytes) 000239 1735099857.718629 trunc from 0x040000 to 0x005148 (0x3aeb7 bytes) 000244 1735099857.737253 fallocate 0x0392ff thru 0x03f9ad (0x66af bytes) 000245 1735099857.740270 trunc from 0x03f9ae to 0x03b1a1 (0x480c bytes) 000246 1735099857.749045 mapread 0x001e98 thru 0x011123 (0xf28c bytes) 000248 1735099857.764621 punch from 0x030e20 to 0x03b1a1 (0xa382 bytes) 000249 1735099857.766966 mapread 0x02ebd9 thru 0x03b1a1 (0xc5c9 bytes) 000250 1735099857.778874 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000251 1735099857.847433 MIRROR_EXTEND to 4 mirrors 000252 1735099857.993629 read 0x013b7e thru 0x01e9b6 (0xae39 bytes) 000254 1735099858.003640 fallocate 0x03cee8 thru 0x03ffff (0x3118 bytes) 000255 1735099858.020695 write 0x032fc0 thru 0x034335 (0x1376 bytes) 000256 1735099858.028967 mapwrite 0x026e0f thru 0x02e6c7 (0x78b9 bytes) 000257 1735099858.046679 trunc from 0x040000 to 0x03014b (0xfeb4 bytes) 000258 1735099858.062521 read 0x014761 thru 0x01b4d5 (0x6d75 bytes) 000259 1735099858.066467 punch from 0x015107 to 0x0202a2 (0xb19c bytes) 000260 1735099858.068505 write 0x0292d1 thru 0x02fd89 (0x6ab9 bytes) 000261 1735099858.069325 read 0x0057d4 thru 0x00d8ef (0x811c bytes) 000262 1735099858.071629 MIRROR_RESYNC 4 mirrors 000263 1735099858.323845 mapwrite 0x03ff3b thru 0x03ffff (0xc5 bytes) 000264 1735099858.344668 trunc from 0x040000 to 0x01b4d9 (0x24b26 bytes) 000266 1735099858.350137 fallocate 0x014c79 thru 0x022ea4 (0xe22c bytes) 000269 1735099858.354521 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000271 1735099858.416172 mapwrite 0x03986b thru 0x03f822 (0x5fb8 bytes) 000272 1735099858.434852 mapread 0x018582 thru 0x025a18 (0xd497 bytes) 000274 1735099858.442135 punch from 0x00400e to 0x0081a3 (0x4196 bytes) 000275 1735099858.444254 read 0x025920 thru 0x02ee1e (0x94ff bytes) 000276 1735099858.446723 punch from 0x02ccca to 0x03bab3 (0xedea bytes) 000277 1735099858.449191 mapread 0x010a1b thru 0x01a957 (0x9f3d bytes) 000278 1735099858.453775 fallocate 0x03f7b5 thru 0x03ffff (0x84b bytes) 000279 1735099858.455831 fallocate 0x032af4 thru 0x03d411 (0xa91e bytes) 000280 1735099858.458015 punch from 0x02317d to 0x0294bd (0x6341 bytes) 000281 1735099858.460529 mapwrite 0x02f99f thru 0x032160 (0x27c2 bytes) 000283 1735099858.469516 read 0x03ddc0 thru 0x03ffff (0x2240 bytes) 000284 1735099858.469959 mapread 0x016e0b thru 0x026bd5 (0xfdcb bytes) 000286 1735099858.474102 mapread 0x03a1bb thru 0x03fb3d (0x5983 bytes) 000288 1735099858.476789 MIRROR_EXTEND to 4 mirrors 000289 1735099858.616384 trunc from 0x040000 to 0x01295e (0x2d6a1 bytes) 000290 1735099858.633582 trunc from 0x01295f to 0x03f0b7 (0x2c759 bytes) 000291 1735099858.638528 punch from 0x036fd9 to 0x0382aa (0x12d2 bytes) 000292 1735099858.639790 read 0x0011d8 thru 0x010396 (0xf1bf bytes) 000294 1735099858.642370 punch from 0x001514 to 0x00acc7 (0x97b4 bytes) 000295 1735099858.643886 write 0x01a997 thru 0x01e151 (0x37bb bytes) 000296 1735099858.647014 trunc from 0x03f0b8 to 0x02ecd3 (0x103e4 bytes) 000297 1735099858.652214 MIRROR_RESYNC 4 mirrors 000298 1735099858.807119 MIRROR_EXTEND to 5 mirrors 000299 1735099858.935593 write 0x0346ea thru 0x03c67d (0x7f94 bytes) 000301 1735099858.946203 mapwrite 0x014601 thru 0x01a10c (0x5b0c bytes) 000302 1735099858.958604 trunc from 0x03c67e to 0x031cda (0xa9a3 bytes) 000303 1735099858.969698 trunc from 0x031cdb to 0x020d59 (0x10f81 bytes) 000304 1735099858.973785 write 0x0379bf thru 0x03944d (0x1a8f bytes) 000305 1735099858.974468 read 0x027795 thru 0x032e54 (0xb6c0 bytes) 000307 1735099858.977447 trunc from 0x03944e to 0x03bbe6 (0x2799 bytes) 000308 1735099858.981147 write 0x0283f6 thru 0x02e2d3 (0x5ede bytes) 000309 1735099858.983204 MIRROR_RESYNC 5 mirrors 000310 1735099859.325327 mapwrite 0x020e77 thru 0x0245ff (0x3789 bytes) 000311 1735099859.370388 mapread 0x007277 thru 0x00c16c (0x4ef6 bytes) 000312 1735099859.376220 MIRROR_EXTEND to 6 mirrors 000313 1735099859.537880 mapread 0x037e7b thru 0x03bbe6 (0x3d6c bytes) 000314 1735099859.553775 mapwrite 0x003458 thru 0x01305a (0xfc03 bytes) 000315 1735099859.609530 mapwrite 0x03629f thru 0x03ffff (0x9d61 bytes) 000316 1735099859.644793 read 0x009d25 thru 0x01350f (0x97eb bytes) 000317 1735099859.645482 mapread 0x023509 thru 0x02ea97 (0xb58f bytes) 000318 1735099859.655683 mapwrite 0x01a880 thru 0x02521d (0xa99e bytes) 000319 1735099859.668454 mapread 0x00d2db thru 0x017128 (0x9e4e bytes) 000320 1735099859.669169 mapread 0x008c91 thru 0x0109e5 (0x7d55 bytes) 000321 1735099859.669710 trunc from 0x040000 to 0x0206b1 (0x1f94e bytes) 000322 1735099859.675489 mapwrite 0x01ae73 thru 0x02115c (0x62ea bytes) 000324 1735099859.691364 trunc from 0x02115d to 0x02ee53 (0xdcf7 bytes) 000325 1735099859.696784 read 0x01ac14 thru 0x026213 (0xb600 bytes) 000327 1735099859.702518 MIRROR_RESYNC 6 mirrors 000328 1735099859.989186 read 0x000dec thru 0x00c644 (0xb859 bytes) 000329 1735099859.992531 read 0x012b42 thru 0x021f33 (0xf3f2 bytes) 000330 1735099859.997360 write 0x01b119 thru 0x0214db (0x63c3 bytes) 000331 1735099860.012608 trunc from 0x02ee54 to 0x026db8 (0x809b bytes) 000332 1735099860.027203 mapwrite 0x029855 thru 0x030ffb (0x77a7 bytes) 000333 1735099860.057401 fallocate 0x02cc82 thru 0x038b91 (0xbf10 bytes) 000334 1735099860.059603 mapread 0x017269 thru 0x0210e4 (0x9e7c bytes) 000335 1735099860.065020 mapread 0x01b804 thru 0x02ad5b (0xf558 bytes) 000336 1735099860.075962 punch from 0x030b68 to 0x030ffb (0x494 bytes) 000337 1735099860.078211 trunc from 0x030ffc to 0x02a91e (0x66dd bytes) 000338 1735099860.086505 MIRROR_RESYNC 6 mirrors 000339 1735099860.420175 read 0x017269 thru 0x022915 (0xb6ad bytes) 000340 1735099860.422850 mapread 0x01f8db thru 0x026a1a (0x7140 bytes) 000341 1735099860.428655 punch from 0x003ecb to 0x00d366 (0x949c bytes) 000342 1735099860.440117 write 0x01d517 thru 0x0242ee (0x6dd8 bytes) 000343 1735099860.443433 write 0x022d76 thru 0x02c434 (0x96bf bytes) 000344 1735099860.444388 punch from 0x02a00a to 0x02c434 (0x242b bytes) 000346 1735099860.448597 read 0x009b1d thru 0x00a690 (0xb74 bytes) 000348 1735099860.451799 punch from 0x008b88 to 0x0148a2 (0xbd1b bytes) 000349 1735099860.455497 MIRROR_RESYNC 6 mirrors 000350 1735099860.834597 punch from 0x01eb1b to 0x02407a (0x5560 bytes) 000351 1735099860.846910 write 0x03e8ad thru 0x03f56f (0xcc3 bytes) 000352 1735099860.847660 mapwrite 0x025622 thru 0x02768b (0x206a bytes) 000353 1735099860.859396 write 0x002995 thru 0x00acd0 (0x833c bytes) 000354 1735099860.863172 fallocate 0x0298fe thru 0x029a4c (0x14f bytes) 000355 1735099860.865481 mapwrite 0x015499 thru 0x01a6ad (0x5215 bytes) 000356 1735099860.874324 trunc from 0x03f570 to 0x00b7eb (0x33d84 bytes) 000357 1735099860.889377 fallocate 0x011ac0 thru 0x01b92b (0x9e6c bytes) 000358 1735099860.891878 fallocate 0x036aee thru 0x03ffff (0x9512 bytes) 000359 1735099860.894188 punch from 0x008a98 to 0x010a84 (0x7fed bytes) 000360 1735099860.896474 mapwrite 0x00f10d thru 0x013916 (0x480a bytes) 000361 1735099860.909318 read 0x005bff thru 0x00d02a (0x742c bytes) 000362 1735099860.909683 fallocate 0x00155b thru 0x00ed52 (0xd7f8 bytes) 000363 1735099860.911892 mapread 0x0058dc thru 0x0136e3 (0xde08 bytes) 000364 1735099860.912538 trunc from 0x01b92c to 0x02c632 (0x10d07 bytes) 000365 1735099860.919009 fallocate 0x00dd51 thru 0x0123e2 (0x4692 bytes) 000366 1735099860.921628 read 0x01d6ec thru 0x027326 (0x9c3b bytes) 000367 1735099860.925552 mapwrite 0x029786 thru 0x02cc41 (0x34bc bytes) 000368 1735099860.937277 punch from 0x01c0a7 to 0x025398 (0x92f2 bytes) 000369 1735099860.939097 mapread 0x00400c thru 0x00e83e (0xa833 bytes) 000370 1735099860.939703 punch from 0x01de4e to 0x026785 (0x8938 bytes) 000371 1735099860.941757 read 0x01f7d7 thru 0x02cc41 (0xd46b bytes) 000372 1735099860.944605 mapread 0x0253ba thru 0x02cc41 (0x7888 bytes) 000373 1735099860.945043 trunc from 0x02cc42 to 0x01d0e3 (0xfb5e bytes) 000375 1735099860.950548 write 0x033491 thru 0x03fc9b (0xc80b bytes) 000376 1735099860.951698 mapwrite 0x0378df thru 0x03ffff (0x8721 bytes) 000378 1735099860.964132 write 0x03ddca thru 0x03ffff (0x2236 bytes) 000379 1735099860.964810 trunc from 0x040000 to 0x02811d (0x17ee2 bytes) 000380 1735099860.970375 read 0x002e58 thru 0x00e592 (0xb73b bytes) 000381 1735099860.970830 write 0x0054a1 thru 0x008720 (0x3280 bytes) 000383 1735099860.971800 fallocate 0x018c85 thru 0x024f4b (0xc2c7 bytes) 000385 1735099860.974339 write 0x007e93 thru 0x015e92 (0xe000 bytes) 000386 1735099860.975169 punch from 0x002419 to 0x00fbdc (0xd7c4 bytes) 000387 1735099860.985388 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000388 1735099861.051227 write 0x01e55f thru 0x02a04d (0xbaef bytes) 000390 1735099861.055754 mapwrite 0x03c84f thru 0x03ffff (0x37b1 bytes) 000391 1735099861.071963 trunc from 0x040000 to 0x035417 (0xabe8 bytes) 000392 1735099861.076580 mapread 0x031547 thru 0x035417 (0x3ed1 bytes) 000393 1735099861.079631 write 0x00bfce thru 0x010f42 (0x4f75 bytes) 000394 1735099861.083230 punch from 0x01db9b to 0x0219d8 (0x3e3e bytes) 000396 1735099861.085500 write 0x029718 thru 0x03346d (0x9d56 bytes) 000397 1735099861.086248 mapread 0x01d7b1 thru 0x01d853 (0xa3 bytes) 000399 1735099861.089952 trunc from 0x035418 to 0x014d89 (0x2068e bytes) 000400 1735099861.098267 write 0x02239b thru 0x027d20 (0x5986 bytes) 000402 1735099861.099512 fallocate 0x0076ab thru 0x011abd (0xa413 bytes) 000403 1735099861.102868 mapwrite 0x009c3f thru 0x00bce8 (0x20aa bytes) 000404 1735099861.113445 read 0x01e5c1 thru 0x027d20 (0x9760 bytes) 000405 1735099861.116744 mapwrite 0x039b65 thru 0x03ffff (0x649b bytes) 000406 1735099861.142102 mapread 0x038e40 thru 0x03cf73 (0x4134 bytes) 000407 1735099861.142714 mapwrite 0x03a681 thru 0x03c929 (0x22a9 bytes) 000409 1735099861.148167 write 0x027aa2 thru 0x02a5d1 (0x2b30 bytes) 000410 1735099861.150628 read 0x022c59 thru 0x023ab5 (0xe5d bytes) 000414 1735099861.152211 punch from 0x00818a to 0x011f23 (0x9d9a bytes) 000415 1735099861.155728 mapwrite 0x03ec56 thru 0x03ffff (0x13aa bytes) 000416 1735099861.166721 fallocate 0x038fdc thru 0x03ffff (0x7024 bytes) 000417 1735099861.170883 mapread 0x000417 thru 0x000600 (0x1ea bytes) 000418 1735099861.177214 read 0x002433 thru 0x003e7d (0x1a4b bytes) 000419 1735099861.177657 mapread 0x011ad2 thru 0x012a10 (0xf3f bytes) 000420 1735099861.180240 MIRROR_EXTEND to 6 mirrors 000421 1735099861.367108 read 0x02e375 thru 0x03d4b4 (0xf140 bytes) 000422 1735099861.371514 mapwrite 0x02b516 thru 0x02f0f3 (0x3bde bytes) 000423 1735099861.411232 mapread 0x0325a5 thru 0x0398dd (0x7339 bytes) 000424 1735099861.415049 read 0x00b168 thru 0x00b264 (0xfd bytes) 000425 1735099861.419793 read 0x01b25b thru 0x01d7e1 (0x2587 bytes) 000426 1735099861.420222 mapwrite 0x009b5f thru 0x01501a (0xb4bc bytes) 000428 1735099861.431286 trunc from 0x040000 to 0x014212 (0x2bded bytes) 000429 1735099861.446286 read 0x00d175 thru 0x00f0dd (0x1f69 bytes) 000430 1735099861.446720 fallocate 0x0313ab thru 0x03f4e4 (0xe13a bytes) 000431 1735099861.449460 read 0x021cef thru 0x02da51 (0xbd63 bytes) 000432 1735099861.456070 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000433 1735099861.528263 MIRROR_EXTEND to 6 mirrors 000434 1735099861.715872 write 0x034d49 thru 0x03ffff (0xb2b7 bytes) 000435 1735099861.736062 read 0x002073 thru 0x00d62e (0xb5bc bytes) 000437 1735099861.739977 punch from 0x02921f to 0x034a3a (0xb81c bytes) 000438 1735099861.746511 fallocate 0x00dac8 thru 0x01c082 (0xe5bb bytes) 000439 1735099861.748422 mapwrite 0x00413d thru 0x00ddcb (0x9c8f bytes) 000440 1735099861.761551 mapwrite 0x02eb5b thru 0x03a5da (0xba80 bytes) 000441 1735099861.775257 trunc from 0x040000 to 0x012570 (0x2da8f bytes) 000442 1735099861.792131 punch from 0x00949f to 0x00c23a (0x2d9c bytes) 000444 1735099861.796692 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000445 1735099861.866232 MIRROR_RESYNC 5 mirrors 000446 1735099862.192679 MIRROR_EXTEND to 6 mirrors 000447 1735099862.364608 mapwrite 0x0285b9 thru 0x02acc7 (0x270f bytes) 000449 1735099862.398289 write 0x0106d1 thru 0x01f7eb (0xf11b bytes) 000450 1735099862.404671 mapwrite 0x01b62f thru 0x027b74 (0xc546 bytes) 000451 1735099862.414694 write 0x02dfe8 thru 0x0343a9 (0x63c2 bytes) 000452 1735099862.415612 read 0x01f9f6 thru 0x027191 (0x779c bytes) 000453 1735099862.416022 read 0x00ca55 thru 0x0184a1 (0xba4d bytes) 000454 1735099862.416407 mapwrite 0x031043 thru 0x031add (0xa9b bytes) 000456 1735099862.423427 read 0x0057a1 thru 0x0064bc (0xd1c bytes) 000457 1735099862.426972 read 0x02a6dc thru 0x0343a9 (0x9cce bytes) 000458 1735099862.427428 read 0x0320b7 thru 0x0343a9 (0x22f3 bytes) 000459 1735099862.427797 read 0x019600 thru 0x020b1b (0x751c bytes) 000460 1735099862.428188 punch from 0x01ad88 to 0x02511e (0xa397 bytes) 000461 1735099862.432096 MIRROR_RESYNC 6 mirrors 000462 1735099862.960904 mapread 0x020f8c thru 0x02f46a (0xe4df bytes) 000463 1735099862.971601 mapread 0x02a08b thru 0x0329be (0x8934 bytes) 000464 1735099862.972025 mapwrite 0x020391 thru 0x025a4b (0x56bb bytes) 000465 1735099863.009751 punch from 0x020164 to 0x02281e (0x26bb bytes) 000466 1735099863.011631 trunc from 0x0343aa to 0x033d90 (0x619 bytes) 000467 1735099863.022668 mapread 0x02147e thru 0x02c314 (0xae97 bytes) 000468 1735099863.028425 write 0x01cd7e thru 0x01f832 (0x2ab5 bytes) 000469 1735099863.028930 mapread 0x02397e thru 0x031f2c (0xe5af bytes) 000470 1735099863.029488 trunc from 0x033d91 to 0x038451 (0x46c1 bytes) 000471 1735099863.035525 MIRROR_RESYNC 6 mirrors 000472 1735099863.639807 mapread 0x000461 thru 0x008e25 (0x89c5 bytes) 000474 1735099863.649064 fallocate 0x019d82 thru 0x01b1bc (0x143b bytes) 000475 1735099863.663871 read 0x0201f0 thru 0x023a2e (0x383f bytes) 000476 1735099863.666202 mapwrite 0x01e391 thru 0x026e91 (0x8b01 bytes) 000477 1735099863.677915 fallocate 0x006b4b thru 0x00867f (0x1b35 bytes) 000478 1735099863.679681 mapread 0x02871c thru 0x02af1b (0x2800 bytes) 000479 1735099863.682922 fallocate 0x00a9de thru 0x0102c1 (0x58e4 bytes) 000480 1735099863.686164 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000481 1735099863.744415 mapread 0x030f57 thru 0x038451 (0x74fb bytes) 000482 1735099863.751615 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000483 1735099863.807244 MIRROR_RESYNC 4 mirrors 000484 1735099864.071881 mapread 0x022cdf thru 0x029a8b (0x6dad bytes) 000485 1735099864.078648 MIRROR_RESYNC 4 mirrors 000486 1735099864.137768 trunc from 0x038452 to 0x0363cf (0x2082 bytes) 000487 1735099864.152335 read 0x01ce12 thru 0x0296f9 (0xc8e8 bytes) 000488 1735099864.157298 MIRROR_EXTEND to 5 mirrors 000489 1735099864.317318 mapread 0x00a158 thru 0x00f11c (0x4fc5 bytes) 000490 1735099864.326684 trunc from 0x0363d0 to 0x00916f (0x2d260 bytes) 000491 1735099864.344146 fallocate 0x029a47 thru 0x037ad6 (0xe090 bytes) 000492 1735099864.346235 fallocate 0x0073ed thru 0x01214c (0xad60 bytes) 000494 1735099864.348275 write 0x03cb28 thru 0x03ffff (0x34d8 bytes) 000495 1735099864.348821 read 0x0271ad thru 0x02fc27 (0x8a7b bytes) 000496 1735099864.351699 trunc from 0x040000 to 0x0247ed (0x1b812 bytes) 000497 1735099864.357609 fallocate 0x025765 thru 0x0308c6 (0xb162 bytes) 000498 1735099864.359793 trunc from 0x0308c7 to 0x028193 (0x8733 bytes) 000501 1735099864.365418 write 0x0349ac thru 0x03a08a (0x56df bytes) 000502 1735099864.366288 write 0x0218aa thru 0x025f9f (0x46f6 bytes) 000503 1735099864.370023 mapwrite 0x028eac thru 0x02f217 (0x636c bytes) 000504 1735099864.381605 read 0x011487 thru 0x01aec0 (0x9a3a bytes) 000505 1735099864.385367 read 0x020a30 thru 0x0236a5 (0x2c76 bytes) 000507 1735099864.386110 write 0x02341b thru 0x030f8a (0xdb70 bytes) 000508 1735099864.387108 read 0x00890b thru 0x00a463 (0x1b59 bytes) 000509 1735099864.390427 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000510 1735099864.458275 punch from 0x008af2 to 0x010a33 (0x7f42 bytes) 000511 1735099864.461553 MIRROR_RESYNC 4 mirrors 000513 1735099864.751859 write 0x039eb1 thru 0x03ffff (0x614f bytes) 000514 1735099864.763668 fallocate 0x02dd08 thru 0x035c78 (0x7f71 bytes) 000515 1735099864.765837 mapwrite 0x00f8f4 thru 0x01d0a7 (0xd7b4 bytes) 000516 1735099864.785598 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000517 1735099864.836254 MIRROR_RESYNC 3 mirrors 000518 1735099865.037799 fallocate 0x01c75a thru 0x0209af (0x4256 bytes) 000519 1735099865.050440 read 0x03bdb9 thru 0x03ffff (0x4247 bytes) 000521 1735099865.054177 trunc from 0x040000 to 0x0065af (0x39a50 bytes) 000522 1735099865.066336 read 0x004861 thru 0x0065af (0x1d4f bytes) 000523 1735099865.068928 write 0x00f57d thru 0x011cce (0x2752 bytes) 000524 1735099865.069611 fallocate 0x01b4e5 thru 0x0242ac (0x8dc8 bytes) 000525 1735099865.072556 punch from 0x020e6b to 0x0242ac (0x3442 bytes) 000526 1735099865.074490 mapwrite 0x015618 thru 0x01f349 (0x9d32 bytes) 000527 1735099865.087678 read 0x01e53d thru 0x01e53d (0x1 bytes) 000528 1735099865.088114 fallocate 0x001573 thru 0x00d44c (0xbeda bytes) 000529 1735099865.090284 fallocate 0x01e72c thru 0x027052 (0x8927 bytes) 000530 1735099865.092451 write 0x01f9ff thru 0x02da62 (0xe064 bytes) 000531 1735099865.093563 read 0x01b3ab thru 0x0285f2 (0xd248 bytes) 000532 1735099865.093976 mapread 0x0237c9 thru 0x02da62 (0xa29a bytes) 000534 1735099865.094653 punch from 0x00c206 to 0x0141fb (0x7ff6 bytes) 000535 1735099865.096807 fallocate 0x0275f8 thru 0x02cc05 (0x560e bytes) 000536 1735099865.098734 read 0x01ca1f thru 0x022559 (0x5b3b bytes) 000537 1735099865.099137 trunc from 0x02da63 to 0x018396 (0x156cc bytes) 000538 1735099865.104360 write 0x020860 thru 0x02114c (0x8ed bytes) 000539 1735099865.104931 trunc from 0x02114d to 0x03963f (0x184f3 bytes) 000540 1735099865.111198 write 0x026706 thru 0x02cc16 (0x6511 bytes) 000541 1735099865.115226 write 0x010998 thru 0x013aa2 (0x310b bytes) 000542 1735099865.115864 mapread 0x0283ad thru 0x035c76 (0xd8ca bytes) 000543 1735099865.116565 fallocate 0x01199e thru 0x019167 (0x77ca bytes) 000544 1735099865.119421 punch from 0x008d60 to 0x011fd6 (0x9277 bytes) 000545 1735099865.124386 mapread 0x008f68 thru 0x017453 (0xe4ec bytes) 000546 1735099865.136620 punch from 0x0041db to 0x0069dc (0x2802 bytes) 000547 1735099865.138239 fallocate 0x01fbbe thru 0x02748f (0x78d2 bytes) 000548 1735099865.139859 punch from 0x02bc30 to 0x03963f (0xda10 bytes) 000551 1735099865.142252 mapwrite 0x0224ad thru 0x026a5c (0x45b0 bytes) 000553 1735099865.148864 fallocate 0x00cd26 thru 0x00ee41 (0x211c bytes) 000554 1735099865.150389 read 0x02879a thru 0x02b5ea (0x2e51 bytes) 000556 1735099865.153072 mapwrite 0x0322d4 thru 0x036b8d (0x48ba bytes) 000557 1735099865.158335 mapwrite 0x01d7dd thru 0x01f584 (0x1da8 bytes) 000558 1735099865.162248 fallocate 0x033aab thru 0x0349f7 (0xf4d bytes) 000559 1735099865.164853 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000560 1735099865.213722 mapwrite 0x011d0c thru 0x013839 (0x1b2e bytes) 000561 1735099865.225475 MIRROR_RESYNC 2 mirrors 000562 1735099865.393295 mapread 0x01ba30 thru 0x0206b6 (0x4c87 bytes) 000563 1735099865.399728 mapwrite 0x02582e thru 0x027617 (0x1dea bytes) 000564 1735099865.429092 MIRROR_RESYNC 2 mirrors 000565 1735099865.607109 mapwrite 0x03d9d1 thru 0x03ffff (0x262f bytes) 000567 1735099865.645123 punch from 0x028208 to 0x032030 (0x9e29 bytes) 000568 1735099865.647673 trunc from 0x040000 to 0x021859 (0x1e7a6 bytes) 000569 1735099865.655225 read 0x0026bd thru 0x00756f (0x4eb3 bytes) 000570 1735099865.658985 trunc from 0x02185a to 0x03f782 (0x1df29 bytes) 000571 1735099865.665488 write 0x006da8 thru 0x013bbd (0xce16 bytes) 000572 1735099865.668460 punch from 0x019973 to 0x01f956 (0x5fe4 bytes) 000573 1735099865.670408 write 0x0113cb thru 0x016c20 (0x5856 bytes) 000574 1735099865.673894 fallocate 0x034d01 thru 0x03ffff (0xb2ff bytes) 000575 1735099865.676227 fallocate 0x02d11d thru 0x037546 (0xa42a bytes) 000576 1735099865.678562 fallocate 0x02310e thru 0x0313a0 (0xe293 bytes) 000577 1735099865.680447 write 0x00d2b4 thru 0x016dad (0x9afa bytes) 000578 1735099865.681122 read 0x03d571 thru 0x03f782 (0x2212 bytes) 000579 1735099865.685481 write 0x0001c8 thru 0x00b800 (0xb639 bytes) 000580 1735099865.686449 read 0x029316 thru 0x032d47 (0x9a32 bytes) 000582 1735099865.687253 read 0x00637e thru 0x008322 (0x1fa5 bytes) 000583 1735099865.687632 trunc from 0x03f783 to 0x02e97e (0x10e04 bytes) 000584 1735099865.693212 read 0x01937a thru 0x01e0fd (0x4d84 bytes) 000585 1735099865.693648 write 0x022b78 thru 0x0298d1 (0x6d5a bytes) 000587 1735099865.695732 MIRROR_EXTEND to 3 mirrors 000588 1735099865.813989 write 0x02b3e4 thru 0x02c78b (0x13a8 bytes) 000589 1735099865.826396 read 0x00bf11 thru 0x012cee (0x6dde bytes) 000590 1735099865.828722 fallocate 0x00f212 thru 0x010e33 (0x1c22 bytes) 000592 1735099865.830585 write 0x00bf2f thru 0x017faf (0xc081 bytes) 000593 1735099865.832723 mapwrite 0x00c47c thru 0x016089 (0x9c0e bytes) 000594 1735099865.841885 trunc from 0x02e97f to 0x03cae4 (0xe166 bytes) 000595 1735099865.852214 fallocate 0x038cc7 thru 0x03ffff (0x7339 bytes) 000596 1735099865.854569 write 0x015e88 thru 0x017ca5 (0x1e1e bytes) 000597 1735099865.857130 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000598 1735099865.920737 mapwrite 0x01eb1c thru 0x024bc1 (0x60a6 bytes) 000599 1735099865.942469 MIRROR_RESYNC 2 mirrors 000601 1735099866.056543 write 0x02d258 thru 0x02ed86 (0x1b2f bytes) 000602 1735099866.068786 punch from 0x012915 to 0x01ec05 (0xc2f1 bytes) 000603 1735099866.070829 read 0x030364 thru 0x03cae4 (0xc781 bytes) 000609 1735099866.074532 punch from 0x031395 to 0x037d59 (0x69c5 bytes) 000610 1735099866.076574 fallocate 0x02909d thru 0x0327aa (0x970e bytes) 000611 1735099866.078257 trunc from 0x03cae5 to 0x0239a8 (0x1913c bytes) 000612 1735099866.084185 mapread 0x0025e6 thru 0x00a993 (0x83ae bytes) 000613 1735099866.088791 trunc from 0x0239a9 to 0x0260ec (0x2744 bytes) 000615 1735099866.093522 punch from 0x001d33 to 0x00f3bd (0xd68b bytes) 000616 1735099866.095577 read 0x018f53 thru 0x0231d5 (0xa283 bytes) 000618 1735099866.099239 trunc from 0x0260ed to 0x0204d6 (0x5c16 bytes) 000619 1735099866.104673 trunc from 0x0204d7 to 0x011a30 (0xeaa6 bytes) 000620 1735099866.110160 fallocate 0x030769 thru 0x036bfc (0x6494 bytes) 000621 1735099866.111781 read 0x010210 thru 0x011a30 (0x1821 bytes) 000622 1735099866.112155 mapread 0x0072c5 thru 0x011a30 (0xa76c bytes) 000623 1735099866.112643 read 0x00e0f9 thru 0x011a30 (0x3938 bytes) 000624 1735099866.112885 read 0x00a029 thru 0x011a30 (0x7a08 bytes) 000625 1735099866.113228 trunc from 0x011a31 to 0x0370b1 (0x25681 bytes) 000626 1735099866.117687 fallocate 0x00e9ef thru 0x019ce0 (0xb2f2 bytes) 000627 1735099866.119760 read 0x026472 thru 0x034da9 (0xe938 bytes) 000628 1735099866.123475 fallocate 0x00a157 thru 0x0175b0 (0xd45a bytes) 000629 1735099866.125360 mapwrite 0x001355 thru 0x008c1f (0x78cb bytes) 000630 1735099866.134933 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000631 1735099866.173205 trunc from 0x0370b2 to 0x03e2f9 (0x7248 bytes) 000632 1735099866.179587 MIRROR_EXTEND to 2 mirrors 000634 1735099866.305175 write 0x034e38 thru 0x038736 (0x38ff bytes) 000635 1735099866.320522 trunc from 0x03e2fa to 0x00eba2 (0x2f757 bytes) 000636 1735099866.332457 MIRROR_EXTEND to 3 mirrors 000637 1735099866.443869 fallocate 0x02c7bc thru 0x02f6bc (0x2f01 bytes) 000638 1735099866.456997 mapread 0x000dec thru 0x00b3a5 (0xa5ba bytes) 000639 1735099866.464916 write 0x012be8 thru 0x01646b (0x3884 bytes) 000640 1735099866.466882 punch from 0x010601 to 0x01646b (0x5e6b bytes) 000641 1735099866.471186 mapread 0x005bc0 thru 0x012d44 (0xd185 bytes) 000643 1735099866.474185 mapread 0x005c97 thru 0x00eb52 (0x8ebc bytes) 000644 1735099866.474661 trunc from 0x01646c to 0x001110 (0x1535b bytes) 000645 1735099866.485681 fallocate 0x02480b thru 0x02827b (0x3a71 bytes) 000646 1735099866.487535 fallocate 0x02a0fe thru 0x031520 (0x7423 bytes) 000647 1735099866.489372 mapread 0x000baf thru 0x00f6b4 (0xeb06 bytes) 000649 1735099866.496387 punch from 0x00443e to 0x00dcaa (0x986d bytes) 000650 1735099866.498533 trunc from 0x031521 to 0x0198e4 (0x17c3c bytes) 000651 1735099866.504714 MIRROR_EXTEND to 4 mirrors 000652 1735099866.605094 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000653 1735099866.671145 read 0x0089ab thru 0x0187f8 (0xfe4e bytes) 000654 1735099866.679374 write 0x038490 thru 0x03ffff (0x7b70 bytes) 000655 1735099866.694303 trunc from 0x040000 to 0x017467 (0x28b98 bytes) 000656 1735099866.706600 fallocate 0x007b13 thru 0x00c031 (0x451f bytes) 000657 1735099866.708757 trunc from 0x017468 to 0x02ed5c (0x178f5 bytes) 000658 1735099866.713463 punch from 0x0175e5 to 0x01eb3c (0x7558 bytes) 000659 1735099866.715305 punch from 0x020d6a to 0x0251d4 (0x446b bytes) 000661 1735099866.717581 trunc from 0x02ed5d to 0x001f66 (0x2cdf6 bytes) 000662 1735099866.723017 fallocate 0x00ea1a thru 0x00eea7 (0x48e bytes) 000663 1735099866.725055 punch from 0x000c69 to 0x001f66 (0x12fe bytes) 000664 1735099866.727241 read 0x00007a thru 0x001f66 (0x1eed bytes) 000665 1735099866.729837 write 0x01fcc4 thru 0x02cc87 (0xcfc4 bytes) 000666 1735099866.732800 MIRROR_EXTEND to 4 mirrors 000667 1735099866.845042 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000668 1735099866.913440 write 0x008a15 thru 0x013cb0 (0xb29c bytes) 000669 1735099866.926287 write 0x00d854 thru 0x010316 (0x2ac3 bytes) 000670 1735099866.926927 trunc from 0x02cc88 to 0x01affc (0x11c8b bytes) 000672 1735099866.937708 mapwrite 0x00a5a3 thru 0x00e832 (0x4290 bytes) 000673 1735099866.944768 mapwrite 0x03d27b thru 0x03ffff (0x2d85 bytes) 000675 1735099866.957859 read 0x026517 thru 0x02737b (0xe65 bytes) 000676 1735099866.963486 MIRROR_RESYNC 3 mirrors 000677 1735099867.119978 mapwrite 0x01c0b1 thru 0x0250a6 (0x8ff6 bytes) 000678 1735099867.150742 fallocate 0x00c563 thru 0x00daf2 (0x1590 bytes) 000680 1735099867.152720 mapwrite 0x02a1c8 thru 0x02d53a (0x3373 bytes) 000682 1735099867.159826 mapwrite 0x0096c9 thru 0x01735b (0xdc93 bytes) 000683 1735099867.171060 MIRROR_EXTEND to 4 mirrors 000684 1735099867.316025 write 0x0065a5 thru 0x009ee5 (0x3941 bytes) 000685 1735099867.326376 punch from 0x023f41 to 0x02968d (0x574d bytes) 000686 1735099867.328152 mapwrite 0x03508e thru 0x03e919 (0x988c bytes) 000688 1735099867.341486 fallocate 0x01a0cf thru 0x02278e (0x86c0 bytes) 000689 1735099867.343878 punch from 0x0222c6 to 0x030ef8 (0xec33 bytes) 000690 1735099867.346026 read 0x00074d thru 0x008de6 (0x869a bytes) 000691 1735099867.349525 write 0x012c4d thru 0x01317e (0x532 bytes) 000693 1735099867.355705 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000694 1735099867.403318 read 0x02fe9f thru 0x0365ef (0x6751 bytes) 000695 1735099867.405633 write 0x026114 thru 0x02a3b6 (0x42a3 bytes) 000696 1735099867.408856 trunc from 0x040000 to 0x03719a (0x8e65 bytes) 000697 1735099867.416863 mapread 0x02caec thru 0x031ead (0x53c2 bytes) 000698 1735099867.418705 mapread 0x00ac7c thru 0x015554 (0xa8d9 bytes) 000700 1735099867.423370 punch from 0x02500d to 0x031585 (0xc579 bytes) 000701 1735099867.428279 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000705 1735099867.474059 fallocate 0x0095e9 thru 0x00e850 (0x5268 bytes) 000706 1735099867.476024 fallocate 0x032f4d thru 0x03fc59 (0xcd0d bytes) 000707 1735099867.477709 punch from 0x00638f to 0x00aec9 (0x4b3b bytes) 000708 1735099867.479172 fallocate 0x002b04 thru 0x006245 (0x3742 bytes) 000709 1735099867.480592 mapwrite 0x0279a2 thru 0x02d819 (0x5e78 bytes) 000711 1735099867.492287 fallocate 0x00c9e5 thru 0x00e0e9 (0x1705 bytes) 000712 1735099867.494021 punch from 0x025d38 to 0x03269f (0xc968 bytes) 000713 1735099867.497291 MIRROR_RESYNC 2 mirrors 000714 1735099867.622139 fallocate 0x0255d5 thru 0x025cfe (0x72a bytes) 000716 1735099867.629774 read 0x013c3a thru 0x015924 (0x1ceb bytes) 000717 1735099867.631592 read 0x00b0d6 thru 0x0180b1 (0xcfdc bytes) 000718 1735099867.634585 mapread 0x012a20 thru 0x017d7a (0x535b bytes) 000719 1735099867.635121 write 0x032ebc thru 0x03f8cd (0xca12 bytes) 000720 1735099867.635830 mapwrite 0x005818 thru 0x005f81 (0x76a bytes) 000721 1735099867.641987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000722 1735099867.670012 fallocate 0x03efa2 thru 0x03ffff (0x105e bytes) 000723 1735099867.671537 read 0x006fc3 thru 0x00b05f (0x409d bytes) 000724 1735099867.673836 fallocate 0x0377e6 thru 0x03c6f2 (0x4f0d bytes) 000725 1735099867.675439 trunc from 0x03fc5a to 0x026384 (0x198d5 bytes) 000726 1735099867.683973 read 0x00a17a thru 0x0129e0 (0x8867 bytes) 000728 1735099867.686905 read 0x0051cc thru 0x00d377 (0x81ac bytes) 000731 1735099867.687631 mapread 0x006dd6 thru 0x00c0e5 (0x5310 bytes) 000732 1735099867.688117 read 0x0141c7 thru 0x01cb2f (0x8969 bytes) 000734 1735099867.688723 mapread 0x00f181 thru 0x01995e (0xa7de bytes) 000737 1735099867.690139 mapwrite 0x02cc81 thru 0x031446 (0x47c6 bytes) 000738 1735099867.704362 mapwrite 0x016ad7 thru 0x01d07d (0x65a7 bytes) 000740 1735099867.711764 read 0x0014fe thru 0x005ec4 (0x49c7 bytes) 000741 1735099867.712057 read 0x01db66 thru 0x0246c4 (0x6b5f bytes) 000743 1735099867.712709 trunc from 0x031447 to 0x008560 (0x28ee6 bytes) 000744 1735099867.717682 fallocate 0x02a47b thru 0x036dd8 (0xc95e bytes) 000746 1735099867.720466 read 0x000030 thru 0x00904f (0x9020 bytes) 000749 1735099867.724849 fallocate 0x010f15 thru 0x017616 (0x6702 bytes) 000750 1735099867.726891 mapread 0x001df9 thru 0x0099ff (0x7c07 bytes) 000751 1735099867.727370 fallocate 0x001471 thru 0x00ffc1 (0xeb51 bytes) 000752 1735099867.729247 write 0x02102b thru 0x025f72 (0x4f48 bytes) 000753 1735099867.729925 write 0x01a498 thru 0x01e407 (0x3f70 bytes) 000754 1735099867.730495 punch from 0x034a16 to 0x036dd8 (0x23c3 bytes) 000755 1735099867.732427 read 0x030441 thru 0x036dd8 (0x6998 bytes) 000758 1735099867.736332 mapread 0x00fbb2 thru 0x01fbae (0xfffd bytes) 000759 1735099867.737009 trunc from 0x036dd9 to 0x01f197 (0x17c41 bytes) 000761 1735099867.742599 read 0x0028b8 thru 0x00f8bb (0xd004 bytes) 000764 1735099867.743601 fallocate 0x00f6f0 thru 0x0147d7 (0x50e8 bytes) 000766 1735099867.745755 fallocate 0x013a55 thru 0x014534 (0xae0 bytes) 000767 1735099867.746986 mapwrite 0x015ff2 thru 0x01de2f (0x7e3e bytes) 000769 1735099867.753733 write 0x014c75 thru 0x01826a (0x35f6 bytes) 000770 1735099867.754436 read 0x01e981 thru 0x01f197 (0x817 bytes) 000771 1735099867.754810 mapread 0x0143bf thru 0x0188a1 (0x44e3 bytes) 000773 1735099867.755873 mapread 0x01c7e3 thru 0x01f197 (0x29b5 bytes) 000774 1735099867.756375 mapwrite 0x0309d9 thru 0x0333b4 (0x29dc bytes) 000775 1735099867.768205 trunc from 0x0333b5 to 0x02abc2 (0x87f2 bytes) 000776 1735099867.771664 read 0x006bbf thru 0x0164e3 (0xf925 bytes) 000778 1735099867.772140 read 0x016feb thru 0x023f7e (0xcf94 bytes) 000780 1735099867.772792 write 0x03617f thru 0x03ab5f (0x49e1 bytes) 000782 1735099867.774003 fallocate 0x024b27 thru 0x0323cf (0xd8a9 bytes) 000785 1735099867.776199 trunc from 0x03ab60 to 0x02aba8 (0xffb7 bytes) 000786 1735099867.779754 punch from 0x026be1 to 0x029ffe (0x341e bytes) 000788 1735099867.781286 fallocate 0x00b601 thru 0x0172bb (0xbcbb bytes) 000791 1735099867.783159 write 0x00aee0 thru 0x00c46d (0x158e bytes) 000793 1735099867.783864 mapread 0x0041e6 thru 0x008aff (0x491a bytes) 000794 1735099867.784406 mapread 0x0081b2 thru 0x015e52 (0xdca1 bytes) 000795 1735099867.784961 read 0x004e36 thru 0x00ee47 (0xa012 bytes) 000797 1735099867.785649 mapwrite 0x017e83 thru 0x01e083 (0x6201 bytes) 000798 1735099867.792873 fallocate 0x012097 thru 0x01fe7d (0xdde7 bytes) 000803 1735099867.795766 fallocate 0x016444 thru 0x02009a (0x9c57 bytes) 000805 1735099867.798215 write 0x035ceb thru 0x03ffff (0xa315 bytes) 000806 1735099867.799204 write 0x01d44e thru 0x02b00e (0xdbc1 bytes) 000807 1735099867.801861 fallocate 0x030968 thru 0x0396ed (0x8d86 bytes) 000808 1735099867.803719 write 0x0140b7 thru 0x014e18 (0xd62 bytes) 000810 1735099867.804849 mapwrite 0x03b111 thru 0x03ffff (0x4eef bytes) 000812 1735099867.810792 read 0x0003d5 thru 0x004a34 (0x4660 bytes) 000813 1735099867.811196 write 0x004785 thru 0x00865e (0x3eda bytes) 000814 1735099867.811657 read 0x01903a thru 0x020137 (0x70fe bytes) 000815 1735099867.811985 punch from 0x02c47f to 0x035829 (0x93ab bytes) 000818 1735099867.814064 fallocate 0x024a32 thru 0x032851 (0xde20 bytes) 000819 1735099867.815617 mapwrite 0x00cfe1 thru 0x00e613 (0x1633 bytes) 000820 1735099867.820114 fallocate 0x00cccd thru 0x01a94c (0xdc80 bytes) 000821 1735099867.821770 mapwrite 0x006e2f thru 0x00b3be (0x4590 bytes) 000822 1735099867.827220 fallocate 0x014813 thru 0x016176 (0x1964 bytes) 000823 1735099867.828976 write 0x03270a thru 0x03ffff (0xd8f6 bytes) 000825 1735099867.832310 read 0x0293c0 thru 0x02d7e9 (0x442a bytes) 000827 1735099867.833239 punch from 0x03c455 to 0x03ffff (0x3bab bytes) 000829 1735099867.838019 read 0x01a18b thru 0x02023e (0x60b4 bytes) 000830 1735099867.838448 mapread 0x00768e thru 0x008a26 (0x1399 bytes) 000831 1735099867.838994 write 0x025cf4 thru 0x0296fc (0x3a09 bytes) 000832 1735099867.839619 read 0x00b9cf thru 0x00dbef (0x2221 bytes) 000833 1735099867.839995 mapwrite 0x0004ff thru 0x000970 (0x472 bytes) 000835 1735099867.845259 fallocate 0x0190d6 thru 0x019cf1 (0xc1c bytes) 000836 1735099867.846978 fallocate 0x00f555 thru 0x01b6c6 (0xc172 bytes) 000837 1735099867.848515 mapread 0x00a844 thru 0x018ce6 (0xe4a3 bytes) 000839 1735099867.849383 read 0x0028c7 thru 0x00c8b3 (0x9fed bytes) 000842 1735099867.850313 punch from 0x014fb5 to 0x019c26 (0x4c72 bytes) 000843 1735099867.852310 write 0x03d51a thru 0x03ffff (0x2ae6 bytes) 000844 1735099867.855600 punch from 0x02da28 to 0x03c09a (0xe673 bytes) 000846 1735099867.858160 mapread 0x015f69 thru 0x02524f (0xf2e7 bytes) 000848 1735099867.859059 punch from 0x0054a7 to 0x013ab6 (0xe610 bytes) 000849 1735099867.861778 read 0x0207e3 thru 0x026644 (0x5e62 bytes) 000850 1735099867.862116 mapread 0x0376fc thru 0x03acfd (0x3602 bytes) 000853 1735099867.866652 punch from 0x0122b9 to 0x018515 (0x625d bytes) 000854 1735099867.868368 write 0x01350a thru 0x01ba04 (0x84fb bytes) 000855 1735099867.870141 mapread 0x01338b thru 0x01ce5e (0x9ad4 bytes) 000857 1735099867.870857 trunc from 0x040000 to 0x02f1a6 (0x10e59 bytes) 000858 1735099867.874223 fallocate 0x026329 thru 0x031030 (0xad08 bytes) 000859 1735099867.875627 fallocate 0x02e771 thru 0x03299c (0x422c bytes) 000860 1735099867.876999 mapread 0x013d8b thru 0x022acb (0xed41 bytes) 000861 1735099867.877426 mapwrite 0x034876 thru 0x03dff3 (0x977e bytes) 000862 1735099867.898686 punch from 0x024ca8 to 0x033c72 (0xefcb bytes) 000863 1735099867.900292 mapwrite 0x01fe69 thru 0x022f51 (0x30e9 bytes) 000864 1735099867.904248 read 0x01f7ef thru 0x02e3e7 (0xebf9 bytes) 000865 1735099867.906775 fallocate 0x006cc7 thru 0x00cc0d (0x5f47 bytes) 000867 1735099867.908864 mapread 0x0280dc thru 0x02be14 (0x3d39 bytes) 000868 1735099867.909435 punch from 0x02f9f0 to 0x03a153 (0xa764 bytes) 000870 1735099867.911607 read 0x0243fe thru 0x02b918 (0x751b bytes) 000873 1735099867.912752 fallocate 0x02756f thru 0x034dda (0xd86c bytes) 000876 1735099867.916332 punch from 0x03dc03 to 0x03dff3 (0x3f1 bytes) 000877 1735099867.918457 read 0x01dc5f thru 0x02d6f7 (0xfa99 bytes) 000878 1735099867.918876 read 0x00bf28 thru 0x018d13 (0xcdec bytes) 000879 1735099867.919276 fallocate 0x015127 thru 0x01aba5 (0x5a7f bytes) 000881 1735099867.921784 mapwrite 0x022383 thru 0x031a73 (0xf6f1 bytes) 000882 1735099867.931277 fallocate 0x02db93 thru 0x0370da (0x9548 bytes) 000883 1735099867.933195 fallocate 0x039382 thru 0x03ffff (0x6c7e bytes) 000884 1735099867.934796 punch from 0x001474 to 0x0025af (0x113c bytes) 000885 1735099867.936021 fallocate 0x003ece thru 0x00da66 (0x9b99 bytes) 000889 1735099867.938751 read 0x002a8b thru 0x00c1ac (0x9722 bytes) 000890 1735099867.940743 read 0x00f5f9 thru 0x01adcd (0xb7d5 bytes) 000892 1735099867.941332 mapread 0x0023ab thru 0x00aca6 (0x88fc bytes) 000893 1735099867.941717 write 0x02667f thru 0x02f4c9 (0x8e4b bytes) 000894 1735099867.942506 write 0x00aea3 thru 0x0102ae (0x540c bytes) 000895 1735099867.943101 fallocate 0x01d43c thru 0x028a13 (0xb5d8 bytes) 000898 1735099867.944998 mapread 0x01a875 thru 0x01ad64 (0x4f0 bytes) 000899 1735099867.945335 mapread 0x005cc1 thru 0x00fb2e (0x9e6e bytes) 000900 1735099867.945681 punch from 0x036b28 to 0x03762b (0xb04 bytes) 000901 1735099867.948258 fallocate 0x02a3b0 thru 0x02f585 (0x51d6 bytes) 000902 1735099867.950101 write 0x01f2eb thru 0x025e67 (0x6b7d bytes) 000903 1735099867.950616 punch from 0x013032 to 0x01bfd2 (0x8fa1 bytes) 000904 1735099867.952350 punch from 0x037d9b to 0x03dff3 (0x6259 bytes) 000905 1735099867.953732 read 0x01f3a3 thru 0x025b0c (0x676a bytes) 000908 1735099867.954926 mapread 0x01e484 thru 0x023b3b (0x56b8 bytes) 000910 1735099867.955518 read 0x032c2d thru 0x038ac0 (0x5e94 bytes) 000914 1735099867.959344 write 0x00e494 thru 0x015116 (0x6c83 bytes) 000915 1735099867.960004 mapread 0x028595 thru 0x033429 (0xae95 bytes) 000916 1735099867.961868 MIRROR_EXTEND to 2 mirrors 000917 1735099868.075548 trunc from 0x03dff4 to 0x01a684 (0x2396f bytes) 000918 1735099868.086579 fallocate 0x02b599 thru 0x031d68 (0x67d0 bytes) 000919 1735099868.088267 fallocate 0x01c6b9 thru 0x01dfa1 (0x18e9 bytes) 000920 1735099868.090115 mapread 0x00c9a2 thru 0x014273 (0x78d2 bytes) 000921 1735099868.095766 trunc from 0x01a685 to 0x008b60 (0x11b24 bytes) 000922 1735099868.102648 fallocate 0x033ead thru 0x03e0dd (0xa231 bytes) 000923 1735099868.104829 read 0x003d2a thru 0x003e76 (0x14d bytes) 000924 1735099868.106980 fallocate 0x032cff thru 0x03dbf0 (0xaef2 bytes) 000925 1735099868.108990 punch from 0x000ffb to 0x008b60 (0x7b66 bytes) 000929 1735099868.111986 fallocate 0x0175f8 thru 0x02479f (0xd1a8 bytes) 000930 1735099868.114721 read 0x004a6e thru 0x008b60 (0x40f3 bytes) 000931 1735099868.117813 mapread 0x005341 thru 0x008b60 (0x3820 bytes) 000932 1735099868.118415 mapwrite 0x011508 thru 0x01af56 (0x9a4f bytes) 000933 1735099868.133670 write 0x033155 thru 0x039c2d (0x6ad9 bytes) 000934 1735099868.134575 mapwrite 0x00a7c6 thru 0x0101c4 (0x59ff bytes) 000936 1735099868.141010 mapread 0x0205fc thru 0x02e49b (0xdea0 bytes) 000937 1735099868.144404 mapwrite 0x03a242 thru 0x03c75f (0x251e bytes) 000938 1735099868.156808 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) 000939 1735099868.186035 fallocate 0x0266ed thru 0x032541 (0xbe55 bytes) 000940 1735099868.187585 read 0x011e51 thru 0x01a687 (0x8837 bytes) 000941 1735099868.190111 fallocate 0x012683 thru 0x019559 (0x6ed7 bytes) 000945 1735099868.192577 write 0x025baf thru 0x03245d (0xc8af bytes) 000946 1735099868.196937 mapread 0x0193f3 thru 0x0293b4 (0xffc2 bytes) 000947 1735099868.200481 trunc from 0x03c760 to 0x02f145 (0xd61a bytes) 000948 1735099868.203914 write 0x02db26 thru 0x039c00 (0xc0db bytes) 000950 1735099868.205249 write 0x01fef9 thru 0x020c3c (0xd44 bytes) 000952 1735099868.206031 mapread 0x03396e thru 0x039c00 (0x6293 bytes) 000953 1735099868.206568 fallocate 0x022ed7 thru 0x028d71 (0x5e9b bytes) 000954 1735099868.208384 write 0x022299 thru 0x02f485 (0xd1ed bytes) 000955 1735099868.209191 mapwrite 0x02af99 thru 0x0320c7 (0x712f bytes) 000956 1735099868.215054 read 0x02ad6c thru 0x02f4f2 (0x4787 bytes) 000957 1735099868.215328 mapread 0x006aaa thru 0x00993e (0x2e95 bytes) 000958 1735099868.217828 read 0x0309b0 thru 0x0326c4 (0x1d15 bytes) 000959 1735099868.218231 mapwrite 0x019811 thru 0x028ca3 (0xf493 bytes) 000960 1735099868.225041 trunc from 0x039c01 to 0x01fd29 (0x19ed7 bytes) 000961 1735099868.228863 write 0x0011c5 thru 0x007ae9 (0x6925 bytes) 000964 1735099868.230278 write 0x028007 thru 0x030d9e (0x8d98 bytes) 000965 1735099868.231125 mapwrite 0x0073b4 thru 0x00c0e5 (0x4d32 bytes) 000966 1735099868.237834 mapwrite 0x00c4a0 thru 0x015db1 (0x9912 bytes) 000968 1735099868.244536 trunc from 0x030d9f to 0x03f96f (0xebd1 bytes) 000969 1735099868.248893 fallocate 0x00d367 thru 0x013ce9 (0x6983 bytes) 000970 1735099868.250669 fallocate 0x0360fb thru 0x03ffff (0x9f05 bytes) 000971 1735099868.252929 punch from 0x00a090 to 0x010cd9 (0x6c4a bytes) 000973 1735099868.254778 write 0x016b9a thru 0x022f7b (0xc3e2 bytes) 000976 1735099868.258842 read 0x004c24 thru 0x00a497 (0x5874 bytes) 000977 1735099868.260670 MIRROR_EXTEND to 2 mirrors 000978 1735099868.375680 MIRROR_EXTEND to 3 mirrors 000979 1735099868.513581 trunc from 0x03f970 to 0x01d268 (0x22707 bytes) 000980 1735099868.527246 read 0x004f2c thru 0x00a8ce (0x59a3 bytes) 000981 1735099868.529488 fallocate 0x0074e3 thru 0x00d498 (0x5fb6 bytes) 000982 1735099868.530931 read 0x005e4e thru 0x00669d (0x850 bytes) 000983 1735099868.531178 mapread 0x001461 thru 0x006132 (0x4cd2 bytes) 000984 1735099868.533571 mapwrite 0x0100c0 thru 0x0195f7 (0x9538 bytes) 000985 1735099868.539834 fallocate 0x03e127 thru 0x03ffff (0x1ed9 bytes) 000986 1735099868.541490 punch from 0x01d337 to 0x023232 (0x5efc bytes) 000987 1735099868.542777 fallocate 0x021782 thru 0x02a0ed (0x896c bytes) 000988 1735099868.544276 fallocate 0x0220be thru 0x02765d (0x55a0 bytes) 000989 1735099868.545681 trunc from 0x040000 to 0x03508c (0xaf73 bytes) 000990 1735099868.549729 punch from 0x000503 to 0x002a77 (0x2575 bytes) 000992 1735099868.551097 read 0x0343ad thru 0x03508c (0xce0 bytes) 000993 1735099868.553344 fallocate 0x0167e1 thru 0x023483 (0xcca3 bytes) 000994 1735099868.554572 write 0x022c1c thru 0x02ef61 (0xc346 bytes) 000996 1735099868.555574 fallocate 0x00c524 thru 0x00e38d (0x1e6a bytes) 000997 1735099868.556897 mapwrite 0x028631 thru 0x036191 (0xdb61 bytes) 000999 1735099868.569133 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x947:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (16s) == sanity-flr test 200a: stress test ===================== 23:11:09 (1735099869) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7903616 .. Extending file size to 4794720 .. Extending file size to 7975744 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8617472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8598560 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8951808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9249504 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9094112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9111296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9340640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9374112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5134 5135 5137 5138 5140 done 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:12:14 (1735099934) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4493056 .. Extending file size to 5691904 .. failed Extending file size to 8560928 .. Extending file size to 7185472 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7672960 .. Extending file size to 8264064 .. done Extending file size to 9069984 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8831360 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 8240 8241 8243 8245 8247 done 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (70s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:13:24 (1735100004) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2936 1284752 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15664 3591356 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21408 3585612 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37072 7176968 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2936 1284752 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15664 3591356 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21408 3585612 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37072 7176968 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0498462 s, 42.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.137138 s, 15.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:13:34 (1735100014) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0632782 s, 33.1 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 23:13:38 (1735100018) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:13:41 (1735100021) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0289631 s, 72.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:13:43 (1735100023) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0421568 s, 49.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:13:46 (1735100026) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0323747 s, 64.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:10 (1735100050) Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305526 s, 34.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb0:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:44 (1735100084) Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0331673 s, 63.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:15:17 (1735100117) Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg342-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg342-server: Pool lustre.test_204f created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0332586 s, 63.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg342-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:16:07 (1735100167) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:16:09 (1735100169) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:16:11 (1735100171) oleg342-server: Pool lustre.test_206 created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg342-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:16:25 (1735100185) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273392 s, 38.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128695 s, 81.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8e74369 Mirror 3: 0x8e74369 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:16:28 (1735100188) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:16:30 (1735100190) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:16:31 (1735100191) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0205329 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2240000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1bc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1bc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1bc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x18a0000 Read this (4 bytes): BBBB 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:16:46 (1735100206) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00238382 s, 3.4 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:16:47 (1735100207) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121782 s, 86.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-24 23:16:48.000000000 -0500 Modify: 2024-12-24 23:16:48.000000000 -0500 Change: 2024-12-24 23:16:48.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:16:50 (1735100210) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100992 s, 104 MB/s oleg342-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg342-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 oleg342-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg342-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg342-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:16:59 (1735100219) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0484451 s, 845 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00801786 s, 511 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 24 23:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 24 23:17 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 758 sec ============ 23:17:02 (1735100222) === sanity-flr: start cleanup 23:17:02 (1735100222) === === sanity-flr: finish cleanup 23:17:05 (1735100225) ===