-----============= acceptance-small: sanity-flr ============----- Tue Apr 8 23:44:40 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg419-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:44:45 (1744170285) === oleg419-client.virtnet: executing check_config_client /mnt/lustre oleg419-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg419-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b64bf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b64bf800.idle_timeout=debug disable quota as required oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:44:54 (1744170294) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7628] == sanity-flr test 0a: lfs mirror create with -N option == 23:44:55 (1744170295) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:44:58 (1744170298) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:44:59 (1744170299) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:45:00 (1744170300) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:45:04 (1744170304) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:45:05 (1744170305) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 23:45:05 (1744170305) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b64bf800.stats=clear osc.lustre-OST0001-osc-ffff8800b64bf800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:45:07 (1744170307) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:45:09 (1744170309) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:45:11 (1744170311) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:45:11 (1744170311) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 23:45:13 (1744170313) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:45:15 (1744170315) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:45:16 (1744170316) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:45:18 (1744170318) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:45:20 (1744170320) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:45:22 (1744170322) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.353192 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229644 s, 45.7 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:45:24 (1744170324) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.0470507 s, 156 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:45:26 (1744170326) Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:45:49 (1744170349) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.160216 s, 72.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:46:01 (1744170361) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:46:20 (1744170380) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.692178 s, 75.7 MB/s -rw-r--r-- 1 root root 52428800 Apr 8 23:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.195s user 0m0.000s sys 0m0.098s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.185s user 0m0.000s sys 0m0.102s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.235s user 0m0.000s sys 0m0.110s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b64bf800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:46:42 (1744170402) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:46:43 (1744170403) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:46:44 (1744170404) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:46:45 (1744170405) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 23:46:46 (1744170406) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.120747 s, 104 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:46:49 (1744170409) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00982964 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0082984 s, 126 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:46:50 (1744170410) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:46:53 (1744170413) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000511863 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00042858 s, 2.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000640617 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000658551 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:46:55 (1744170415) create test files with size 15744352 .. 953f2efcca19e8e2b024ccb8e1979504 - 4ae8d7dfe9b49e978f1e21812a42b7a5 - df088a48f4ea34712b5d6b0ba04ff3ec - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 23:46:59 (1744170419) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:47:02 (1744170422) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:47:03 (1744170423) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020425 s, 51.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00886083 s, 118 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:47:05 (1744170425) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0193937 s, 108 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0797937 s, 52.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0329978 s, 63.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:47:08 (1744170428) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 23:47:08 (1744170428) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:47:09 (1744170429) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119235 s, 87.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:47:11 (1744170431) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0850274 s, 37.0 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:47:13 (1744170433) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:47:15 (1744170435) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.090947 s, 115 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:47:38 (1744170458) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a5943800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0657851 s, 159 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b5877000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:47:41 (1744170461) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:47:47 (1744170467) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:47:54 (1744170474) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 23:47:54 (1744170474) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0664972 s, 47.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.04961 s, 63.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9f9288a909a8c32255304bc1211aa2fc - mirror 1 checksum: 7781a4f26b672922c9430c1cd0065187 - mirror 2 checksum: 9f9288a909a8c32255304bc1211aa2fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 23:47:55 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 23:47:55 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9f9288a909a8c32255304bc1211aa2fc - mirror 1 checksum: 7781a4f26b672922c9430c1cd0065187 - mirror 2 checksum: 7781a4f26b672922c9430c1cd0065187 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 23:47:55 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 23:47:55 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:47:56 (1744170476) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00916082 s, 114 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 527360.. 528383: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:47:58 (1744170478) Failing mds1 on oleg419-server Stopping /mnt/lustre-mds1 (opts:) on oleg419-server 23:48:05 (1744170485) shut down facet: mds1 facet_host: oleg419-server facet_failover_host: oleg419-server Failover mds1 to oleg419-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 23:48:18 (1744170498) targets are mounted 23:48:18 (1744170498) facet_failover done oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111622 s, 93.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:48:25 (1744170505) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0079306 s, 2.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00432112 s, 4.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:48:27 (1744170507) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00809569 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00344217 s, 5.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:48:29 (1744170509) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00183225 s, 8.9 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:48:31 (1744170511) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00897637 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:48:33 (1744170513) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00143986 s, 2.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00623824 s, 657 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:48:34 (1744170514) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:48:38 (1744170518) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:48:56 (1744170536) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:49:12 (1744170552) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:49:29 (1744170569) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 23:49:30 (1744170570) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26314 mirror split pid 26315 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26314 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26315 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 23:50:32 (1744170632) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 633 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1744170633.087008 write 0x037329 thru 0x03c031 (0x4d09 bytes) 000002 1744170633.090898 mapwrite 0x024f11 thru 0x033eb7 (0xefa7 bytes) fallocating to largest ever: 0x10c62 000003 1744170633.108196 fallocate 0x01017d thru 0x010c61 (0xae5 bytes) 000004 1744170633.110147 read 0x035e2f thru 0x03becd (0x609f bytes) truncating to largest ever: 0x34d43 000006 1744170633.111326 trunc from 0x03c032 to 0x034d42 (0x72ef bytes) 000007 1744170633.115635 mapwrite 0x00b51a thru 0x00dd71 (0x2858 bytes) 000010 1744170633.123176 mapwrite 0x00cd08 thru 0x01664b (0x9944 bytes) 000011 1744170633.131353 read 0x022d8a thru 0x02d58e (0xa805 bytes) 000013 1744170633.132012 read 0x017e9f thru 0x026002 (0xe164 bytes) 000015 1744170633.134592 read 0x024534 thru 0x0344ea (0xffb7 bytes) 000016 1744170633.134989 read 0x00c694 thru 0x010abb (0x4428 bytes) 000017 1744170633.135245 trunc from 0x034d43 to 0x017fb9 (0x1cd89 bytes) 000018 1744170633.139424 write 0x008e7d thru 0x0105c5 (0x7749 bytes) 000019 1744170633.140263 mapread 0x00f77e thru 0x017fb9 (0x883c bytes) 000021 1744170633.141219 punch from 0x00868d to 0x00d31b (0x4c8f bytes) 000023 1744170633.146488 punch from 0x014b35 to 0x017fb9 (0x3485 bytes) 000024 1744170633.148396 mapread 0x012806 thru 0x017fb9 (0x57b4 bytes) 000026 1744170633.152865 write 0x032390 thru 0x032ef0 (0xb61 bytes) 000028 1744170633.153753 trunc from 0x032ef1 to 0x01b9e6 (0x1750a bytes) 000029 1744170633.158827 mapwrite 0x0297e9 thru 0x0340af (0xa8c7 bytes) 000030 1744170633.175225 mapwrite 0x017cbf thru 0x024edc (0xd21e bytes) 000031 1744170633.182627 mapwrite 0x00dca8 thru 0x019ef3 (0xc24c bytes) 000032 1744170633.192564 MIRROR_EXTEND to 2 mirrors 000033 1744170633.326630 trunc from 0x0340b0 to 0x01bf3b (0x18174 bytes) 000034 1744170633.345732 trunc from 0x01bf3c to 0x028bbe (0xcc83 bytes) 000036 1744170633.354218 MIRROR_RESYNC 2 mirrors 000037 1744170633.461141 fallocate 0x002c36 thru 0x012441 (0xf80c bytes) fallocating to largest ever: 0x40000 000038 1744170633.471400 fallocate 0x03635f thru 0x03ffff (0x9ca1 bytes) 000039 1744170633.474084 MIRROR_EXTEND to 3 mirrors 000040 1744170633.585181 write 0x024520 thru 0x03188e (0xd36f bytes) 000041 1744170633.599219 trunc from 0x040000 to 0x03b50f (0x4af0 bytes) 000042 1744170633.610015 trunc from 0x03b510 to 0x03d38b (0x1e7c bytes) 000043 1744170633.615990 MIRROR_RESYNC 3 mirrors 000044 1744170633.798922 mapwrite 0x00e95b thru 0x0178c6 (0x8f6c bytes) 000045 1744170633.834061 fallocate 0x015ad5 thru 0x01770c (0x1c38 bytes) 000047 1744170633.836406 read 0x03bb59 thru 0x03c6e3 (0xb8b bytes) 000048 1744170633.838385 read 0x034a79 thru 0x0387c8 (0x3d50 bytes) 000051 1744170633.842435 mapread 0x035173 thru 0x03d38b (0x8219 bytes) 000052 1744170633.843037 read 0x0117d6 thru 0x019895 (0x80c0 bytes) 000053 1744170633.843427 write 0x00d297 thru 0x00fd87 (0x2af1 bytes) 000054 1744170633.844092 write 0x03d10c thru 0x03ffff (0x2ef4 bytes) 000055 1744170633.844791 punch from 0x03af9d to 0x03ffff (0x5063 bytes) 000056 1744170633.855235 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000057 1744170633.913474 fallocate 0x02e345 thru 0x034141 (0x5dfd bytes) 000058 1744170633.915258 mapwrite 0x00bff5 thru 0x010dd6 (0x4de2 bytes) 000061 1744170633.925968 read 0x031b38 thru 0x03ffff (0xe4c8 bytes) 000063 1744170633.929365 mapread 0x00b46e thru 0x00e225 (0x2db8 bytes) 000065 1744170633.930316 read 0x03af79 thru 0x03ffff (0x5087 bytes) 000066 1744170633.930684 write 0x02f8ac thru 0x03da83 (0xe1d8 bytes) 000067 1744170633.936089 fallocate 0x01ae61 thru 0x026d27 (0xbec7 bytes) 000069 1744170633.938460 punch from 0x03101c to 0x03c71f (0xb704 bytes) 000070 1744170633.943195 punch from 0x01a587 to 0x025d03 (0xb77d bytes) 000072 1744170633.945318 write 0x03756c thru 0x03ffff (0x8a94 bytes) 000073 1744170633.948136 trunc from 0x040000 to 0x027286 (0x18d79 bytes) 000074 1744170633.958400 punch from 0x005b73 to 0x00ca2f (0x6ebd bytes) 000076 1744170633.960177 read 0x000400 thru 0x00f37a (0xef7b bytes) 000077 1744170633.962738 fallocate 0x032fc3 thru 0x03eda7 (0xbde5 bytes) 000078 1744170633.964697 trunc from 0x027287 to 0x000f4d (0x26339 bytes) 000080 1744170633.969798 punch from 0x0003db to 0x000f4d (0xb73 bytes) 000081 1744170633.971617 read 0x0008b8 thru 0x000f4d (0x696 bytes) 000082 1744170633.973397 read 0x0000d7 thru 0x000f4d (0xe77 bytes) 000083 1744170633.973854 read 0x000b45 thru 0x000f4d (0x409 bytes) 000084 1744170633.974246 fallocate 0x036624 thru 0x03ffff (0x99dc bytes) 000085 1744170633.976089 punch from 0x01dd7a to 0x02390b (0x5b92 bytes) 000087 1744170633.979287 MIRROR_RESYNC 2 mirrors 000088 1744170634.084499 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000089 1744170634.120247 mapwrite 0x026aef thru 0x028632 (0x1b44 bytes) 000090 1744170634.135664 mapwrite 0x01fc32 thru 0x02a00b (0xa3da bytes) 000091 1744170634.145523 mapread 0x02e835 thru 0x0310a6 (0x2872 bytes) 000092 1744170634.150672 write 0x001fcd thru 0x005056 (0x308a bytes) 000093 1744170634.155863 mapread 0x020166 thru 0x020fe3 (0xe7e bytes) 000094 1744170634.156477 read 0x039137 thru 0x03a5c0 (0x148a bytes) 000095 1744170634.158633 mapread 0x00de52 thru 0x011820 (0x39cf bytes) 000096 1744170634.161970 punch from 0x02bd93 to 0x0319ef (0x5c5d bytes) 000099 1744170634.165013 write 0x039154 thru 0x03ffff (0x6eac bytes) 000100 1744170634.165890 punch from 0x03099a to 0x03207b (0x16e2 bytes) 000101 1744170634.168389 mapwrite 0x032692 thru 0x039d1e (0x768d bytes) 000102 1744170634.179437 mapwrite 0x00a8bd thru 0x01077b (0x5ebf bytes) 000103 1744170634.184725 trunc from 0x040000 to 0x03e6fd (0x1902 bytes) 000105 1744170634.189095 write 0x026eea thru 0x034f0f (0xe026 bytes) 000106 1744170634.190121 fallocate 0x015778 thru 0x015fcc (0x855 bytes) 000107 1744170634.191989 punch from 0x03600f to 0x03e6fd (0x86ef bytes) 000108 1744170634.193798 mapwrite 0x02f4f5 thru 0x0344c3 (0x4fcf bytes) 000110 1744170634.201034 mapwrite 0x018ba7 thru 0x022592 (0x99ec bytes) 000111 1744170634.207775 fallocate 0x0316df thru 0x035510 (0x3e32 bytes) 000112 1744170634.209315 read 0x00b304 thru 0x00f07a (0x3d77 bytes) 000113 1744170634.209659 mapwrite 0x0346f1 thru 0x03fc86 (0xb596 bytes) 000115 1744170634.225105 mapwrite 0x028031 thru 0x036a9c (0xea6c bytes) 000116 1744170634.233230 mapread 0x01c0ba thru 0x01ff1f (0x3e66 bytes) 000117 1744170634.233832 mapread 0x026e11 thru 0x031273 (0xa463 bytes) 000119 1744170634.234994 punch from 0x01aac9 to 0x026bd4 (0xc10c bytes) 000122 1744170634.237971 read 0x015d4b thru 0x016488 (0x73e bytes) 000123 1744170634.238352 mapread 0x01a728 thru 0x022d91 (0x866a bytes) 000124 1744170634.241192 trunc from 0x03fc87 to 0x02b20c (0x14a7a bytes) 000125 1744170634.246447 mapwrite 0x0225a5 thru 0x029055 (0x6ab1 bytes) 000126 1744170634.254484 fallocate 0x03b8dc thru 0x03b952 (0x77 bytes) 000127 1744170634.256584 mapread 0x01744c thru 0x02262d (0xb1e2 bytes) 000128 1744170634.257195 mapread 0x00cd2b thru 0x01a5f4 (0xd8ca bytes) 000130 1744170634.258038 fallocate 0x0334d2 thru 0x03b083 (0x7bb2 bytes) 000131 1744170634.260053 punch from 0x02e776 to 0x034df8 (0x6683 bytes) 000132 1744170634.261898 trunc from 0x03b084 to 0x02abb6 (0x104cd bytes) 000133 1744170634.266681 mapwrite 0x00a8b5 thru 0x00e862 (0x3fae bytes) 000134 1744170634.274134 read 0x01e945 thru 0x0255da (0x6c96 bytes) 000135 1744170634.274444 mapread 0x00ca08 thru 0x01b3ac (0xe9a5 bytes) 000136 1744170634.275051 mapread 0x0235d2 thru 0x02abb6 (0x75e5 bytes) 000138 1744170634.275913 punch from 0x00abb4 to 0x015250 (0xa69d bytes) 000140 1744170634.278725 write 0x0346cf thru 0x03a4ab (0x5ddd bytes) 000141 1744170634.279500 mapwrite 0x02375f thru 0x02d7e0 (0xa082 bytes) 000143 1744170634.292293 read 0x02257c thru 0x0242c9 (0x1d4e bytes) 000144 1744170634.292634 write 0x0150db thru 0x01ae8a (0x5db0 bytes) 000145 1744170634.293376 mapread 0x02b85d thru 0x03027f (0x4a23 bytes) 000146 1744170634.293965 mapread 0x036169 thru 0x03a4ab (0x4343 bytes) 000148 1744170634.298883 read 0x025da3 thru 0x03115a (0xb3b8 bytes) 000150 1744170634.299646 trunc from 0x03a4ac to 0x0158e0 (0x24bcb bytes) 000153 1744170634.306253 mapread 0x005d66 thru 0x014b13 (0xedae bytes) 000154 1744170634.306887 trunc from 0x0158e1 to 0x016d9e (0x14be bytes) 000155 1744170634.311766 write 0x008bfc thru 0x00bef7 (0x32fc bytes) 000158 1744170634.312992 trunc from 0x016d9f to 0x02ebf3 (0x17e55 bytes) 000159 1744170634.317414 mapwrite 0x00642e thru 0x00e197 (0x7d6a bytes) 000160 1744170634.325328 read 0x00e9d7 thru 0x0191e2 (0xa80c bytes) 000161 1744170634.328189 mapwrite 0x026497 thru 0x032b49 (0xc6b3 bytes) 000163 1744170634.344451 fallocate 0x00a9a2 thru 0x0165b0 (0xbc0f bytes) 000164 1744170634.346432 read 0x02fab8 thru 0x032b49 (0x3092 bytes) 000165 1744170634.346786 punch from 0x01d453 to 0x01ffac (0x2b5a bytes) 000166 1744170634.348899 punch from 0x012790 to 0x0159d1 (0x3242 bytes) 000168 1744170634.351691 fallocate 0x010008 thru 0x018149 (0x8142 bytes) 000174 1744170634.355470 punch from 0x024f10 to 0x030dc2 (0xbeb3 bytes) 000175 1744170634.357720 fallocate 0x02440c thru 0x02ff8d (0xbb82 bytes) 000176 1744170634.359419 mapread 0x007d1d thru 0x016c10 (0xeef4 bytes) 000178 1744170634.363841 fallocate 0x01efc3 thru 0x0273c2 (0x8400 bytes) 000179 1744170634.366115 mapread 0x02c8f9 thru 0x02e6b5 (0x1dbd bytes) 000181 1744170634.367088 trunc from 0x032b4a to 0x0235d2 (0xf577 bytes) 000182 1744170634.372243 mapwrite 0x01faea thru 0x023a7f (0x3f96 bytes) 000183 1744170634.385427 fallocate 0x03d816 thru 0x03ffff (0x27ea bytes) 000184 1744170634.387118 mapread 0x017278 thru 0x023a7f (0xc808 bytes) 000185 1744170634.387737 punch from 0x01c750 to 0x023a7f (0x7330 bytes) 000186 1744170634.389903 trunc from 0x023a80 to 0x003653 (0x2042c bytes) 000187 1744170634.395677 trunc from 0x003654 to 0x019803 (0x161b0 bytes) 000188 1744170634.400385 read 0x00d5df thru 0x013a55 (0x6477 bytes) 000189 1744170634.404282 fallocate 0x01be3f thru 0x01e1f6 (0x23b8 bytes) 000190 1744170634.406695 trunc from 0x01e1f7 to 0x00613c (0x180ba bytes) 000191 1744170634.414116 read 0x003660 thru 0x00613c (0x2add bytes) 000192 1744170634.414555 punch from 0x00035b to 0x003303 (0x2fa9 bytes) 000193 1744170634.416872 mapread 0x001b84 thru 0x00613c (0x45b9 bytes) 000195 1744170634.419891 fallocate 0x02ead6 thru 0x033857 (0x4d82 bytes) 000197 1744170634.422042 mapwrite 0x005130 thru 0x006173 (0x1044 bytes) 000198 1744170634.428654 punch from 0x012949 to 0x01ba82 (0x913a bytes) 000199 1744170634.431464 punch from 0x01565f to 0x02454a (0xeeec bytes) 000201 1744170634.434211 fallocate 0x03777e thru 0x03ffff (0x8882 bytes) 000202 1744170634.436064 fallocate 0x0028a6 thru 0x0095a4 (0x6cff bytes) 000203 1744170634.438214 mapread 0x01cce2 thru 0x0219d2 (0x4cf1 bytes) 000204 1744170634.442599 punch from 0x02978e to 0x02fb13 (0x6386 bytes) 000205 1744170634.445136 punch from 0x024dda to 0x02ea1e (0x9c45 bytes) 000206 1744170634.446871 fallocate 0x01684a thru 0x0236ac (0xce63 bytes) 000209 1744170634.449713 mapread 0x015268 thru 0x01a55d (0x52f6 bytes) 000210 1744170634.450662 punch from 0x00fc4b to 0x01a722 (0xaad8 bytes) 000211 1744170634.452917 fallocate 0x031cac thru 0x0370e2 (0x5437 bytes) 000212 1744170634.454688 write 0x00142a thru 0x003755 (0x232c bytes) 000213 1744170634.455142 fallocate 0x036148 thru 0x038fbd (0x2e76 bytes) 000214 1744170634.457758 mapread 0x0183f3 thru 0x02337f (0xaf8d bytes) 000215 1744170634.463651 trunc from 0x0370e3 to 0x0231f9 (0x13ee9 bytes) 000217 1744170634.469525 MIRROR_EXTEND to 2 mirrors 000218 1744170634.589233 fallocate 0x03e22d thru 0x03ffff (0x1dd3 bytes) 000219 1744170634.604886 mapread 0x01ed69 thru 0x0249ca (0x5c62 bytes) 000220 1744170634.612739 write 0x029170 thru 0x02f966 (0x67f7 bytes) 000221 1744170634.617112 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000222 1744170634.660364 mapread 0x010b7d thru 0x01ba75 (0xaef9 bytes) 000223 1744170634.668747 read 0x0148d2 thru 0x01d2de (0x8a0d bytes) 000225 1744170634.669771 read 0x0135cd thru 0x01bde9 (0x881d bytes) 000226 1744170634.671680 MIRROR_EXTEND to 2 mirrors 000229 1744170634.822578 mapread 0x0211d1 thru 0x024da6 (0x3bd6 bytes) 000230 1744170634.834505 trunc from 0x040000 to 0x018278 (0x27d87 bytes) 000231 1744170634.852785 trunc from 0x018279 to 0x035fbf (0x1dd47 bytes) 000232 1744170634.860465 write 0x016a34 thru 0x017e12 (0x13df bytes) 000235 1744170634.864495 write 0x010cb6 thru 0x01e2a4 (0xd5ef bytes) 000236 1744170634.865672 mapwrite 0x009685 thru 0x013ddc (0xa758 bytes) 000237 1744170634.876874 punch from 0x01a65f to 0x029096 (0xea38 bytes) 000238 1744170634.878741 write 0x029216 thru 0x0355f8 (0xc3e3 bytes) 000239 1744170634.884266 MIRROR_RESYNC 2 mirrors 000240 1744170634.990613 fallocate 0x02bf14 thru 0x030411 (0x44fe bytes) 000241 1744170635.000263 write 0x0383e0 thru 0x03c6bd (0x42de bytes) 000243 1744170635.001397 trunc from 0x03c6be to 0x0185fd (0x240c0 bytes) 000244 1744170635.012081 mapread 0x000ddf thru 0x00e1ca (0xd3ec bytes) 000246 1744170635.022526 read 0x0085ec thru 0x00f2ea (0x6cff bytes) 000247 1744170635.023029 punch from 0x00a6f2 to 0x0185fd (0xdf0c bytes) 000248 1744170635.025711 punch from 0x017284 to 0x0185fd (0x137a bytes) 000249 1744170635.027483 fallocate 0x02a929 thru 0x02cba5 (0x227d bytes) 000250 1744170635.030490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000253 1744170635.067497 punch from 0x029a94 to 0x02cba5 (0x3112 bytes) 000254 1744170635.069303 write 0x00d1d6 thru 0x00e407 (0x1232 bytes) 000256 1744170635.073763 trunc from 0x02cba6 to 0x01dd9f (0xee06 bytes) 000257 1744170635.078024 punch from 0x000919 to 0x002d2b (0x2413 bytes) 000258 1744170635.079841 punch from 0x01ab4b to 0x01dd9f (0x3255 bytes) 000259 1744170635.081632 fallocate 0x03b24b thru 0x03f091 (0x3e47 bytes) 000260 1744170635.084081 mapread 0x027fe0 thru 0x037b86 (0xfba7 bytes) 000261 1744170635.091659 punch from 0x02776d to 0x027e99 (0x72d bytes) 000262 1744170635.093145 trunc from 0x03f092 to 0x03d0a5 (0x1fec bytes) 000264 1744170635.099000 trunc from 0x03d0a6 to 0x018213 (0x24e92 bytes) 000265 1744170635.104291 read 0x014f0b thru 0x018213 (0x3309 bytes) 000266 1744170635.107142 write 0x013e22 thru 0x015368 (0x1547 bytes) 000267 1744170635.110262 write 0x0059ce thru 0x00eae8 (0x911b bytes) 000268 1744170635.114322 mapread 0x00532d thru 0x00e152 (0x8e26 bytes) 000269 1744170635.114942 punch from 0x008e26 to 0x00e9d5 (0x5bb0 bytes) 000270 1744170635.120054 trunc from 0x018214 to 0x0132a6 (0x4f6d bytes) 000273 1744170635.126100 write 0x014cd6 thru 0x01aa78 (0x5da3 bytes) 000274 1744170635.126926 trunc from 0x01aa79 to 0x0276d3 (0xcc5b bytes) 000275 1744170635.133733 mapwrite 0x00693b thru 0x0148d8 (0xdf9e bytes) 000278 1744170635.147552 mapread 0x0057cd thru 0x00ecbc (0x94f0 bytes) 000279 1744170635.148133 trunc from 0x0276d4 to 0x00014c (0x27587 bytes) 000282 1744170635.154847 write 0x0235b5 thru 0x027320 (0x3d6c bytes) 000283 1744170635.155801 mapwrite 0x01adc5 thru 0x022505 (0x7741 bytes) 000284 1744170635.166869 mapread 0x01345d thru 0x017c4a (0x47ee bytes) 000287 1744170635.168426 punch from 0x011685 to 0x01f008 (0xd984 bytes) 000288 1744170635.170861 write 0x0019df thru 0x00cb8f (0xb1b1 bytes) 000289 1744170635.174159 mapread 0x0099b3 thru 0x0160b9 (0xc707 bytes) 000290 1744170635.177696 fallocate 0x039cc5 thru 0x03ffff (0x633b bytes) 000291 1744170635.179973 punch from 0x00e0b7 to 0x00e624 (0x56e bytes) 000292 1744170635.182206 fallocate 0x02bb81 thru 0x039cb2 (0xe132 bytes) 000293 1744170635.183966 trunc from 0x040000 to 0x000dbe (0x3f241 bytes) 000294 1744170635.188155 fallocate 0x03a244 thru 0x03aa22 (0x7df bytes) 000295 1744170635.190135 read 0x000fd2 thru 0x004b31 (0x3b60 bytes) 000296 1744170635.193977 read 0x001bb0 thru 0x006b69 (0x4fba bytes) 000297 1744170635.194431 write 0x02b989 thru 0x02d1e2 (0x185a bytes) 000298 1744170635.195004 mapwrite 0x0161e3 thru 0x0214c8 (0xb2e6 bytes) 000299 1744170635.203232 punch from 0x0239ea to 0x023e1f (0x436 bytes) 000301 1744170635.205763 trunc from 0x03aa23 to 0x0227d3 (0x1824f bytes) 000302 1744170635.210927 MIRROR_EXTEND to 2 mirrors 000303 1744170635.330090 punch from 0x009f6f to 0x019476 (0xf508 bytes) 000304 1744170635.343374 fallocate 0x024dcf thru 0x0340f8 (0xf32a bytes) 000305 1744170635.345413 fallocate 0x030107 thru 0x03ce6d (0xcd67 bytes) 000306 1744170635.347417 mapread 0x023877 thru 0x0272f3 (0x3a7d bytes) 000307 1744170635.353304 mapread 0x01fdca thru 0x02b047 (0xb27e bytes) 000308 1744170635.353829 trunc from 0x03ce6e to 0x012a98 (0x2a3d5 bytes) 000309 1744170635.368679 MIRROR_EXTEND to 3 mirrors 000310 1744170635.474736 write 0x029068 thru 0x02f0c7 (0x6060 bytes) 000311 1744170635.484331 fallocate 0x019197 thru 0x01f616 (0x6480 bytes) 000312 1744170635.485838 write 0x0046e4 thru 0x00c988 (0x82a5 bytes) 000314 1744170635.489312 mapread 0x00d466 thru 0x016da1 (0x993c bytes) 000315 1744170635.494491 fallocate 0x00c4a5 thru 0x0181af (0xbd0b bytes) 000316 1744170635.496584 punch from 0x024601 to 0x026056 (0x1a56 bytes) 000317 1744170635.499656 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000318 1744170635.559114 punch from 0x02ab24 to 0x02f0c7 (0x45a4 bytes) 000319 1744170635.561488 mapread 0x00cac5 thru 0x0156e5 (0x8c21 bytes) 000322 1744170635.570923 read 0x00181b thru 0x008aa4 (0x728a bytes) 000323 1744170635.575454 read 0x017812 thru 0x023e47 (0xc636 bytes) 000324 1744170635.579772 write 0x004217 thru 0x008197 (0x3f81 bytes) 000326 1744170635.583180 write 0x021345 thru 0x02ebd5 (0xd891 bytes) 000328 1744170635.586803 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000329 1744170635.630006 punch from 0x00c84a to 0x018a4c (0xc203 bytes) 000330 1744170635.634277 MIRROR_EXTEND to 2 mirrors 000332 1744170635.767757 trunc from 0x02f0c8 to 0x00e052 (0x21075 bytes) 000333 1744170635.785133 punch from 0x00a9b6 to 0x00e052 (0x369d bytes) 000334 1744170635.786890 fallocate 0x00f4c2 thru 0x01aac3 (0xb602 bytes) 000335 1744170635.788931 mapwrite 0x034ed8 thru 0x039ad8 (0x4c01 bytes) 000336 1744170635.807065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000338 1744170635.837137 mapwrite 0x0145b8 thru 0x01b5eb (0x7034 bytes) 000340 1744170635.846956 write 0x003859 thru 0x00c192 (0x893a bytes) 000341 1744170635.851387 trunc from 0x039ad9 to 0x0331c7 (0x6911 bytes) 000342 1744170635.856281 trunc from 0x0331c8 to 0x03a79b (0x75d4 bytes) 000343 1744170635.862577 MIRROR_EXTEND to 2 mirrors 000344 1744170635.997937 trunc from 0x03a79c to 0x035d51 (0x4a4a bytes) 000345 1744170636.015481 write 0x00dfe1 thru 0x00e9df (0x9ff bytes) 000346 1744170636.018985 mapwrite 0x02828b thru 0x028fe4 (0xd5a bytes) 000347 1744170636.027634 mapread 0x016591 thru 0x016593 (0x3 bytes) 000349 1744170636.031032 fallocate 0x002f50 thru 0x00ad0e (0x7dbf bytes) 000350 1744170636.032981 trunc from 0x035d52 to 0x00cd19 (0x29038 bytes) 000353 1744170636.039499 punch from 0x00c153 to 0x00cd19 (0xbc7 bytes) 000354 1744170636.041144 punch from 0x003993 to 0x00618a (0x27f8 bytes) 000356 1744170636.042880 mapwrite 0x01c3bd thru 0x02a39c (0xdfe0 bytes) 000357 1744170636.060721 write 0x00ce0b thru 0x0117e4 (0x49da bytes) 000358 1744170636.063500 punch from 0x01ccc8 to 0x01df7e (0x12b7 bytes) 000360 1744170636.065592 mapwrite 0x0283eb thru 0x02ea68 (0x667e bytes) 000361 1744170636.078424 punch from 0x0069c1 to 0x00d678 (0x6cb8 bytes) 000363 1744170636.080943 punch from 0x014bf0 to 0x01d046 (0x8457 bytes) 000364 1744170636.082712 fallocate 0x014c6e thru 0x01a9ce (0x5d61 bytes) 000368 1744170636.085159 punch from 0x0198ea to 0x028421 (0xeb38 bytes) 000369 1744170636.086724 punch from 0x00e770 to 0x019366 (0xabf7 bytes) 000370 1744170636.088308 write 0x012223 thru 0x01f4b7 (0xd295 bytes) 000371 1744170636.093673 punch from 0x0206b9 to 0x02a141 (0x9a89 bytes) 000373 1744170636.095643 mapread 0x00a820 thru 0x011457 (0x6c38 bytes) 000374 1744170636.099251 write 0x0190d2 thru 0x022baf (0x9ade bytes) 000375 1744170636.102415 read 0x0009fa thru 0x001a81 (0x1088 bytes) 000376 1744170636.107598 mapread 0x027cf3 thru 0x02dc75 (0x5f83 bytes) 000378 1744170636.108379 fallocate 0x027607 thru 0x027837 (0x231 bytes) 000379 1744170636.110216 punch from 0x01a610 to 0x0220b9 (0x7aaa bytes) 000380 1744170636.114322 mapwrite 0x02e212 thru 0x033dae (0x5b9d bytes) 000381 1744170636.125464 punch from 0x00a4f4 to 0x018b4a (0xe657 bytes) 000383 1744170636.127184 punch from 0x0037eb to 0x00acb7 (0x74cd bytes) 000384 1744170636.128445 mapwrite 0x01e69a thru 0x01e80f (0x176 bytes) 000385 1744170636.133567 punch from 0x00e8d5 to 0x016b20 (0x824c bytes) 000386 1744170636.134787 write 0x03fddc thru 0x03ffff (0x224 bytes) 000387 1744170636.135334 mapwrite 0x0211ae thru 0x02881c (0x766f bytes) 000388 1744170636.144393 write 0x00e634 thru 0x0146c0 (0x608d bytes) 000390 1744170636.147619 mapread 0x00e67e thru 0x013136 (0x4ab9 bytes) 000391 1744170636.148205 mapread 0x01993c thru 0x026907 (0xcfcc bytes) 000392 1744170636.148687 mapwrite 0x037dbe thru 0x03c55c (0x479f bytes) 000393 1744170636.156779 trunc from 0x040000 to 0x0115a5 (0x2ea5a bytes) 000394 1744170636.162647 read 0x0023ad thru 0x007bfb (0x584f bytes) 000395 1744170636.164945 write 0x001633 thru 0x007b24 (0x64f2 bytes) 000396 1744170636.165671 fallocate 0x02760a thru 0x031069 (0x9a60 bytes) 000398 1744170636.167581 write 0x002769 thru 0x00a1c2 (0x7a5a bytes) 000400 1744170636.168549 write 0x01a1fc thru 0x020622 (0x6427 bytes) 000403 1744170636.172248 mapwrite 0x007a73 thru 0x012fe6 (0xb574 bytes) 000405 1744170636.179362 trunc from 0x03106a to 0x02d579 (0x3af0 bytes) 000407 1744170636.183193 fallocate 0x035a38 thru 0x0360f2 (0x6bb bytes) 000409 1744170636.184980 read 0x00bfc0 thru 0x00f4c3 (0x3504 bytes) 000410 1744170636.186208 MIRROR_EXTEND to 3 mirrors 000411 1744170636.295025 punch from 0x0164b8 to 0x025d52 (0xf89b bytes) 000413 1744170636.303643 write 0x02dea7 thru 0x0332ab (0x5405 bytes) 000414 1744170636.306506 mapread 0x02e5f9 thru 0x0332ab (0x4cb3 bytes) 000415 1744170636.307039 read 0x026e83 thru 0x02e023 (0x71a1 bytes) 000416 1744170636.309600 read 0x023806 thru 0x026fd0 (0x37cb bytes) 000417 1744170636.312680 read 0x0000ed thru 0x007017 (0x6f2b bytes) 000419 1744170636.313419 fallocate 0x01f974 thru 0x02ef94 (0xf621 bytes) 000420 1744170636.315241 trunc from 0x0332ac to 0x026e0e (0xc49d bytes) 000421 1744170636.323701 mapwrite 0x02dc45 thru 0x03a691 (0xca4d bytes) 000422 1744170636.338317 mapread 0x0095d5 thru 0x013ead (0xa8d9 bytes) 000423 1744170636.338825 fallocate 0x01d508 thru 0x0200f3 (0x2bec bytes) 000424 1744170636.340281 mapwrite 0x025bf3 thru 0x0283b7 (0x27c5 bytes) 000426 1744170636.345120 mapread 0x0330f1 thru 0x034ff5 (0x1f05 bytes) 000427 1744170636.345602 fallocate 0x029280 thru 0x030367 (0x70e8 bytes) 000428 1744170636.347950 write 0x001076 thru 0x001390 (0x31b bytes) 000429 1744170636.348386 fallocate 0x02d4d6 thru 0x031c9e (0x47c9 bytes) 000430 1744170636.349616 trunc from 0x03a692 to 0x0039b4 (0x36cdd bytes) 000431 1744170636.354656 MIRROR_EXTEND to 4 mirrors 000433 1744170636.436661 read 0x002b7b thru 0x0039b4 (0xe3a bytes) 000434 1744170636.439159 trunc from 0x0039b5 to 0x0261d2 (0x2281e bytes) 000435 1744170636.453587 mapwrite 0x004eaf thru 0x00903c (0x418e bytes) 000437 1744170636.463738 read 0x0088ae thru 0x0114a6 (0x8bf9 bytes) 000438 1744170636.466692 trunc from 0x0261d3 to 0x035485 (0xf2b3 bytes) 000439 1744170636.471306 mapwrite 0x02430a thru 0x0336b9 (0xf3b0 bytes) 000440 1744170636.480240 mapwrite 0x006c66 thru 0x0164b4 (0xf84f bytes) 000441 1744170636.486397 punch from 0x033673 to 0x035485 (0x1e13 bytes) 000442 1744170636.487836 trunc from 0x035486 to 0x0313e8 (0x409d bytes) 000443 1744170636.492724 MIRROR_RESYNC 4 mirrors 000444 1744170636.626947 read 0x021ddf thru 0x02f7a6 (0xd9c8 bytes) 000446 1744170636.630370 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000448 1744170636.694190 MIRROR_RESYNC 3 mirrors 000449 1744170636.738780 trunc from 0x0313e9 to 0x00a6e9 (0x26cff bytes) 000450 1744170636.755618 mapwrite 0x01654f thru 0x01aba8 (0x465a bytes) 000452 1744170636.774416 read 0x00e17c thru 0x0138a3 (0x5728 bytes) 000454 1744170636.776945 read 0x018261 thru 0x01aba8 (0x2948 bytes) 000455 1744170636.777322 write 0x03ffee thru 0x03ffff (0x12 bytes) 000456 1744170636.777901 read 0x00a5a0 thru 0x01659b (0xbffc bytes) 000459 1744170636.781760 read 0x03e35e thru 0x03ffff (0x1ca2 bytes) 000460 1744170636.782173 write 0x03a568 thru 0x03ffff (0x5a98 bytes) 000461 1744170636.782763 trunc from 0x040000 to 0x0320e5 (0xdf1a bytes) 000462 1744170636.790893 write 0x025f15 thru 0x02700f (0x10fb bytes) 000463 1744170636.791551 mapwrite 0x035ccd thru 0x03ffff (0xa333 bytes) 000464 1744170636.807348 trunc from 0x040000 to 0x01ce25 (0x231da bytes) 000465 1744170636.811399 fallocate 0x0138e5 thru 0x02079f (0xcebb bytes) 000466 1744170636.813190 trunc from 0x0207a0 to 0x0274e7 (0x6d48 bytes) 000467 1744170636.817551 fallocate 0x036a89 thru 0x03c626 (0x5b9e bytes) 000469 1744170636.819830 mapread 0x0076d9 thru 0x00c401 (0x4d29 bytes) 000470 1744170636.820581 punch from 0x016174 to 0x01a214 (0x40a1 bytes) 000471 1744170636.823063 mapwrite 0x03dca7 thru 0x03ffff (0x2359 bytes) 000472 1744170636.836197 read 0x01f5f8 thru 0x02765b (0x8064 bytes) 000473 1744170636.839751 fallocate 0x00d759 thru 0x01b543 (0xddeb bytes) 000475 1744170636.842298 trunc from 0x040000 to 0x00b8ee (0x34711 bytes) 000476 1744170636.848281 mapwrite 0x00a356 thru 0x015a03 (0xb6ae bytes) 000477 1744170636.874571 write 0x01f85b thru 0x01f9b4 (0x15a bytes) 000478 1744170636.875208 trunc from 0x01f9b5 to 0x0323f1 (0x12a3d bytes) 000480 1744170636.879096 fallocate 0x02e21d thru 0x033e95 (0x5c79 bytes) 000482 1744170636.881291 fallocate 0x000a5b thru 0x009755 (0x8cfb bytes) 000484 1744170636.884912 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000485 1744170636.933956 mapwrite 0x00c310 thru 0x01c1ad (0xfe9e bytes) 000486 1744170636.951297 write 0x016022 thru 0x017087 (0x1066 bytes) 000487 1744170636.953207 write 0x019215 thru 0x022ce8 (0x9ad4 bytes) 000489 1744170636.955650 read 0x006ada thru 0x00bec7 (0x53ee bytes) 000490 1744170636.958071 mapwrite 0x014c79 thru 0x0235c7 (0xe94f bytes) 000491 1744170636.968273 punch from 0x02389f to 0x02a88e (0x6ff0 bytes) 000492 1744170636.969789 mapread 0x00691a thru 0x012381 (0xba68 bytes) 000493 1744170636.971528 write 0x02f023 thru 0x034c40 (0x5c1e bytes) 000494 1744170636.975695 mapwrite 0x03baf1 thru 0x03ffff (0x450f bytes) 000498 1744170636.990327 mapwrite 0x01bd21 thru 0x02895a (0xcc3a bytes) 000500 1744170637.000480 write 0x03b74b thru 0x03ffff (0x48b5 bytes) 000501 1744170637.002642 MIRROR_EXTEND to 3 mirrors 000502 1744170637.119078 punch from 0x03a581 to 0x03d862 (0x32e2 bytes) 000503 1744170637.127077 read 0x03923d thru 0x03d9e4 (0x47a8 bytes) 000504 1744170637.129292 mapwrite 0x03e5af thru 0x03ffff (0x1a51 bytes) 000505 1744170637.135028 punch from 0x0036a4 to 0x01361b (0xff78 bytes) 000506 1744170637.136424 mapread 0x011bd1 thru 0x01a908 (0x8d38 bytes) 000507 1744170637.141349 mapread 0x03a71e thru 0x03e551 (0x3e34 bytes) 000508 1744170637.141954 read 0x010447 thru 0x01dc53 (0xd80d bytes) 000509 1744170637.142383 write 0x012416 thru 0x021f0c (0xfaf7 bytes) 000510 1744170637.144401 fallocate 0x024c60 thru 0x02b36f (0x6710 bytes) 000511 1744170637.145847 write 0x02e0c9 thru 0x03202a (0x3f62 bytes) 000512 1744170637.148454 read 0x028bb4 thru 0x034fa1 (0xc3ee bytes) 000513 1744170637.148839 fallocate 0x021f27 thru 0x0221cf (0x2a9 bytes) 000514 1744170637.151215 trunc from 0x040000 to 0x028b8e (0x17471 bytes) 000515 1744170637.156587 trunc from 0x028b8f to 0x0103dd (0x187b1 bytes) 000516 1744170637.160416 punch from 0x00f08a to 0x0103dd (0x1354 bytes) 000517 1744170637.161791 write 0x025cda thru 0x032961 (0xcc88 bytes) 000518 1744170637.162939 mapread 0x02debf thru 0x0306d0 (0x2812 bytes) 000519 1744170637.163503 fallocate 0x011dcb thru 0x0187e4 (0x6a1a bytes) 000520 1744170637.165301 mapread 0x028daa thru 0x032051 (0x92a8 bytes) 000521 1744170637.167454 MIRROR_EXTEND to 4 mirrors 000522 1744170637.272249 fallocate 0x01dc98 thru 0x0211b1 (0x351a bytes) 000523 1744170637.282695 mapwrite 0x02e8bb thru 0x030ca6 (0x23ec bytes) 000524 1744170637.292022 MIRROR_EXTEND to 5 mirrors 000527 1744170637.390319 trunc from 0x032962 to 0x011c47 (0x20d1a bytes) 000528 1744170637.405793 MIRROR_EXTEND to 6 mirrors 000529 1744170637.505801 write 0x0294a0 thru 0x03292f (0x9490 bytes) 000530 1744170637.516504 MIRROR_EXTEND to 7 mirrors 000531 1744170637.627203 write 0x0300bd thru 0x03795d (0x78a1 bytes) 000532 1744170637.637088 read 0x021fc1 thru 0x02c697 (0xa6d7 bytes) 000533 1744170637.640841 MIRROR_EXTEND to 8 mirrors 000534 1744170637.757000 trunc from 0x03795e to 0x01b312 (0x1c64b bytes) 000536 1744170637.773029 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000537 1744170637.824978 fallocate 0x024e54 thru 0x033a9b (0xec48 bytes) 000538 1744170637.827200 fallocate 0x03285d thru 0x03582c (0x2fd0 bytes) 000542 1744170637.830051 mapwrite 0x0283ae thru 0x02ef2f (0x6b82 bytes) 000543 1744170637.844762 fallocate 0x0064e3 thru 0x0066a1 (0x1bf bytes) 000545 1744170637.846748 write 0x000a3f thru 0x00f546 (0xeb08 bytes) 000546 1744170637.852923 write 0x03596b thru 0x03ffff (0xa695 bytes) 000548 1744170637.853863 read 0x0366d9 thru 0x03ffff (0x9927 bytes) 000549 1744170637.854240 fallocate 0x029057 thru 0x029634 (0x5de bytes) 000550 1744170637.856217 write 0x0380f7 thru 0x03ee0f (0x6d19 bytes) 000551 1744170637.856723 write 0x005d2a thru 0x005dcb (0xa2 bytes) 000553 1744170637.857337 mapwrite 0x03001c thru 0x03667b (0x6660 bytes) 000554 1744170637.867451 fallocate 0x01fd88 thru 0x02c3b7 (0xc630 bytes) 000556 1744170637.870474 punch from 0x019a35 to 0x025186 (0xb752 bytes) 000558 1744170637.872525 read 0x00cf29 thru 0x00ddbc (0xe94 bytes) 000559 1744170637.872934 mapread 0x0215ad thru 0x030568 (0xefbc bytes) 000560 1744170637.877215 read 0x02fdad thru 0x03ecfb (0xef4f bytes) 000562 1744170637.877872 write 0x03b30f thru 0x03ffff (0x4cf1 bytes) 000564 1744170637.878827 write 0x007173 thru 0x00c1ea (0x5078 bytes) 000565 1744170637.880836 MIRROR_RESYNC 7 mirrors 000567 1744170638.168459 mapread 0x0202cf thru 0x02e5a5 (0xe2d7 bytes) 000569 1744170638.178091 mapwrite 0x025a60 thru 0x02d0e3 (0x7684 bytes) 000570 1744170638.208518 write 0x01bba8 thru 0x0287da (0xcc33 bytes) 000571 1744170638.212067 write 0x03a3a0 thru 0x03ffff (0x5c60 bytes) 000572 1744170638.214608 read 0x019eef thru 0x028ed0 (0xefe2 bytes) 000573 1744170638.215089 fallocate 0x00cea3 thru 0x0131da (0x6338 bytes) 000574 1744170638.217921 mapread 0x00a967 thru 0x00db50 (0x31ea bytes) 000575 1744170638.220421 write 0x03c4cd thru 0x03eedb (0x2a0f bytes) 000576 1744170638.220947 mapwrite 0x00908d thru 0x014371 (0xb2e5 bytes) 000579 1744170638.229905 MIRROR_EXTEND to 8 mirrors 000580 1744170638.342929 write 0x00c172 thru 0x01a959 (0xe7e8 bytes) 000581 1744170638.355788 read 0x0348f0 thru 0x03ffff (0xb710 bytes) 000582 1744170638.358510 write 0x01c6dc thru 0x026d47 (0xa66c bytes) 000583 1744170638.362943 mapread 0x03bcb8 thru 0x03ffff (0x4348 bytes) 000584 1744170638.363583 punch from 0x017fef to 0x027e10 (0xfe22 bytes) 000585 1744170638.369301 mapwrite 0x01f738 thru 0x024aa8 (0x5371 bytes) 000586 1744170638.379289 mapwrite 0x01d47d thru 0x029d68 (0xc8ec bytes) 000587 1744170638.391596 fallocate 0x018791 thru 0x01e941 (0x61b1 bytes) 000588 1744170638.394864 write 0x027190 thru 0x028eb2 (0x1d23 bytes) 000589 1744170638.395355 fallocate 0x01a1f7 thru 0x01a83b (0x645 bytes) 000590 1744170638.397536 read 0x01d519 thru 0x0225a6 (0x508e bytes) 000591 1744170638.397941 mapwrite 0x025f9c thru 0x02bcf2 (0x5d57 bytes) 000592 1744170638.405182 read 0x0057ba thru 0x0146ad (0xeef4 bytes) 000593 1744170638.409153 mapwrite 0x03cdb0 thru 0x03e56b (0x17bc bytes) 000594 1744170638.413528 punch from 0x03a4e5 to 0x03ffff (0x5b1b bytes) 000596 1744170638.416801 MIRROR_RESYNC 8 mirrors 000597 1744170638.655052 punch from 0x019f1a to 0x0204f9 (0x65e0 bytes) 000598 1744170638.665799 write 0x003446 thru 0x009107 (0x5cc2 bytes) 000599 1744170638.669733 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000600 1744170638.725689 punch from 0x02d342 to 0x037d2a (0xa9e9 bytes) 000601 1744170638.727405 punch from 0x01146b to 0x01e223 (0xcdb9 bytes) 000602 1744170638.729970 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000603 1744170638.779402 MIRROR_RESYNC 6 mirrors 000604 1744170639.069692 mapread 0x03fffd thru 0x03ffff (0x3 bytes) 000606 1744170639.071439 read 0x02e0a3 thru 0x03cff1 (0xef4f bytes) 000608 1744170639.074442 punch from 0x03a7dd to 0x03befa (0x171e bytes) 000609 1744170639.084059 read 0x00f244 thru 0x013c44 (0x4a01 bytes) 000610 1744170639.088143 mapwrite 0x006be1 thru 0x012e3d (0xc25d bytes) 000611 1744170639.097662 trunc from 0x040000 to 0x00ab0a (0x354f5 bytes) 000612 1744170639.107572 mapread 0x007c31 thru 0x00ab0a (0x2eda bytes) 000613 1744170639.108134 trunc from 0x00ab0b to 0x028671 (0x1db67 bytes) 000614 1744170639.113414 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000615 1744170639.166281 mapwrite 0x00f8a7 thru 0x0165f9 (0x6d53 bytes) 000616 1744170639.178619 MIRROR_RESYNC 5 mirrors 000617 1744170639.381955 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000618 1744170639.457893 read 0x01a75a thru 0x02677b (0xc022 bytes) 000619 1744170639.460474 punch from 0x0064fc to 0x010bec (0xa6f1 bytes) 000620 1744170639.469516 punch from 0x01d2b6 to 0x01eb2d (0x1878 bytes) 000623 1744170639.471518 read 0x00fb98 thru 0x01fae9 (0xff52 bytes) 000624 1744170639.475132 punch from 0x005bcd to 0x00d3bd (0x77f1 bytes) 000625 1744170639.476877 punch from 0x00f154 to 0x01cfde (0xde8b bytes) 000626 1744170639.478196 mapread 0x00df8c thru 0x015c80 (0x7cf5 bytes) 000629 1744170639.489637 punch from 0x022dda to 0x028671 (0x5898 bytes) 000631 1744170639.492205 trunc from 0x028672 to 0x029584 (0xf13 bytes) 000632 1744170639.505394 write 0x006996 thru 0x014030 (0xd69b bytes) 000634 1744170639.508715 fallocate 0x0093d0 thru 0x015ac9 (0xc6fa bytes) 000635 1744170639.510525 mapread 0x00d307 thru 0x0197ca (0xc4c4 bytes) 000636 1744170639.517031 mapread 0x026373 thru 0x029584 (0x3212 bytes) 000637 1744170639.522875 read 0x010031 thru 0x01dd2d (0xdcfd bytes) 000639 1744170639.525874 fallocate 0x025305 thru 0x02592f (0x62b bytes) 000640 1744170639.529860 MIRROR_RESYNC 4 mirrors 000641 1744170639.668680 mapread 0x010c91 thru 0x01e780 (0xdaf0 bytes) 000642 1744170639.678313 trunc from 0x029585 to 0x006655 (0x22f2f bytes) 000643 1744170639.696054 write 0x0339e0 thru 0x03dd70 (0xa391 bytes) 000644 1744170639.697150 trunc from 0x03dd71 to 0x02e993 (0xf3dd bytes) 000645 1744170639.704059 punch from 0x00737d to 0x009433 (0x20b7 bytes) 000646 1744170639.707231 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000647 1744170639.766609 read 0x01428e thru 0x0159c2 (0x1735 bytes) 000648 1744170639.768717 trunc from 0x02e994 to 0x030735 (0x1da2 bytes) 000649 1744170639.775175 punch from 0x00b725 to 0x00c560 (0xe3c bytes) 000650 1744170639.778385 MIRROR_RESYNC 3 mirrors 000651 1744170639.885624 fallocate 0x016c40 thru 0x0228f6 (0xbcb7 bytes) 000652 1744170639.896191 trunc from 0x030736 to 0x02255b (0xe1da bytes) 000653 1744170639.905725 punch from 0x00e821 to 0x014ecd (0x66ad bytes) 000654 1744170639.907290 fallocate 0x01011d thru 0x014f2f (0x4e13 bytes) 000655 1744170639.908885 punch from 0x00a04a to 0x018091 (0xe048 bytes) 000656 1744170639.910423 mapwrite 0x01bdf0 thru 0x028e0e (0xd01f bytes) 000657 1744170639.931687 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000658 1744170639.987423 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000659 1744170640.024385 punch from 0x02207d to 0x02584e (0x37d2 bytes) 000660 1744170640.026518 trunc from 0x028e0f to 0x03cd2c (0x13f1e bytes) 000661 1744170640.033352 punch from 0x02442c to 0x02d47f (0x9054 bytes) 000664 1744170640.035774 read 0x0000a2 thru 0x0026ac (0x260b bytes) 000665 1744170640.037976 punch from 0x02e5bf to 0x03703d (0x8a7f bytes) 000666 1744170640.039999 punch from 0x036f0a to 0x039dc8 (0x2ebf bytes) 000667 1744170640.041350 mapread 0x004e64 thru 0x0095d6 (0x4773 bytes) 000668 1744170640.045439 fallocate 0x02e3a1 thru 0x03d419 (0xf079 bytes) 000669 1744170640.047793 mapread 0x0292e2 thru 0x02c759 (0x3478 bytes) 000671 1744170640.051132 write 0x029e54 thru 0x02be8d (0x203a bytes) 000672 1744170640.051633 read 0x0210c0 thru 0x02b5c0 (0xa501 bytes) 000673 1744170640.054725 trunc from 0x03d41a to 0x02bc45 (0x117d4 bytes) 000674 1744170640.058377 fallocate 0x033440 thru 0x03fac3 (0xc684 bytes) 000675 1744170640.059874 write 0x013920 thru 0x01c279 (0x895a bytes) 000676 1744170640.060761 trunc from 0x02bc46 to 0x03f826 (0x13be1 bytes) 000677 1744170640.064318 read 0x021242 thru 0x02715a (0x5f19 bytes) 000678 1744170640.064750 trunc from 0x03f827 to 0x023a6e (0x1bdb8 bytes) 000679 1744170640.067731 trunc from 0x023a6f to 0x012731 (0x1133d bytes) 000680 1744170640.070763 punch from 0x01057f to 0x012731 (0x21b3 bytes) 000681 1744170640.071982 fallocate 0x03ded0 thru 0x03ffff (0x2130 bytes) 000682 1744170640.073132 trunc from 0x012732 to 0x018d62 (0x6631 bytes) 000683 1744170640.076276 read 0x004275 thru 0x0137aa (0xf536 bytes) 000684 1744170640.078521 punch from 0x0154b8 to 0x018d62 (0x38ab bytes) 000685 1744170640.079798 mapwrite 0x01b02b thru 0x0287ef (0xd7c5 bytes) 000686 1744170640.092475 fallocate 0x0067eb thru 0x00c9b9 (0x61cf bytes) 000687 1744170640.094095 write 0x012b87 thru 0x0188b0 (0x5d2a bytes) 000688 1744170640.095988 punch from 0x0166d5 to 0x0183ba (0x1ce6 bytes) 000690 1744170640.099430 fallocate 0x02f3dd thru 0x03d989 (0xe5ad bytes) 000691 1744170640.100882 fallocate 0x02db47 thru 0x034f5f (0x7419 bytes) 000692 1744170640.102464 fallocate 0x030d8d thru 0x032365 (0x15d9 bytes) 000693 1744170640.104086 trunc from 0x034f60 to 0x0153cf (0x1fb90 bytes) 000694 1744170640.108866 read 0x0125a4 thru 0x0153cf (0x2e2c bytes) 000696 1744170640.109929 mapread 0x0014d5 thru 0x0107e1 (0xf30d bytes) 000697 1744170640.110589 trunc from 0x0153d0 to 0x00a308 (0xb0c7 bytes) 000698 1744170640.115408 trunc from 0x00a309 to 0x0284c9 (0x1e1c1 bytes) 000699 1744170640.119204 punch from 0x019ca2 to 0x020321 (0x6680 bytes) 000700 1744170640.120700 mapwrite 0x01368a thru 0x021636 (0xdfad bytes) 000701 1744170640.131355 read 0x01ea88 thru 0x022746 (0x3cbf bytes) 000702 1744170640.131746 trunc from 0x0284ca to 0x004db7 (0x23712 bytes) 000703 1744170640.136305 punch from 0x003d25 to 0x004900 (0xbdc bytes) 000704 1744170640.138120 mapwrite 0x021d68 thru 0x028744 (0x69dd bytes) 000705 1744170640.152085 mapwrite 0x0003d4 thru 0x002329 (0x1f56 bytes) 000706 1744170640.155839 punch from 0x016967 to 0x022e99 (0xc533 bytes) 000708 1744170640.157801 punch from 0x01584e to 0x0226bd (0xce70 bytes) 000709 1744170640.159199 read 0x0279a6 thru 0x027e1c (0x477 bytes) 000710 1744170640.159565 punch from 0x001d2d to 0x00cdfa (0xb0ce bytes) 000711 1744170640.161028 mapwrite 0x00847d thru 0x00cd1e (0x48a2 bytes) 000712 1744170640.167742 trunc from 0x028745 to 0x02588b (0x2eb9 bytes) 000713 1744170640.171670 punch from 0x00ae54 to 0x018791 (0xd93e bytes) 000716 1744170640.173916 fallocate 0x029345 thru 0x02c2ab (0x2f67 bytes) 000717 1744170640.175423 punch from 0x02b67f to 0x02c2ab (0xc2d bytes) 000718 1744170640.176722 fallocate 0x027299 thru 0x032ff8 (0xbd60 bytes) 000719 1744170640.178231 read 0x01bfea thru 0x028b4f (0xcb66 bytes) 000720 1744170640.181053 read 0x00e8e3 thru 0x017501 (0x8c1f bytes) 000721 1744170640.181479 trunc from 0x032ff9 to 0x0335df (0x5e7 bytes) 000722 1744170640.186517 MIRROR_EXTEND to 2 mirrors 000723 1744170640.314486 mapread 0x00e6af thru 0x015527 (0x6e79 bytes) 000724 1744170640.321624 mapread 0x0293cc thru 0x0316ba (0x82ef bytes) 000725 1744170640.329841 MIRROR_RESYNC 2 mirrors 000726 1744170640.371897 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000728 1744170640.404581 punch from 0x02365c to 0x03022c (0xcbd1 bytes) 000730 1744170640.408860 mapread 0x007ae4 thru 0x011675 (0x9b92 bytes) 000732 1744170640.415270 trunc from 0x0335e0 to 0x0010a5 (0x3253a bytes) 000733 1744170640.423588 punch from 0x0005b8 to 0x0010a5 (0xaee bytes) 000734 1744170640.425219 trunc from 0x0010a6 to 0x02e59d (0x2d4f8 bytes) 000735 1744170640.428705 mapread 0x01641b thru 0x02583b (0xf421 bytes) 000736 1744170640.434977 MIRROR_EXTEND to 2 mirrors 000737 1744170640.543328 trunc from 0x02e59e to 0x012a19 (0x1bb84 bytes) 000739 1744170640.558990 mapread 0x004fe6 thru 0x00b326 (0x6341 bytes) 000740 1744170640.565378 MIRROR_RESYNC 2 mirrors 000742 1744170640.648773 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000743 1744170640.687544 write 0x03a23b thru 0x03ffff (0x5dc5 bytes) 000745 1744170640.691779 fallocate 0x000c4e thru 0x002d75 (0x2128 bytes) 000746 1744170640.693607 trunc from 0x040000 to 0x02ed07 (0x112f8 bytes) 000747 1744170640.702763 mapwrite 0x000f96 thru 0x00c091 (0xb0fc bytes) 000748 1744170640.717607 mapread 0x00a4a0 thru 0x0162bd (0xbe1e bytes) 000749 1744170640.720661 write 0x022828 thru 0x02a99a (0x8173 bytes) 000750 1744170640.723423 read 0x02c46a thru 0x02ed07 (0x289e bytes) 000754 1744170640.727256 mapwrite 0x006eb1 thru 0x0111ae (0xa2fe bytes) 000755 1744170640.735443 mapwrite 0x004014 thru 0x00e1f9 (0xa1e6 bytes) 000759 1744170640.743112 mapread 0x011681 thru 0x011f5e (0x8de bytes) 000761 1744170640.743943 punch from 0x026991 to 0x02ed07 (0x8377 bytes) 000762 1744170640.747028 mapwrite 0x002a68 thru 0x004249 (0x17e2 bytes) 000763 1744170640.751161 write 0x017528 thru 0x02288b (0xb364 bytes) 000764 1744170640.751930 fallocate 0x016a5b thru 0x01ba47 (0x4fed bytes) 000767 1744170640.755035 mapwrite 0x002767 thru 0x004574 (0x1e0e bytes) 000768 1744170640.760337 mapread 0x02a8ec thru 0x02ccb2 (0x23c7 bytes) 000770 1744170640.763385 write 0x0295ef thru 0x02cbfe (0x3610 bytes) 000772 1744170640.764262 write 0x01d2dc thru 0x01ef77 (0x1c9c bytes) 000773 1744170640.764695 mapread 0x005ced thru 0x006cd2 (0xfe6 bytes) 000777 1744170640.766146 read 0x0045b7 thru 0x00aa5a (0x64a4 bytes) 000778 1744170640.766545 mapread 0x01b01c thru 0x02293b (0x7920 bytes) 000779 1744170640.766960 trunc from 0x02ed08 to 0x00c8fa (0x2240d bytes) 000780 1744170640.771766 mapread 0x0016a7 thru 0x00697f (0x52d9 bytes) 000782 1744170640.772624 trunc from 0x00c8fb to 0x0212b2 (0x149b8 bytes) 000784 1744170640.776209 mapread 0x01085d thru 0x01fb72 (0xf316 bytes) 000785 1744170640.779223 write 0x014496 thru 0x015a1e (0x1589 bytes) 000786 1744170640.780020 fallocate 0x0185c0 thru 0x01cc13 (0x4654 bytes) 000787 1744170640.781619 read 0x014552 thru 0x0212b2 (0xcd61 bytes) 000788 1744170640.781903 mapread 0x012265 thru 0x012d83 (0xb1f bytes) 000790 1744170640.782462 trunc from 0x0212b3 to 0x03ba57 (0x1a7a5 bytes) 000791 1744170640.785927 punch from 0x012433 to 0x01dd4b (0xb919 bytes) 000792 1744170640.788649 mapread 0x0254a5 thru 0x02e6f5 (0x9251 bytes) 000793 1744170640.791913 mapread 0x03ac49 thru 0x03ba57 (0xe0f bytes) 000795 1744170640.793852 punch from 0x036904 to 0x038724 (0x1e21 bytes) 000796 1744170640.795517 punch from 0x022bb5 to 0x024443 (0x188f bytes) 000798 1744170640.798210 read 0x00043c thru 0x002fcd (0x2b92 bytes) 000799 1744170640.798651 punch from 0x0179e7 to 0x017e54 (0x46e bytes) 000800 1744170640.800184 punch from 0x004934 to 0x00aa65 (0x6132 bytes) 000801 1744170640.801778 trunc from 0x03ba58 to 0x03c803 (0xdac bytes) 000802 1744170640.806562 mapread 0x013723 thru 0x022022 (0xe900 bytes) 000803 1744170640.809039 trunc from 0x03c804 to 0x00a649 (0x321ba bytes) 000806 1744170640.813647 mapwrite 0x0045d7 thru 0x00a197 (0x5bc1 bytes) 000807 1744170640.821989 trunc from 0x00a64a to 0x00a8b2 (0x269 bytes) 000808 1744170640.825933 trunc from 0x00a8b3 to 0x006e1f (0x3a93 bytes) 000809 1744170640.829762 fallocate 0x014f6e thru 0x01b4ce (0x6561 bytes) 000811 1744170640.831799 fallocate 0x024563 thru 0x02ff96 (0xba34 bytes) 000812 1744170640.834916 MIRROR_EXTEND to 2 mirrors 000813 1744170640.937985 write 0x018a54 thru 0x0200c4 (0x7671 bytes) 000814 1744170640.947986 mapwrite 0x039576 thru 0x03ffff (0x6a8a bytes) 000815 1744170640.966561 read 0x012840 thru 0x01df56 (0xb717 bytes) 000816 1744170640.969253 punch from 0x033616 to 0x03d12e (0x9b19 bytes) 000817 1744170640.971229 trunc from 0x040000 to 0x032060 (0xdf9f bytes) 000818 1744170640.975995 mapwrite 0x00a7db thru 0x010f47 (0x676d bytes) 000819 1744170640.986443 mapread 0x03074d thru 0x032060 (0x1914 bytes) 000820 1744170640.992798 MIRROR_RESYNC 2 mirrors 000821 1744170641.125472 fallocate 0x03827d thru 0x03ffff (0x7d83 bytes) 000822 1744170641.133639 punch from 0x0166e5 to 0x01a056 (0x3972 bytes) 000823 1744170641.135276 mapread 0x014cfe thru 0x01b271 (0x6574 bytes) 000824 1744170641.141111 mapread 0x00efd1 thru 0x01a61d (0xb64d bytes) 000825 1744170641.144610 write 0x00a53a thru 0x00da91 (0x3558 bytes) 000826 1744170641.145370 fallocate 0x00d33d thru 0x015454 (0x8118 bytes) 000827 1744170641.147379 punch from 0x03ebbe to 0x03ffff (0x1442 bytes) 000829 1744170641.150119 mapread 0x02eb78 thru 0x0398de (0xad67 bytes) 000832 1744170641.155682 fallocate 0x001e4e thru 0x007252 (0x5405 bytes) 000833 1744170641.158611 MIRROR_RESYNC 2 mirrors 000834 1744170641.282635 read 0x023067 thru 0x02475d (0x16f7 bytes) 000835 1744170641.284112 fallocate 0x02a559 thru 0x03040b (0x5eb3 bytes) 000836 1744170641.292923 mapwrite 0x01fe75 thru 0x02a10a (0xa296 bytes) 000837 1744170641.305117 mapwrite 0x023156 thru 0x029b37 (0x69e2 bytes) 000839 1744170641.311089 mapread 0x021d1f thru 0x02feec (0xe1ce bytes) 000840 1744170641.311603 punch from 0x0213a7 to 0x024669 (0x32c3 bytes) 000841 1744170641.313328 write 0x02b323 thru 0x0337bf (0x849d bytes) 000842 1744170641.316165 fallocate 0x015a8b thru 0x01c533 (0x6aa9 bytes) 000844 1744170641.318075 punch from 0x02b3fa to 0x03a5bf (0xf1c6 bytes) 000845 1744170641.321276 mapread 0x03c012 thru 0x03ffff (0x3fee bytes) 000846 1744170641.323614 punch from 0x02cf1e to 0x02f167 (0x224a bytes) 000848 1744170641.326956 mapread 0x031f23 thru 0x03b975 (0x9a53 bytes) 000850 1744170641.330323 read 0x008a80 thru 0x011723 (0x8ca4 bytes) 000853 1744170641.333526 mapwrite 0x03113c thru 0x03324d (0x2112 bytes) 000854 1744170641.337747 read 0x03d340 thru 0x03ffff (0x2cc0 bytes) 000858 1744170641.339109 fallocate 0x035191 thru 0x03ffff (0xae6f bytes) 000859 1744170641.340966 mapread 0x03030e thru 0x03081d (0x510 bytes) 000860 1744170641.341548 fallocate 0x039a8a thru 0x03efe1 (0x5558 bytes) 000861 1744170641.344087 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000862 1744170641.380604 mapwrite 0x02c39e thru 0x03b83f (0xf4a2 bytes) 000863 1744170641.397241 trunc from 0x040000 to 0x038477 (0x7b88 bytes) 000864 1744170641.409528 punch from 0x029ded to 0x02ab2d (0xd41 bytes) 000865 1744170641.411051 mapread 0x027fc4 thru 0x02d4b8 (0x54f5 bytes) 000866 1744170641.413845 trunc from 0x038478 to 0x0297e1 (0xec96 bytes) 000867 1744170641.418289 punch from 0x018b5a to 0x01f648 (0x6aef bytes) 000869 1744170641.420333 mapread 0x0166f1 thru 0x01d313 (0x6c23 bytes) 000870 1744170641.423382 mapread 0x01852b thru 0x027358 (0xee2e bytes) 000872 1744170641.424427 trunc from 0x0297e2 to 0x022455 (0x738c bytes) 000873 1744170641.428227 punch from 0x00678a to 0x00774f (0xfc6 bytes) 000874 1744170641.430613 punch from 0x00ac10 to 0x00b867 (0xc58 bytes) 000875 1744170641.432106 punch from 0x00ae24 to 0x01a965 (0xfb42 bytes) 000876 1744170641.434231 punch from 0x003c09 to 0x01305d (0xf455 bytes) 000877 1744170641.435775 fallocate 0x0218fe thru 0x021c01 (0x304 bytes) 000879 1744170641.437664 read 0x007917 thru 0x00ecab (0x7395 bytes) 000880 1744170641.440837 punch from 0x01ca4c to 0x022455 (0x5a0a bytes) 000881 1744170641.442649 trunc from 0x022456 to 0x03c395 (0x19f40 bytes) 000882 1744170641.446372 trunc from 0x03c396 to 0x022266 (0x1a12f bytes) 000883 1744170641.449952 mapread 0x01610a thru 0x01af61 (0x4e58 bytes) 000885 1744170641.452924 write 0x00f1b5 thru 0x013d50 (0x4b9c bytes) 000886 1744170641.454669 trunc from 0x022267 to 0x03a8d3 (0x1866d bytes) 000888 1744170641.458477 punch from 0x00599d to 0x00f32c (0x9990 bytes) 000889 1744170641.462050 mapwrite 0x031607 thru 0x031dd6 (0x7d0 bytes) 000890 1744170641.468669 trunc from 0x03a8d4 to 0x02300b (0x178c8 bytes) 000891 1744170641.472233 mapread 0x00db1a thru 0x018ec1 (0xb3a8 bytes) 000892 1744170641.474931 mapwrite 0x02c369 thru 0x036139 (0x9dd1 bytes) 000893 1744170641.488954 punch from 0x008b70 to 0x014289 (0xb71a bytes) 000894 1744170641.490658 punch from 0x019823 to 0x01ec36 (0x5414 bytes) 000895 1744170641.492513 trunc from 0x03613a to 0x006307 (0x2fe32 bytes) 000897 1744170641.496802 read 0x00511b thru 0x006307 (0x11ed bytes) 000899 1744170641.499762 punch from 0x003fc4 to 0x006307 (0x2344 bytes) 000900 1744170641.501229 mapwrite 0x027a1a thru 0x033d43 (0xc32a bytes) 000901 1744170641.517771 read 0x00f4c6 thru 0x01b980 (0xc4bb bytes) 000902 1744170641.518258 mapwrite 0x02ffe0 thru 0x03c512 (0xc533 bytes) 000903 1744170641.537019 trunc from 0x03c513 to 0x02a43d (0x120d5 bytes) 000904 1744170641.541481 read 0x00c3a3 thru 0x01c2a8 (0xff06 bytes) 000905 1744170641.541926 read 0x01f921 thru 0x02a43d (0xab1d bytes) 000906 1744170641.542282 mapwrite 0x01bb91 thru 0x021ae4 (0x5f54 bytes) 000907 1744170641.549282 mapread 0x01139f thru 0x017b70 (0x67d2 bytes) 000908 1744170641.549949 mapwrite 0x02caea thru 0x03416d (0x7684 bytes) 000910 1744170641.564886 MIRROR_EXTEND to 2 mirrors 000912 1744170641.694913 punch from 0x00c2b0 to 0x0112fb (0x504c bytes) 000913 1744170641.702806 trunc from 0x03416e to 0x03e713 (0xa5a6 bytes) 000914 1744170641.711179 punch from 0x00aa79 to 0x0163a1 (0xb929 bytes) 000915 1744170641.713069 punch from 0x02a9e4 to 0x03280f (0x7e2c bytes) 000916 1744170641.714938 write 0x023519 thru 0x030602 (0xd0ea bytes) 000917 1744170641.718227 trunc from 0x03e714 to 0x00dcdc (0x30a37 bytes) 000918 1744170641.722287 write 0x0398f5 thru 0x03ffff (0x670b bytes) 000919 1744170641.723047 fallocate 0x03f82f thru 0x03ffff (0x7d1 bytes) 000920 1744170641.724391 read 0x010669 thru 0x01d5ef (0xcf87 bytes) 000921 1744170641.726705 read 0x03bc50 thru 0x03ffff (0x43b0 bytes) 000922 1744170641.727024 punch from 0x031743 to 0x03d1e6 (0xbaa4 bytes) 000923 1744170641.730117 mapwrite 0x00d7a1 thru 0x011c44 (0x44a4 bytes) 000924 1744170641.739317 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2a:0x0]: Invalid argument (22) 000926 1744170641.774435 fallocate 0x03a928 thru 0x03ffff (0x56d8 bytes) 000927 1744170641.776384 read 0x019e6d thru 0x0291c7 (0xf35b bytes) 000928 1744170641.780277 mapwrite 0x022f11 thru 0x031054 (0xe144 bytes) 000929 1744170641.794150 mapread 0x012e4c thru 0x01c460 (0x9615 bytes) 000930 1744170641.796694 mapwrite 0x0003d3 thru 0x003cf9 (0x3927 bytes) 000931 1744170641.803395 fallocate 0x02a32b thru 0x034a6b (0xa741 bytes) 000933 1744170641.805325 write 0x00f93a thru 0x0114e6 (0x1bad bytes) 000934 1744170641.807743 mapread 0x014de9 thru 0x01fe48 (0xb060 bytes) 000935 1744170641.808358 mapread 0x01068f thru 0x0127ad (0x211f bytes) 000937 1744170641.809296 mapwrite 0x0269dc thru 0x02c688 (0x5cad bytes) 000938 1744170641.816152 fallocate 0x020d1a thru 0x0216c4 (0x9ab bytes) 000939 1744170641.817979 mapread 0x0034d9 thru 0x012af0 (0xf618 bytes) 000940 1744170641.818609 fallocate 0x01aee5 thru 0x01cdf7 (0x1f13 bytes) 000941 1744170641.820421 mapwrite 0x0162c1 thru 0x0254d8 (0xf218 bytes) 000942 1744170641.828541 fallocate 0x0271a2 thru 0x029211 (0x2070 bytes) 000943 1744170641.830501 punch from 0x00f816 to 0x01a50a (0xacf5 bytes) 000944 1744170641.832357 punch from 0x03f6d7 to 0x03ffff (0x929 bytes) 000946 1744170641.834214 write 0x011f45 thru 0x015efc (0x3fb8 bytes) 000949 1744170641.838058 fallocate 0x039047 thru 0x03ffff (0x6fb9 bytes) 000950 1744170641.840060 mapwrite 0x0050a0 thru 0x00d250 (0x81b1 bytes) 000951 1744170641.849765 MIRROR_EXTEND to 2 mirrors 000952 1744170641.994103 fallocate 0x02b827 thru 0x0355e4 (0x9dbe bytes) 000953 1744170642.009331 MIRROR_RESYNC 2 mirrors 000954 1744170642.150168 punch from 0x01e8f3 to 0x02c79f (0xdead bytes) 000955 1744170642.160018 write 0x03d26c thru 0x03ffff (0x2d94 bytes) 000957 1744170642.162208 mapread 0x03a3ed thru 0x03ffff (0x5c13 bytes) 000959 1744170642.167759 fallocate 0x026c53 thru 0x02b6cb (0x4a79 bytes) 000960 1744170642.169900 mapwrite 0x025ccf thru 0x029b77 (0x3ea9 bytes) 000961 1744170642.181060 mapwrite 0x022666 thru 0x028ec3 (0x685e bytes) 000962 1744170642.189427 fallocate 0x0185d4 thru 0x01ad36 (0x2763 bytes) 000963 1744170642.191430 mapwrite 0x008bd0 thru 0x015ba3 (0xcfd4 bytes) 000964 1744170642.206069 fallocate 0x03dc66 thru 0x03ffff (0x239a bytes) 000965 1744170642.208054 mapread 0x033371 thru 0x035f10 (0x2ba0 bytes) 000966 1744170642.210754 trunc from 0x040000 to 0x00e02b (0x31fd4 bytes) 000967 1744170642.215886 punch from 0x00493c to 0x00741a (0x2adf bytes) 000970 1744170642.217955 mapwrite 0x00f4a3 thru 0x01779b (0x82f9 bytes) 000971 1744170642.229176 write 0x01e8be thru 0x026847 (0x7f8a bytes) 000972 1744170642.231122 MIRROR_RESYNC 2 mirrors 000973 1744170642.334987 read 0x01e4e2 thru 0x026847 (0x8366 bytes) 000974 1744170642.337377 trunc from 0x026848 to 0x02febb (0x9674 bytes) 000975 1744170642.352867 mapread 0x006a34 thru 0x010940 (0x9f0d bytes) 000976 1744170642.362091 mapwrite 0x00b87e thru 0x00cfaa (0x172d bytes) 000981 1744170642.368693 trunc from 0x02febc to 0x009615 (0x268a6 bytes) 000982 1744170642.375109 write 0x033682 thru 0x03e2c8 (0xac47 bytes) 000983 1744170642.376228 read 0x0283cc thru 0x02b794 (0x33c9 bytes) 000985 1744170642.379916 mapwrite 0x02c28c thru 0x02dafd (0x1872 bytes) 000986 1744170642.386362 fallocate 0x01a640 thru 0x01b742 (0x1103 bytes) 000987 1744170642.388015 fallocate 0x019d90 thru 0x027ca0 (0xdf11 bytes) 000989 1744170642.389782 fallocate 0x00cb6a thru 0x0110ad (0x4544 bytes) 000990 1744170642.391544 mapread 0x022e72 thru 0x023c8f (0xe1e bytes) 000991 1744170642.394876 MIRROR_RESYNC 2 mirrors 000992 1744170642.511934 MIRROR_RESYNC 2 mirrors 000993 1744170642.549685 mapread 0x033c26 thru 0x0349fa (0xdd5 bytes) 000994 1744170642.551965 trunc from 0x03e2c9 to 0x02ba5e (0x1286a bytes) 000995 1744170642.566486 write 0x02fdb9 thru 0x038587 (0x87cf bytes) 000996 1744170642.567493 mapwrite 0x015e97 thru 0x0188c6 (0x2a30 bytes) 000997 1744170642.576070 punch from 0x025ee2 to 0x02f685 (0x97a4 bytes) 000998 1744170642.577803 fallocate 0x010e9c thru 0x014b6b (0x3cd0 bytes) 000999 1744170642.579487 fallocate 0x03e9a5 thru 0x03ffff (0x165b bytes) 001000 1744170642.580779 trunc from 0x038588 to 0x00a125 (0x2e462 bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 23:50:43 (1744170643) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000405:0xae5:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 23:50:45 (1744170645) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2947168 .. Extending file size to 3289408 .. failed Extending file size to 5961504 .. Extending file size to 6017920 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9127552 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8331872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8905984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9224192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8921920 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 4051 4052 4053 4055 4058 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:51:49 (1744170709) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6504512 .. Extending file size to 6283168 .. done Extending file size to 7194400 .. Extending file size to 7827520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8297888 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8439872 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8644320 .. Extending file size to 8665760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8699584 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8811808 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8715424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9014496 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8942528 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9227136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 8486 8487 8489 8491 8493 done 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (62s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:52:51 (1744170771) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2828 1284860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3288 1284400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15860 3591160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21760 3585260 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37620 7176420 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2828 1284860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3288 1284400 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15860 3591160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21760 3585260 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37620 7176420 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0291889 s, 71.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0398373 s, 52.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:52:55 (1744170775) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0223167 s, 94.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 23:52:57 (1744170777) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:52:58 (1744170778) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0192484 s, 109 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xed3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:53:00 (1744170780) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0181698 s, 115 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:53:01 (1744170781) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.018061 s, 116 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9a:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:53:24 (1744170804) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132177 s, 79.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xed4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xed4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xed5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xed5:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:53:57 (1744170837) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0199564 s, 105 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (30s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:54:27 (1744170867) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg419-server: Pool lustre.test_204f created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0195454 s, 107 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg419-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:55:14 (1744170914) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:55:15 (1744170915) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:55:17 (1744170917) oleg419-server: Pool lustre.test_206 created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg419-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:55:29 (1744170929) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177055 s, 59.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00901575 s, 116 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xe828a491 Mirror 3: 0xe828a491 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:55:31 (1744170931) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:55:32 (1744170932) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:55:32 (1744170932) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00665943 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ca0000 Read this (4 bytes): BBBB 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:55:45 (1744170945) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00225776 s, 3.6 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:55:47 (1744170947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105768 s, 99.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 23:55:47.000000000 -0400 Modify: 2025-04-08 23:55:47.000000000 -0400 Change: 2025-04-08 23:55:47.000000000 -0400 Birth: - PASS 210a (1s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:55:48 (1744170948) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00897134 s, 117 MB/s oleg419-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg419-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg419-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg419-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg419-server: O/280000401/d7/c7: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:56:00 (1744170960) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0264074 s, 1.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00525144 s, 780 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 8 23:56 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 8 23:56 /mnt/lustre/f211.sanity-flr PASS 211 (1s) == sanity-flr test complete, duration 681 sec ============ 23:56:01 (1744170961) === sanity-flr: start cleanup 23:56:02 (1744170962) === === sanity-flr: finish cleanup 23:56:03 (1744170963) ===