-----============= acceptance-small: sanity-flr ============----- Tue Apr 8 23:44:27 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg452-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:44:32 (1744170272) === oleg452-client.virtnet: executing check_config_client /mnt/lustre oleg452-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg452-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6d59000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6d59000.idle_timeout=debug disable quota as required oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:44:39 (1744170279) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7031] == sanity-flr test 0a: lfs mirror create with -N option == 23:44:39 (1744170279) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:44:43 (1744170283) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:44:44 (1744170284) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:44:44 (1744170284) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:44:50 (1744170290) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:44:50 (1744170290) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:44:51 (1744170291) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6d59000.stats=clear osc.lustre-OST0001-osc-ffff8800b6d59000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:44:53 (1744170293) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:44:55 (1744170295) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:44:57 (1744170297) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:44:58 (1744170298) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 23:45:00 (1744170300) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:45:01 (1744170301) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:45:02 (1744170302) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:45:03 (1744170303) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:45:06 (1744170306) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:45:07 (1744170307) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0595592 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317026 s, 33.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:45:09 (1744170309) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.102825 s, 143 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:45:11 (1744170311) Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:45:33 (1744170333) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.115365 s, 72.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:45:44 (1744170344) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:46:00 (1744170360) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.774237 s, 67.7 MB/s -rw-r--r-- 1 root root 52428800 Apr 8 23:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.290s user 0m0.000s sys 0m0.129s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.258s user 0m0.000s sys 0m0.116s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.245s user 0m0.000s sys 0m0.107s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d59000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:46:20 (1744170380) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:46:21 (1744170381) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:46:22 (1744170382) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:46:23 (1744170383) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 23:46:24 (1744170384) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.103144 s, 122 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:46:27 (1744170387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109517 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00909934 s, 115 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:46:29 (1744170389) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:46:31 (1744170391) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000519358 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000546772 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000557516 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000731362 s, 1.4 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:46:33 (1744170393) create test files with size 15744854 .. 4fe42c6f447d0211ad74c4deb11cf99c - cfc1a4c62480e00b561f65489c202964 - 8d0e4f638ca11b64cc376d71b8352c6c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 23:46:38 (1744170398) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:46:41 (1744170401) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:46:42 (1744170402) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243753 s, 43.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104921 s, 99.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:46:44 (1744170404) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0261086 s, 80.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0837378 s, 50.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0399275 s, 52.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:46:47 (1744170407) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 23:46:47 (1744170407) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:46:48 (1744170408) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00743412 s, 141 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:46:50 (1744170410) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:46:51 (1744170411) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:46:53 (1744170413) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0579736 s, 181 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:47:17 (1744170437) llite.lustre-ffff8800a52b0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0614559 s, 171 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff880130c0c800.xattr_cache=1 PASS 44e (2s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:47:19 (1744170439) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:47:26 (1744170446) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:47:32 (1744170452) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:47:33 (1744170453) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0554288 s, 56.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0456271 s, 68.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 54df282790fcaf34d0cc83d63719b2b8 - mirror 1 checksum: c5b51291ead1f143e7d71a32558eacbd - mirror 2 checksum: 54df282790fcaf34d0cc83d63719b2b8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 23:47:33 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 23:47:33 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 54df282790fcaf34d0cc83d63719b2b8 - mirror 1 checksum: c5b51291ead1f143e7d71a32558eacbd - mirror 2 checksum: c5b51291ead1f143e7d71a32558eacbd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 23:47:33 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 23:47:33 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:47:36 (1744170456) Failing mds1 on oleg452-server Stopping /mnt/lustre-mds1 (opts:) on oleg452-server 23:47:37 (1744170457) shut down facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server Failover mds1 to oleg452-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 23:47:50 (1744170470) targets are mounted 23:47:50 (1744170470) facet_failover done oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00968258 s, 108 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:47:55 (1744170475) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:47:55 (1744170475) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:47:56 (1744170476) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00221141 s, 7.4 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:47:58 (1744170478) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:47:58 (1744170478) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:47:59 (1744170479) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:48:03 (1744170483) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:48:20 (1744170500) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:48:37 (1744170517) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:48:54 (1744170534) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 23:48:55 (1744170535) mirror create pid 22487 mirror split pid 22488 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6744: 22488 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6744: 22488 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22487 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22488 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 23:49:57 (1744170597) Seed set to 597 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x16d0e 000001 1744170597.799946 trunc from 00000000 to 0x016d0d (0x16d0e bytes) 000004 1744170597.805125 trunc from 0x016d0e to 0x006f4c (0xfdc1 bytes) 000005 1744170597.810163 mapread 0x001d5f thru 0x006f4c (0x51ee bytes) 000006 1744170597.817481 mapread 0x001565 thru 0x006f4c (0x59e8 bytes) truncating to largest ever: 0x233cf 000009 1744170597.818335 trunc from 0x006f4d to 0x0233ce (0x1c482 bytes) 000010 1744170597.822548 mapread 0x0028c9 thru 0x00be12 (0x954a bytes) truncating to largest ever: 0x3145e 000013 1744170597.826126 trunc from 0x0233cf to 0x03145d (0xe08f bytes) 000016 1744170597.830593 mapread 0x0312e4 thru 0x03145d (0x17a bytes) 000017 1744170597.834265 write 0x03e274 thru 0x03ffff (0x1d8c bytes) 000018 1744170597.834812 write 0x0224da thru 0x02810e (0x5c35 bytes) 000023 1744170597.839886 trunc from 0x040000 to 0x006498 (0x39b67 bytes) 000024 1744170597.843925 mapread 0x0063dd thru 0x006498 (0xbc bytes) 000025 1744170597.844481 mapwrite 0x01dc73 thru 0x02b931 (0xdcbf bytes) 000026 1744170597.884398 trunc from 0x02b932 to 0x02abc6 (0xd6b bytes) 000027 1744170597.888520 mapwrite 0x02aa5e thru 0x0378c1 (0xce64 bytes) 000032 1744170597.914531 mapwrite 0x008627 thru 0x00c27a (0x3c54 bytes) 000033 1744170597.929229 mapwrite 0x0304a7 thru 0x0320cb (0x1c25 bytes) 000034 1744170597.938863 read 0x0188cf thru 0x027c90 (0xf3c2 bytes) 000039 1744170597.941324 write 0x0180d9 thru 0x020335 (0x825d bytes) 000042 1744170597.942369 read 0x034f8e thru 0x0378c1 (0x2934 bytes) 000043 1744170597.942700 trunc from 0x0378c2 to 0x01ac15 (0x1ccac bytes) 000044 1744170597.946567 trunc from 0x01ac16 to 0x02ead1 (0x13ebc bytes) 000046 1744170597.949920 mapwrite 0x0328cb thru 0x039ae5 (0x721b bytes) 000047 1744170597.968261 write 0x036579 thru 0x03d7d7 (0x725f bytes) 000050 1744170597.969530 read 0x014021 thru 0x020512 (0xc4f2 bytes) 000052 1744170597.970081 write 0x00c85f thru 0x010832 (0x3fd4 bytes) 000054 1744170597.970628 write 0x023fab thru 0x02a3cf (0x6425 bytes) 000055 1744170597.971014 write 0x01dbf6 thru 0x02bc5e (0xe069 bytes) 000057 1744170597.972148 read 0x005124 thru 0x008fd1 (0x3eae bytes) 000058 1744170597.972525 mapread 0x01a3c7 thru 0x01c6da (0x2314 bytes) 000059 1744170597.973007 read 0x024e63 thru 0x0328b2 (0xda50 bytes) 000060 1744170597.974915 MIRROR_EXTEND to 2 mirrors 000061 1744170598.078622 MIRROR_RESYNC 2 mirrors 000062 1744170598.123662 MIRROR_RESYNC 2 mirrors 000066 1744170598.161698 mapwrite 0x0079b8 thru 0x010082 (0x86cb bytes) 000069 1744170598.203841 write 0x002892 thru 0x0106d7 (0xde46 bytes) 000070 1744170598.204798 read 0x00fec3 thru 0x015b44 (0x5c82 bytes) 000071 1744170598.206810 write 0x013553 thru 0x014e2f (0x18dd bytes) 000072 1744170598.207206 write 0x008b44 thru 0x015d80 (0xd23d bytes) 000073 1744170598.207812 mapread 0x009c3e thru 0x0111bc (0x757f bytes) 000075 1744170598.208759 mapwrite 0x003921 thru 0x005e0f (0x24ef bytes) 000076 1744170598.221521 write 0x00800c thru 0x016f8a (0xef7f bytes) 000077 1744170598.224880 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000078 1744170598.262087 read 0x033178 thru 0x03964b (0x64d4 bytes) 000080 1744170598.264627 mapwrite 0x03b810 thru 0x03ffff (0x47f0 bytes) 000081 1744170598.282889 mapread 0x01e0ca thru 0x02812f (0xa066 bytes) 000082 1744170598.286710 read 0x033ede thru 0x03f76b (0xb88e bytes) 000084 1744170598.287239 read 0x03c7bb thru 0x03ffff (0x3845 bytes) 000086 1744170598.287739 write 0x0250ee thru 0x02e249 (0x915c bytes) 000087 1744170598.288282 mapwrite 0x03b932 thru 0x03e1f7 (0x28c6 bytes) 000088 1744170598.301242 MIRROR_EXTEND to 2 mirrors 000089 1744170598.400935 read 0x010753 thru 0x011cf9 (0x15a7 bytes) 000090 1744170598.402384 trunc from 0x040000 to 0x00ff21 (0x300de bytes) 000092 1744170598.421467 write 0x002f4f thru 0x00cf23 (0x9fd5 bytes) 000093 1744170598.424188 mapwrite 0x0328d8 thru 0x03ffff (0xd728 bytes) 000097 1744170598.450709 MIRROR_RESYNC 2 mirrors 000101 1744170598.537831 read 0x01ee1e thru 0x02a15b (0xb33e bytes) 000102 1744170598.540098 mapwrite 0x01fe1f thru 0x0274d5 (0x76b7 bytes) 000104 1744170598.581758 write 0x03d0ba thru 0x03ffff (0x2f46 bytes) 000105 1744170598.584842 write 0x03829b thru 0x03957c (0x12e2 bytes) 000106 1744170598.585383 read 0x031261 thru 0x03e4ea (0xd28a bytes) 000107 1744170598.591035 MIRROR_RESYNC 2 mirrors 000108 1744170598.680680 trunc from 0x040000 to 0x00f0b5 (0x30f4a bytes) 000111 1744170598.700591 read 0x006dcd thru 0x00f0b5 (0x82e9 bytes) 000114 1744170598.703644 trunc from 0x00f0b6 to 0x0060a0 (0x9015 bytes) 000115 1744170598.708578 mapread 0x0054a0 thru 0x005cf3 (0x854 bytes) 000116 1744170598.710572 trunc from 0x0060a1 to 0x02eecb (0x28e2b bytes) 000117 1744170598.715741 mapwrite 0x014ca3 thru 0x0241cd (0xf52b bytes) 000118 1744170598.745672 write 0x010144 thru 0x016ed5 (0x6d92 bytes) 000123 1744170598.747655 read 0x01347b thru 0x01b980 (0x8506 bytes) 000124 1744170598.748051 mapwrite 0x0152bd thru 0x024148 (0xee8c bytes) 000127 1744170598.767712 read 0x0221cc thru 0x02987c (0x76b1 bytes) 000128 1744170598.770802 mapread 0x0184b9 thru 0x025606 (0xd14e bytes) 000133 1744170598.773835 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000136 1744170598.808725 trunc from 0x02eecc to 0x0071ce (0x27cfd bytes) 000139 1744170598.815738 MIRROR_EXTEND to 2 mirrors 000140 1744170598.905691 mapwrite 0x022b99 thru 0x028dcf (0x6237 bytes) 000142 1744170598.935699 read 0x01b796 thru 0x028dcf (0xd63a bytes) 000145 1744170598.937915 mapread 0x0013c8 thru 0x007da1 (0x69da bytes) 000146 1744170598.940367 write 0x00c649 thru 0x01b300 (0xecb8 bytes) 000149 1744170598.941530 mapwrite 0x0005cc thru 0x008b74 (0x85a9 bytes) 000151 1744170598.957443 write 0x03fa65 thru 0x03ffff (0x59b bytes) 000152 1744170598.958101 read 0x01ea86 thru 0x023173 (0x46ee bytes) 000153 1744170598.958502 trunc from 0x040000 to 0x0167b3 (0x2984c bytes) 000156 1744170598.967932 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000157 1744170598.993823 mapread 0x01084b thru 0x0167b3 (0x5f69 bytes) 000158 1744170598.997193 trunc from 0x0167b4 to 0x00b27d (0xb536 bytes) 000169 1744170599.003123 mapwrite 0x0375e9 thru 0x03ffff (0x8a17 bytes) 000171 1744170599.021727 write 0x01e0ec thru 0x02839e (0xa2b3 bytes) 000172 1744170599.025387 trunc from 0x040000 to 0x01b0e4 (0x24f1b bytes) 000173 1744170599.029061 mapwrite 0x030737 thru 0x03fc0d (0xf4d7 bytes) 000177 1744170599.052817 read 0x037a45 thru 0x03a791 (0x2d4d bytes) 000179 1744170599.053484 write 0x035b92 thru 0x03fb69 (0x9fd8 bytes) 000186 1744170599.056488 mapread 0x026c8e thru 0x02bb59 (0x4ecc bytes) 000189 1744170599.060416 trunc from 0x03fc0e to 0x02a645 (0x155c8 bytes) 000190 1744170599.065113 mapread 0x0065c8 thru 0x00e284 (0x7cbd bytes) 000192 1744170599.069497 read 0x00bc47 thru 0x017d96 (0xc150 bytes) 000193 1744170599.069858 mapread 0x022c06 thru 0x02912a (0x6525 bytes) 000194 1744170599.070304 write 0x021f8a thru 0x025d69 (0x3de0 bytes) 000195 1744170599.070813 mapread 0x0154f2 thru 0x02056a (0xb079 bytes) 000199 1744170599.075632 read 0x013d52 thru 0x0160d8 (0x2387 bytes) 000200 1744170599.077595 MIRROR_EXTEND to 2 mirrors 000201 1744170599.185268 mapread 0x01def5 thru 0x020786 (0x2892 bytes) 000202 1744170599.189859 write 0x03a95c thru 0x03ffff (0x56a4 bytes) 000204 1744170599.206845 MIRROR_EXTEND to 3 mirrors 000206 1744170599.302751 trunc from 0x040000 to 0x01b9ed (0x24612 bytes) 000207 1744170599.320916 mapwrite 0x031d41 thru 0x03ffff (0xe2bf bytes) 000208 1744170599.346547 mapread 0x0264a2 thru 0x031f57 (0xbab6 bytes) 000209 1744170599.348762 write 0x01293b thru 0x01e57f (0xbc45 bytes) 000211 1744170599.352254 mapwrite 0x0302fe thru 0x03ffff (0xfd02 bytes) truncating to largest ever: 0x38b61 000212 1744170599.366436 trunc from 0x040000 to 0x038b60 (0x749f bytes) 000213 1744170599.370339 mapwrite 0x030ac2 thru 0x03ed99 (0xe2d8 bytes) 000215 1744170599.391078 mapread 0x017b12 thru 0x025bec (0xe0db bytes) 000216 1744170599.395314 trunc from 0x03ed9a to 0x007185 (0x37c14 bytes) 000218 1744170599.399743 MIRROR_RESYNC 3 mirrors 000221 1744170599.497254 trunc from 0x007186 to 0x02df80 (0x26dfb bytes) 000224 1744170599.515815 MIRROR_RESYNC 3 mirrors 000225 1744170599.624265 read 0x014d6f thru 0x016044 (0x12d6 bytes) 000227 1744170599.625820 trunc from 0x02df81 to 0x0212d2 (0xccae bytes) 000230 1744170599.644297 read 0x020b2c thru 0x0212d2 (0x7a7 bytes) 000234 1744170599.648544 read 0x021254 thru 0x0212d2 (0x7f bytes) 000235 1744170599.648936 mapread 0x011e75 thru 0x013f22 (0x20ae bytes) 000236 1744170599.649514 read 0x0061f9 thru 0x00b4af (0x52b7 bytes) 000239 1744170599.651822 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000242 1744170599.694010 mapwrite 0x03ff81 thru 0x03ffff (0x7f bytes) 000245 1744170599.709540 mapread 0x03f6cb thru 0x03ffff (0x935 bytes) 000247 1744170599.711816 MIRROR_EXTEND to 3 mirrors 000248 1744170599.799620 mapwrite 0x01359c thru 0x0197d9 (0x623e bytes) 000250 1744170599.845592 trunc from 0x040000 to 0x009135 (0x36eca bytes) 000251 1744170599.853390 mapwrite 0x031926 thru 0x03326c (0x1947 bytes) 000252 1744170599.869853 read 0x00b673 thru 0x018195 (0xcb23 bytes) 000254 1744170599.872773 read 0x01e906 thru 0x027987 (0x9082 bytes) 000258 1744170599.876573 trunc from 0x03326d to 0x013412 (0x1fe5a bytes) 000259 1744170599.880154 write 0x01d1a4 thru 0x028344 (0xb1a1 bytes) 000261 1744170599.881419 mapwrite 0x01b80a thru 0x028ccb (0xd4c2 bytes) 000263 1744170599.901251 read 0x0285c2 thru 0x028ccb (0x70a bytes) 000265 1744170599.901980 write 0x03ea2e thru 0x03ffff (0x15d2 bytes) 000267 1744170599.904305 MIRROR_RESYNC 3 mirrors 000269 1744170600.008429 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000271 1744170600.059715 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000272 1744170600.090317 read 0x03dd73 thru 0x03ffff (0x228d bytes) 000273 1744170600.091947 mapwrite 0x01b538 thru 0x02a498 (0xef61 bytes) 000274 1744170600.116497 mapwrite 0x02b17f thru 0x039aa1 (0xe923 bytes) 000276 1744170600.134476 trunc from 0x040000 to 0x00106c (0x3ef93 bytes) 000279 1744170600.142973 read 0x000221 thru 0x00106c (0xe4c bytes) 000280 1744170600.144375 read 0x000b48 thru 0x00106c (0x525 bytes) 000285 1744170600.146294 MIRROR_EXTEND to 2 mirrors 000290 1744170600.225304 write 0x02fdf2 thru 0x0335b7 (0x37c6 bytes) 000291 1744170600.239219 write 0x014cb4 thru 0x01a4ee (0x583b bytes) 000292 1744170600.241617 read 0x0240c0 thru 0x029351 (0x5292 bytes) truncating to largest ever: 0x3c8c5 000293 1744170600.243771 trunc from 0x0335b8 to 0x03c8c4 (0x930d bytes) truncating to largest ever: 0x3f1da 000294 1744170600.252839 trunc from 0x03c8c5 to 0x03f1d9 (0x2915 bytes) 000296 1744170600.258358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000298 1744170600.293567 write 0x0395db thru 0x03fd9b (0x67c1 bytes) 000299 1744170600.296361 read 0x03490d thru 0x03fd9b (0xb48f bytes) 000300 1744170600.298711 read 0x008af5 thru 0x01731e (0xe82a bytes) 000305 1744170600.304096 write 0x026c2d thru 0x029805 (0x2bd9 bytes) 000306 1744170600.304814 read 0x019c37 thru 0x01d7a6 (0x3b70 bytes) 000307 1744170600.305249 trunc from 0x03fd9c to 0x00bd93 (0x34008 bytes) 000309 1744170600.310525 read 0x0058ed thru 0x007451 (0x1b65 bytes) 000316 1744170600.314305 MIRROR_EXTEND to 2 mirrors 000318 1744170600.415129 read 0x001a13 thru 0x003d82 (0x2370 bytes) 000319 1744170600.417909 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000320 1744170600.446872 trunc from 0x00bd94 to 0x036832 (0x2aa9f bytes) 000321 1744170600.457482 mapread 0x031654 thru 0x031c21 (0x5ce bytes) 000323 1744170600.459269 mapread 0x01e448 thru 0x024700 (0x62b9 bytes) 000324 1744170600.462824 mapwrite 0x02f59a thru 0x0323f4 (0x2e5b bytes) 000325 1744170600.480688 trunc from 0x036833 to 0x001f3f (0x348f3 bytes) 000328 1744170600.486665 MIRROR_EXTEND to 2 mirrors 000329 1744170600.572568 mapread 0x001e31 thru 0x001f3f (0x10f bytes) 000330 1744170600.573070 mapwrite 0x03d56d thru 0x03ffff (0x2a93 bytes) 000332 1744170600.618036 write 0x001f0f thru 0x0023b0 (0x4a2 bytes) 000336 1744170600.621658 read 0x025a50 thru 0x028362 (0x2913 bytes) 000337 1744170600.623321 mapread 0x016f1e thru 0x0251f7 (0xe2da bytes) 000339 1744170600.628085 trunc from 0x040000 to 0x00314f (0x3ceb0 bytes) 000340 1744170600.633090 write 0x028706 thru 0x035b8d (0xd488 bytes) 000341 1744170600.633868 mapwrite 0x033855 thru 0x037498 (0x3c44 bytes) 000342 1744170600.651996 read 0x0321bd thru 0x037498 (0x52dc bytes) 000344 1744170600.652726 mapwrite 0x00eed8 thru 0x01ed9e (0xfec7 bytes) 000345 1744170600.669372 read 0x00dc63 thru 0x00fd27 (0x20c5 bytes) 000346 1744170600.669769 mapwrite 0x005afe thru 0x00e12c (0x862f bytes) 000348 1744170600.684556 read 0x017bdc thru 0x01f66a (0x7a8f bytes) 000349 1744170600.684988 trunc from 0x037499 to 0x03e2d7 (0x6e3f bytes) 000350 1744170600.689489 mapwrite 0x026946 thru 0x029417 (0x2ad2 bytes) 000351 1744170600.704443 mapread 0x02e8d9 thru 0x033294 (0x49bc bytes) 000352 1744170600.706358 MIRROR_RESYNC 2 mirrors 000356 1744170600.794576 trunc from 0x03e2d8 to 0x01443a (0x29e9d bytes) 000358 1744170600.815100 mapread 0x00eb7b thru 0x01443a (0x58c0 bytes) 000360 1744170600.821268 mapwrite 0x032197 thru 0x03ffff (0xde69 bytes) 000361 1744170600.846063 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000364 1744170600.876274 mapread 0x03e934 thru 0x03ffff (0x16cc bytes) 000366 1744170600.878779 mapwrite 0x024c9b thru 0x02a084 (0x53ea bytes) 000368 1744170600.896015 trunc from 0x040000 to 0x00db4e (0x324b1 bytes) 000370 1744170600.899958 read 0x00d4ed thru 0x00db4e (0x662 bytes) 000371 1744170600.901423 read 0x004c3d thru 0x00c128 (0x74ec bytes) 000374 1744170600.904013 write 0x03aa3e thru 0x03ffff (0x55c2 bytes) 000375 1744170600.905799 MIRROR_EXTEND to 2 mirrors 000376 1744170601.012708 write 0x01062b thru 0x016d90 (0x6766 bytes) 000377 1744170601.028876 write 0x037ba2 thru 0x03f6ae (0x7b0d bytes) 000379 1744170601.031807 read 0x02a8ab thru 0x0356d7 (0xae2d bytes) 000381 1744170601.034509 read 0x010863 thru 0x01a8f9 (0xa097 bytes) 000382 1744170601.038876 MIRROR_RESYNC 2 mirrors 000384 1744170601.135316 trunc from 0x040000 to 0x034ed0 (0xb12f bytes) 000386 1744170601.159550 mapread 0x01ca4c thru 0x01d31d (0x8d2 bytes) 000387 1744170601.163069 read 0x01b7b4 thru 0x025aba (0xa307 bytes) 000389 1744170601.166016 write 0x013001 thru 0x013f54 (0xf54 bytes) 000391 1744170601.169362 trunc from 0x034ed1 to 0x037ae7 (0x2c17 bytes) 000392 1744170601.175373 mapwrite 0x00c74c thru 0x00ea7f (0x2334 bytes) 000393 1744170601.189633 mapwrite 0x02b129 thru 0x0338e1 (0x87b9 bytes) 000399 1744170601.205481 trunc from 0x037ae8 to 0x0270cf (0x10a18 bytes) 000400 1744170601.209014 mapread 0x00c1bd thru 0x013b2c (0x7970 bytes) 000403 1744170601.210215 mapwrite 0x0329e3 thru 0x037768 (0x4d86 bytes) 000405 1744170601.228562 trunc from 0x037769 to 0x039097 (0x192f bytes) 000406 1744170601.231805 read 0x004678 thru 0x00e666 (0x9fef bytes) 000407 1744170601.233902 mapwrite 0x007b59 thru 0x016ae0 (0xef88 bytes) 000408 1744170601.247109 write 0x03e141 thru 0x03ffff (0x1ebf bytes) 000410 1744170601.249220 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000411 1744170601.287020 read 0x02fbbc thru 0x03bc93 (0xc0d8 bytes) 000412 1744170601.289308 write 0x00dfe3 thru 0x01c960 (0xe97e bytes) 000415 1744170601.292599 mapread 0x038ac3 thru 0x03a4b8 (0x19f6 bytes) 000417 1744170601.293366 write 0x02837c thru 0x03239a (0xa01f bytes) 000418 1744170601.297293 mapread 0x002110 thru 0x010495 (0xe386 bytes) 000419 1744170601.301446 mapwrite 0x03f31a thru 0x03ffff (0xce6 bytes) 000424 1744170601.318270 read 0x02868d thru 0x02d6ef (0x5063 bytes) 000428 1744170601.319411 trunc from 0x040000 to 0x03309d (0xcf62 bytes) 000431 1744170601.324899 trunc from 0x03309e to 0x00a6b6 (0x289e7 bytes) 000432 1744170601.328574 trunc from 0x00a6b7 to 0x015e59 (0xb7a3 bytes) 000434 1744170601.333338 read 0x0125c8 thru 0x015e59 (0x3892 bytes) 000437 1744170601.336834 write 0x01bf03 thru 0x01f553 (0x3651 bytes) 000438 1744170601.337490 mapread 0x00c032 thru 0x01672e (0xa6fd bytes) 000440 1744170601.340022 trunc from 0x01f554 to 0x0242b5 (0x4d62 bytes) 000441 1744170601.344092 trunc from 0x0242b6 to 0x004963 (0x1f952 bytes) 000442 1744170601.347370 mapwrite 0x038edd thru 0x03c7f4 (0x3918 bytes) 000443 1744170601.368130 mapwrite 0x02ab5e thru 0x02b207 (0x6aa bytes) 000444 1744170601.379314 mapread 0x012624 thru 0x015de7 (0x37c4 bytes) 000446 1744170601.384143 read 0x035f1a thru 0x03c7f4 (0x68db bytes) 000448 1744170601.384655 mapwrite 0x025a95 thru 0x032916 (0xce82 bytes) 000449 1744170601.399689 trunc from 0x03c7f5 to 0x02645d (0x16397 bytes) 000450 1744170601.403618 read 0x01926e thru 0x022ddd (0x9b70 bytes) 000451 1744170601.406316 mapwrite 0x0131d3 thru 0x0169d9 (0x3807 bytes) 000452 1744170601.419124 MIRROR_EXTEND to 2 mirrors 000453 1744170601.510982 mapwrite 0x00fe79 thru 0x01320f (0x3397 bytes) 000456 1744170601.555315 trunc from 0x02645e to 0x00d71a (0x18d43 bytes) 000457 1744170601.563602 mapwrite 0x012677 thru 0x01d780 (0xb10a bytes) 000458 1744170601.585107 write 0x02d972 thru 0x03947c (0xbb0b bytes) 000460 1744170601.586279 mapwrite 0x0283e1 thru 0x030818 (0x8438 bytes) 000462 1744170601.602280 mapread 0x00e957 thru 0x014832 (0x5edc bytes) 000464 1744170601.603031 trunc from 0x03947d to 0x02921e (0x1025e bytes) 000465 1744170601.608415 MIRROR_RESYNC 2 mirrors 000470 1744170601.692290 mapread 0x026fee thru 0x02921e (0x2231 bytes) 000471 1744170601.697080 MIRROR_EXTEND to 3 mirrors 000473 1744170601.788130 read 0x0195aa thru 0x0269d1 (0xd428 bytes) 000474 1744170601.792020 trunc from 0x02921f to 0x032ae8 (0x98ca bytes) 000477 1744170601.809065 mapread 0x02fb75 thru 0x0312fd (0x1789 bytes) 000478 1744170601.812988 mapread 0x018eef thru 0x023e2a (0xaf3c bytes) 000481 1744170601.817540 write 0x00c73f thru 0x010d26 (0x45e8 bytes) 000482 1744170601.820016 write 0x01fa99 thru 0x0205bd (0xb25 bytes) 000483 1744170601.820573 trunc from 0x032ae9 to 0x018aa4 (0x1a044 bytes) 000484 1744170601.824706 mapread 0x0077e9 thru 0x0111b9 (0x99d1 bytes) 000486 1744170601.825560 mapwrite 0x023ff2 thru 0x0267a7 (0x27b6 bytes) 000489 1744170601.843000 write 0x00a5c9 thru 0x00e3e4 (0x3e1c bytes) 000490 1744170601.843621 mapread 0x01c38b thru 0x0267a7 (0xa41d bytes) 000492 1744170601.847344 MIRROR_EXTEND to 4 mirrors 000495 1744170601.938483 write 0x0373a6 thru 0x03ffff (0x8c5a bytes) 000496 1744170601.952519 mapread 0x00ab5c thru 0x013da3 (0x9248 bytes) 000498 1744170601.958286 MIRROR_RESYNC 4 mirrors 000499 1744170602.094172 write 0x02d9a9 thru 0x03842d (0xaa85 bytes) 000500 1744170602.108036 mapwrite 0x03a9c6 thru 0x03ffff (0x563a bytes) 000501 1744170602.125262 read 0x018247 thru 0x02221c (0x9fd6 bytes) 000502 1744170602.127783 mapwrite 0x006668 thru 0x00c795 (0x612e bytes) 000504 1744170602.143926 write 0x02f538 thru 0x03035d (0xe26 bytes) 000506 1744170602.144739 write 0x030d9c thru 0x034254 (0x34b9 bytes) 000509 1744170602.145624 read 0x01fc3e thru 0x025b2e (0x5ef1 bytes) 000510 1744170602.147957 write 0x02c620 thru 0x039389 (0xcd6a bytes) 000511 1744170602.148668 write 0x02796f thru 0x03262e (0xacc0 bytes) 000512 1744170602.149326 read 0x01ff3c thru 0x02ee82 (0xef47 bytes) 000513 1744170602.149659 mapread 0x035c2c thru 0x03fc83 (0xa058 bytes) 000523 1744170602.152356 read 0x01e70c thru 0x024f63 (0x6858 bytes) 000524 1744170602.152718 trunc from 0x040000 to 0x00765f (0x389a0 bytes) 000526 1744170602.157689 mapwrite 0x0056ad thru 0x0071ad (0x1b01 bytes) 000527 1744170602.171360 trunc from 0x007660 to 0x0244c3 (0x1ce64 bytes) 000528 1744170602.176114 MIRROR_EXTEND to 5 mirrors 000532 1744170602.269903 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000533 1744170602.317487 mapwrite 0x030d5e thru 0x0378a6 (0x6b49 bytes) 000534 1744170602.348689 write 0x037da5 thru 0x038e12 (0x106e bytes) 000535 1744170602.350415 MIRROR_EXTEND to 5 mirrors 000536 1744170602.439241 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000537 1744170602.492422 write 0x024c9c thru 0x032296 (0xd5fb bytes) 000538 1744170602.508709 trunc from 0x038e13 to 0x0231af (0x15c63 bytes) 000539 1744170602.515897 read 0x017323 thru 0x022cdf (0xb9bd bytes) 000540 1744170602.518039 write 0x00db4b thru 0x016c52 (0x9108 bytes) 000541 1744170602.521273 write 0x0357d0 thru 0x03ffff (0xa830 bytes) 000544 1744170602.522567 write 0x0074c1 thru 0x01095a (0x949a bytes) 000545 1744170602.523100 mapread 0x029cf6 thru 0x02d6fc (0x3a07 bytes) 000546 1744170602.525354 read 0x02ef36 thru 0x03bb61 (0xcc2c bytes) 000547 1744170602.526797 MIRROR_EXTEND to 5 mirrors 000553 1744170602.638605 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000554 1744170602.676452 read 0x01c2cf thru 0x0274f5 (0xb227 bytes) 000556 1744170602.678980 write 0x00fc1b thru 0x01e8d7 (0xecbd bytes) 000557 1744170602.683127 trunc from 0x040000 to 0x03b368 (0x4c97 bytes) 000558 1744170602.690759 MIRROR_EXTEND to 5 mirrors 000560 1744170602.786672 trunc from 0x03b369 to 0x00962c (0x31d3c bytes) 000561 1744170602.811100 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000562 1744170602.860755 MIRROR_RESYNC 4 mirrors 000564 1744170602.984006 write 0x00b199 thru 0x012221 (0x7089 bytes) 000565 1744170602.997530 trunc from 0x012222 to 0x02cd9a (0x1ab79 bytes) 000566 1744170603.004366 read 0x024501 thru 0x027a85 (0x3585 bytes) 000567 1744170603.006798 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000568 1744170603.053387 mapread 0x004b9b thru 0x01125a (0xc6c0 bytes) 000571 1744170603.060402 trunc from 0x02cd9b to 0x013828 (0x19572 bytes) 000574 1744170603.064957 trunc from 0x013829 to 0x00606e (0xd7ba bytes) 000576 1744170603.069341 mapwrite 0x024a65 thru 0x02a345 (0x58e1 bytes) 000577 1744170603.087739 MIRROR_EXTEND to 4 mirrors 000578 1744170603.182566 write 0x038901 thru 0x03ffff (0x76ff bytes) 000579 1744170603.197736 trunc from 0x040000 to 0x01bf4f (0x240b0 bytes) 000580 1744170603.206096 mapwrite 0x0373aa thru 0x03ffff (0x8c56 bytes) 000582 1744170603.231242 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000584 1744170603.285503 read 0x0347e6 thru 0x03e91a (0xa135 bytes) 000585 1744170603.288650 write 0x03780e thru 0x03ffff (0x87f2 bytes) 000586 1744170603.290930 MIRROR_RESYNC 3 mirrors 000587 1744170603.417859 mapwrite 0x006b47 thru 0x0116c2 (0xab7c bytes) 000588 1744170603.466802 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000589 1744170603.517418 mapread 0x00e1da thru 0x016ee9 (0x8d10 bytes) 000591 1744170603.522383 write 0x02e8df thru 0x034241 (0x5963 bytes) 000593 1744170603.525509 write 0x005e82 thru 0x0079a9 (0x1b28 bytes) 000594 1744170603.528160 mapread 0x0354aa thru 0x03ffff (0xab56 bytes) 000595 1744170603.533219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000596 1744170603.570563 mapread 0x00a895 thru 0x016b6b (0xc2d7 bytes) 000597 1744170603.577457 mapread 0x021b4e thru 0x026ec8 (0x537b bytes) 000599 1744170603.580786 read 0x03ca87 thru 0x03dd68 (0x12e2 bytes) 000607 1744170603.584522 mapwrite 0x025708 thru 0x033829 (0xe122 bytes) 000611 1744170603.601382 mapread 0x03f3db thru 0x03ffff (0xc25 bytes) 000612 1744170603.604292 read 0x00c7c3 thru 0x019ae7 (0xd325 bytes) 000613 1744170603.604771 mapread 0x021f0a thru 0x025e68 (0x3f5f bytes) 000614 1744170603.605236 trunc from 0x040000 to 0x03bc9c (0x4363 bytes) 000615 1744170603.612491 mapread 0x030a55 thru 0x03bc9c (0xb248 bytes) 000616 1744170603.613065 mapread 0x029e31 thru 0x038e1b (0xefeb bytes) 000617 1744170603.613463 write 0x00154d thru 0x00a342 (0x8df6 bytes) 000618 1744170603.615870 mapread 0x0192e2 thru 0x0211c2 (0x7ee1 bytes) 000619 1744170603.616369 mapread 0x037281 thru 0x03a595 (0x3315 bytes) 000620 1744170603.616803 mapwrite 0x01d71f thru 0x02a906 (0xd1e8 bytes) 000622 1744170603.631704 mapwrite 0x035dfe thru 0x03ffff (0xa202 bytes) 000623 1744170603.647650 mapwrite 0x006986 thru 0x00fa0c (0x9087 bytes) 000624 1744170603.661904 read 0x023b91 thru 0x02420c (0x67c bytes) 000625 1744170603.662289 mapread 0x02d935 thru 0x035f8e (0x865a bytes) 000627 1744170603.662955 write 0x00ce4c thru 0x00e6fd (0x18b2 bytes) 000628 1744170603.663370 mapread 0x034baa thru 0x036506 (0x195d bytes) 000630 1744170603.663812 mapwrite 0x035180 thru 0x03ffff (0xae80 bytes) 000633 1744170603.678540 write 0x025e4a thru 0x02b532 (0x56e9 bytes) 000638 1744170603.680432 mapwrite 0x00854c thru 0x00ecd8 (0x678d bytes) 000640 1744170603.696054 mapread 0x00c43c thru 0x014758 (0x831d bytes) 000642 1744170603.696627 read 0x020e07 thru 0x0290c0 (0x82ba bytes) 000643 1744170603.696925 mapread 0x038474 thru 0x03ffff (0x7b8c bytes) 000646 1744170603.697932 mapwrite 0x030a81 thru 0x0366bf (0x5c3f bytes) 000649 1744170603.710785 read 0x0119ae thru 0x015262 (0x38b5 bytes) 000650 1744170603.711101 mapread 0x00ccba thru 0x010042 (0x3389 bytes) 000652 1744170603.711596 write 0x016ab2 thru 0x01c606 (0x5b55 bytes) 000653 1744170603.712064 mapread 0x007632 thru 0x00bf40 (0x490f bytes) 000656 1744170603.712925 read 0x0022ed thru 0x00d203 (0xaf17 bytes) 000657 1744170603.713167 read 0x03e7d9 thru 0x03ffff (0x1827 bytes) 000658 1744170603.713381 trunc from 0x040000 to 0x02904f (0x16fb0 bytes) 000659 1744170603.717064 read 0x022af6 thru 0x02904f (0x655a bytes) 000661 1744170603.717573 mapwrite 0x0076a9 thru 0x01469b (0xcff3 bytes) 000665 1744170603.733101 read 0x00ce59 thru 0x012b65 (0x5d0d bytes) 000666 1744170603.733513 mapwrite 0x009939 thru 0x013ba7 (0xa26f bytes) 000668 1744170603.745562 read 0x002e29 thru 0x0103b3 (0xd58b bytes) 000669 1744170603.745899 trunc from 0x029050 to 0x019720 (0xf92f bytes) 000673 1744170603.750522 trunc from 0x019721 to 0x0133d8 (0x6348 bytes) 000674 1744170603.753728 read 0x00cefb thru 0x0128d7 (0x59dd bytes) 000675 1744170603.754043 mapwrite 0x030f4b thru 0x03745b (0x6511 bytes) 000677 1744170603.775884 mapwrite 0x01d6d0 thru 0x02ca14 (0xf345 bytes) 000678 1744170603.793064 write 0x03cc47 thru 0x03ffff (0x33b9 bytes) 000681 1744170603.794608 mapread 0x02bf68 thru 0x03776f (0xb808 bytes) 000682 1744170603.795220 write 0x0312c7 thru 0x03e6fc (0xd436 bytes) 000683 1744170603.796070 mapwrite 0x0308c5 thru 0x035f32 (0x566e bytes) 000684 1744170603.812031 mapread 0x03a011 thru 0x03ffff (0x5fef bytes) 000686 1744170603.812816 mapread 0x009ad9 thru 0x00d1c0 (0x36e8 bytes) 000687 1744170603.813337 read 0x03e6d1 thru 0x03f596 (0xec6 bytes) 000689 1744170603.814006 read 0x01e411 thru 0x029edf (0xbacf bytes) 000690 1744170603.814414 write 0x0091ca thru 0x00a58c (0x13c3 bytes) 000692 1744170603.815172 mapread 0x0155bc thru 0x01d843 (0x8288 bytes) 000693 1744170603.815640 write 0x01f343 thru 0x022846 (0x3504 bytes) 000694 1744170603.816143 trunc from 0x040000 to 0x036090 (0x9f6f bytes) 000696 1744170603.820745 mapwrite 0x00a16d thru 0x019310 (0xf1a4 bytes) 000700 1744170603.839119 read 0x002427 thru 0x004916 (0x24f0 bytes) 000704 1744170603.839989 mapwrite 0x0075b2 thru 0x00c0c3 (0x4b12 bytes) 000706 1744170603.852446 trunc from 0x036091 to 0x017066 (0x1f02a bytes) 000707 1744170603.856058 mapwrite 0x0016bf thru 0x0108bb (0xf1fd bytes) 000711 1744170603.873047 mapread 0x00f455 thru 0x017066 (0x7c12 bytes) 000713 1744170603.873918 read 0x00c62f thru 0x017066 (0xaa38 bytes) 000714 1744170603.874288 trunc from 0x017067 to 0x0363ed (0x1f387 bytes) 000715 1744170603.879106 MIRROR_EXTEND to 2 mirrors 000716 1744170603.978363 read 0x029be7 thru 0x02dc3e (0x4058 bytes) 000717 1744170603.980534 mapwrite 0x008605 thru 0x01179d (0x9199 bytes) 000718 1744170604.028614 trunc from 0x0363ee to 0x0062a9 (0x30144 bytes) 000720 1744170604.038480 MIRROR_EXTEND to 3 mirrors 000721 1744170604.134422 MIRROR_RESYNC 3 mirrors 000724 1744170604.226696 read 0x000b69 thru 0x0062a9 (0x5741 bytes) 000725 1744170604.229153 write 0x029f4e thru 0x02fa76 (0x5b29 bytes) 000726 1744170604.246358 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000727 1744170604.299487 mapread 0x028190 thru 0x02fa76 (0x78e7 bytes) 000728 1744170604.304279 mapread 0x011fd2 thru 0x019f51 (0x7f80 bytes) 000730 1744170604.310654 MIRROR_EXTEND to 3 mirrors 000731 1744170604.408164 write 0x03fd57 thru 0x03ffff (0x2a9 bytes) 000734 1744170604.422835 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000735 1744170604.466983 read 0x02c963 thru 0x02e077 (0x1715 bytes) 000736 1744170604.468593 write 0x0077bd thru 0x01134c (0x9b90 bytes) 000738 1744170604.471449 mapwrite 0x025d74 thru 0x0354d6 (0xf763 bytes) 000739 1744170604.491616 mapread 0x032b21 thru 0x03fde2 (0xd2c2 bytes) 000741 1744170604.493738 trunc from 0x040000 to 0x029c1c (0x163e3 bytes) 000742 1744170604.500902 mapread 0x00309e thru 0x006644 (0x35a7 bytes) 000743 1744170604.504957 MIRROR_RESYNC 2 mirrors 000745 1744170604.587811 write 0x01f4ee thru 0x026624 (0x7137 bytes) 000746 1744170604.605874 mapwrite 0x02ddad thru 0x0338ca (0x5b1e bytes) 000748 1744170604.636796 MIRROR_EXTEND to 3 mirrors 000749 1744170604.750649 mapread 0x01b50c thru 0x027d45 (0xc83a bytes) 000750 1744170604.758532 read 0x00673d thru 0x00ea55 (0x8319 bytes) 000752 1744170604.762350 mapread 0x01932e thru 0x0248b7 (0xb58a bytes) 000753 1744170604.764127 read 0x02ad18 thru 0x0338ca (0x8bb3 bytes) 000754 1744170604.768580 MIRROR_EXTEND to 4 mirrors 000756 1744170604.855580 MIRROR_RESYNC 4 mirrors 000757 1744170604.946854 mapread 0x013731 thru 0x019ed4 (0x67a4 bytes) 000758 1744170604.951906 mapwrite 0x028af3 thru 0x02d446 (0x4954 bytes) 000760 1744170604.996554 MIRROR_RESYNC 4 mirrors 000762 1744170605.130960 mapwrite 0x02a9c2 thru 0x0328e4 (0x7f23 bytes) 000764 1744170605.170546 read 0x02c2d5 thru 0x032277 (0x5fa3 bytes) 000765 1744170605.170937 read 0x02aa05 thru 0x0338ca (0x8ec6 bytes) 000766 1744170605.172507 MIRROR_RESYNC 4 mirrors 000768 1744170605.292223 mapwrite 0x01dece thru 0x022dbc (0x4eef bytes) 000770 1744170605.334868 mapread 0x019992 thru 0x023398 (0x9a07 bytes) 000771 1744170605.335489 mapwrite 0x0316a0 thru 0x03525e (0x3bbf bytes) 000772 1744170605.358328 mapwrite 0x034ccb thru 0x03c2e2 (0x7618 bytes) 000774 1744170605.378124 mapread 0x007479 thru 0x00b648 (0x41d0 bytes) 000776 1744170605.383296 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000778 1744170605.435519 read 0x039d40 thru 0x03c2e2 (0x25a3 bytes) 000779 1744170605.438866 MIRROR_EXTEND to 4 mirrors 000783 1744170605.533042 read 0x0072e6 thru 0x00ed7a (0x7a95 bytes) 000784 1744170605.535316 write 0x01d1e9 thru 0x01e57e (0x1396 bytes) 000785 1744170605.551590 mapread 0x014324 thru 0x017f17 (0x3bf4 bytes) 000786 1744170605.555617 MIRROR_RESYNC 4 mirrors 000787 1744170605.688311 read 0x001b4d thru 0x004f31 (0x33e5 bytes) 000789 1744170605.690194 trunc from 0x03c2e3 to 0x01a0a2 (0x22240 bytes) 000790 1744170605.710307 trunc from 0x01a0a3 to 0x00a531 (0xfb71 bytes) 000791 1744170605.716607 MIRROR_EXTEND to 5 mirrors 000792 1744170605.813311 mapread 0x00709d thru 0x008e24 (0x1d88 bytes) 000795 1744170605.818916 mapread 0x009102 thru 0x00a531 (0x1430 bytes) 000796 1744170605.822299 read 0x00015c thru 0x004c00 (0x4aa5 bytes) 000798 1744170605.825478 mapread 0x008af5 thru 0x0090d3 (0x5df bytes) 000799 1744170605.827296 read 0x009cea thru 0x00a531 (0x848 bytes) 000800 1744170605.828521 mapread 0x004fea thru 0x00a531 (0x5548 bytes) 000801 1744170605.831067 read 0x004de2 thru 0x00a531 (0x5750 bytes) 000802 1744170605.832288 mapwrite 0x0083c2 thru 0x00b6d3 (0x3312 bytes) 000804 1744170605.865423 trunc from 0x00b6d4 to 0x021eee (0x1681b bytes) 000806 1744170605.872517 MIRROR_EXTEND to 6 mirrors 000808 1744170605.964309 read 0x01aff6 thru 0x021eee (0x6ef9 bytes) 000811 1744170605.970172 MIRROR_RESYNC 6 mirrors 000812 1744170606.129719 read 0x00cd3a thru 0x00fa94 (0x2d5b bytes) 000813 1744170606.131622 mapwrite 0x01f619 thru 0x0228c8 (0x32b0 bytes) 000814 1744170606.168525 MIRROR_EXTEND to 7 mirrors 000822 1744170606.265429 MIRROR_RESYNC 7 mirrors 000825 1744170606.430546 read 0x013941 thru 0x018eaa (0x556a bytes) 000826 1744170606.433025 MIRROR_EXTEND to 8 mirrors 000828 1744170606.533494 write 0x0284b9 thru 0x032175 (0x9cbd bytes) 000830 1744170606.549274 read 0x006c1d thru 0x00cf23 (0x6307 bytes) 000831 1744170606.551629 trunc from 0x032176 to 0x024d5d (0xd418 bytes) 000832 1744170606.559054 read 0x0065e0 thru 0x015770 (0xf191 bytes) 000834 1744170606.562039 write 0x033c52 thru 0x03ffff (0xc3ae bytes) 000835 1744170606.562764 trunc from 0x040000 to 0x02158a (0x1ea75 bytes) 000836 1744170606.566370 mapwrite 0x03706a thru 0x03ffff (0x8f96 bytes) 000838 1744170606.586430 trunc from 0x040000 to 0x0233c6 (0x1cc39 bytes) 000839 1744170606.590117 read 0x0067d4 thru 0x00c399 (0x5bc6 bytes) 000840 1744170606.590548 write 0x000541 thru 0x00501e (0x4ade bytes) 000843 1744170606.591846 write 0x023485 thru 0x02c124 (0x8ca0 bytes) 000845 1744170606.594052 mapwrite 0x0255e6 thru 0x0343e9 (0xee04 bytes) 000852 1744170606.613896 write 0x033c73 thru 0x035f9b (0x2329 bytes) 000856 1744170606.615257 trunc from 0x035f9c to 0x02c852 (0x9749 bytes) 000858 1744170606.619519 trunc from 0x02c853 to 0x00544f (0x27403 bytes) 000859 1744170606.623299 write 0x00edba thru 0x01c498 (0xd6df bytes) 000860 1744170606.624365 mapwrite 0x033379 thru 0x03f438 (0xc0c0 bytes) 000863 1744170606.647788 MIRROR_EXTEND to 9 mirrors 000864 1744170606.739557 read 0x006a27 thru 0x0153c5 (0xe99f bytes) 000865 1744170606.742371 trunc from 0x03f439 to 0x02fc4f (0xf7e9 bytes) 000866 1744170606.764465 MIRROR_EXTEND to 10 mirrors 000867 1744170606.855689 read 0x00c11a thru 0x00c6b8 (0x59f bytes) 000868 1744170606.857628 trunc from 0x02fc50 to 0x0260f3 (0x9b5c bytes) 000869 1744170606.875740 read 0x0159e8 thru 0x01ccf9 (0x7312 bytes) 000870 1744170606.878847 mapread 0x00d758 thru 0x01514b (0x79f4 bytes) 000872 1744170606.879607 write 0x031868 thru 0x03ffff (0xe798 bytes) 000873 1744170606.880512 write 0x03dcf1 thru 0x03ffff (0x230f bytes) 000874 1744170606.881060 write 0x01bab4 thru 0x0295f3 (0xdb40 bytes) 000875 1744170606.883532 read 0x02d982 thru 0x039563 (0xbbe2 bytes) 000876 1744170606.883852 write 0x03b234 thru 0x03ffff (0x4dcc bytes) 000877 1744170606.884416 write 0x01af37 thru 0x021ba4 (0x6c6e bytes) 000878 1744170606.884959 trunc from 0x040000 to 0x00fe23 (0x301dc bytes) 000880 1744170606.889613 mapwrite 0x0385c3 thru 0x03e074 (0x5ab2 bytes) 000881 1744170606.908397 write 0x015976 thru 0x02010a (0xa795 bytes) 000882 1744170606.910135 MIRROR_RESYNC 10 mirrors 000883 1744170607.167550 mapread 0x0184b6 thru 0x023cde (0xb829 bytes) 000884 1744170607.173008 read 0x035525 thru 0x03e074 (0x8b50 bytes) 000885 1744170607.174874 mapread 0x00ef0a thru 0x018c07 (0x9cfe bytes) 000890 1744170607.177328 trunc from 0x03e075 to 0x007506 (0x36b6e bytes) 000892 1744170607.194712 mapwrite 0x023b77 thru 0x028ca1 (0x512b bytes) 000893 1744170607.216840 read 0x004813 thru 0x00aec3 (0x66b1 bytes) 000896 1744170607.220542 read 0x0124de thru 0x01764e (0x5171 bytes) 000898 1744170607.221110 trunc from 0x028ca2 to 0x0060f1 (0x22bb0 bytes) 000899 1744170607.225120 trunc from 0x0060f2 to 0x0269b8 (0x208c7 bytes) 000901 1744170607.228846 write 0x00e097 thru 0x01ca36 (0xe9a0 bytes) 000902 1744170607.232394 trunc from 0x0269b9 to 0x020110 (0x68a8 bytes) 000904 1744170607.236575 trunc from 0x020111 to 0x023c81 (0x3b71 bytes) 000905 1744170607.239902 read 0x004bb2 thru 0x01188f (0xccde bytes) 000906 1744170607.240363 trunc from 0x023c82 to 0x012717 (0x1156a bytes) 000908 1744170607.245997 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000909 1744170607.296283 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000910 1744170607.340688 mapwrite 0x00eceb thru 0x01738b (0x86a1 bytes) 000912 1744170607.365468 read 0x00aa8b thru 0x01259c (0x7b12 bytes) 000913 1744170607.365949 mapwrite 0x00b691 thru 0x011375 (0x5ce5 bytes) 000915 1744170607.385710 trunc from 0x01738c to 0x008e56 (0xe535 bytes) 000917 1744170607.389580 trunc from 0x008e57 to 0x00c610 (0x37ba bytes) 000920 1744170607.394943 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000921 1744170607.441322 mapwrite 0x00e3a1 thru 0x015b96 (0x77f6 bytes) 000922 1744170607.466300 read 0x0135bb thru 0x015b96 (0x25dc bytes) 000923 1744170607.466637 write 0x009e71 thru 0x01410f (0xa29f bytes) 000924 1744170607.467220 mapwrite 0x023f0b thru 0x02e4f7 (0xa5ed bytes) 000925 1744170607.489078 mapread 0x01bf02 thru 0x029f1c (0xe01b bytes) 000928 1744170607.492022 trunc from 0x02e4f8 to 0x021155 (0xd3a2 bytes) 000931 1744170607.497219 trunc from 0x021156 to 0x00a737 (0x16a1e bytes) 000932 1744170607.500859 write 0x014292 thru 0x01c314 (0x8083 bytes) 000934 1744170607.501828 mapread 0x0012ed thru 0x00578f (0x44a3 bytes) 000935 1744170607.504136 write 0x020102 thru 0x02fa3b (0xf93a bytes) 000936 1744170607.505132 mapread 0x02f62a thru 0x02fa3b (0x412 bytes) 000938 1744170607.505906 mapwrite 0x00f3d0 thru 0x014ea2 (0x5ad3 bytes) 000939 1744170607.518788 read 0x00ea1f thru 0x011aa2 (0x3084 bytes) 000942 1744170607.519716 mapwrite 0x0236fb thru 0x03032c (0xcc32 bytes) 000943 1744170607.540406 MIRROR_EXTEND to 8 mirrors 000945 1744170607.634486 MIRROR_EXTEND to 9 mirrors 000947 1744170607.729719 mapread 0x01820c thru 0x027af1 (0xf8e6 bytes) 000948 1744170607.738536 MIRROR_EXTEND to 10 mirrors 000951 1744170607.840237 write 0x037fdf thru 0x03ffff (0x8021 bytes) 000953 1744170607.854163 read 0x03bdfc thru 0x03ffff (0x4204 bytes) 000954 1744170607.854562 mapread 0x037355 thru 0x03ffff (0x8cab bytes) 000955 1744170607.855075 mapread 0x01268e thru 0x0217f0 (0xf163 bytes) 000957 1744170607.861099 read 0x0306e2 thru 0x033835 (0x3154 bytes) 000961 1744170607.866293 MIRROR_EXTEND to 11 mirrors 000963 1744170607.981568 mapread 0x03a854 thru 0x03ffff (0x57ac bytes) 000965 1744170607.988546 read 0x01992a thru 0x024b3f (0xb216 bytes) 000966 1744170607.992034 MIRROR_RESYNC 11 mirrors 000967 1744170608.256572 mapread 0x009e00 thru 0x00fa18 (0x5c19 bytes) 000968 1744170608.260458 mapwrite 0x001d8c thru 0x00636e (0x45e3 bytes) 000970 1744170608.303484 MIRROR_EXTEND to 12 mirrors 000971 1744170608.400638 mapread 0x01283f thru 0x0159e7 (0x31a9 bytes) 000972 1744170608.407950 MIRROR_RESYNC 12 mirrors 000974 1744170608.653699 mapwrite 0x0371ae thru 0x03f276 (0x80c9 bytes) 000975 1744170608.694992 mapwrite 0x0228e2 thru 0x0328c7 (0xffe6 bytes) 000977 1744170608.712124 trunc from 0x040000 to 0x02ea11 (0x115ee bytes) 000978 1744170608.716499 mapread 0x0181d0 thru 0x01af41 (0x2d72 bytes) 000979 1744170608.718621 trunc from 0x02ea12 to 0x014831 (0x1a1e0 bytes) 000981 1744170608.723250 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) 000982 1744170608.770123 mapwrite 0x00fd39 thru 0x015f7f (0x6247 bytes) 000983 1744170608.792212 mapwrite 0x033ea6 thru 0x03b2d2 (0x742d bytes) 000984 1744170608.810756 read 0x0323eb thru 0x03ae12 (0x8a28 bytes) 000986 1744170608.811540 read 0x02042e thru 0x027c4e (0x7821 bytes) 000987 1744170608.815347 MIRROR_EXTEND to 12 mirrors 000989 1744170608.906440 mapread 0x0345b2 thru 0x0373de (0x2e2d bytes) 000990 1744170608.911487 mapread 0x019d30 thru 0x023891 (0x9b62 bytes) 000991 1744170608.916817 mapwrite 0x01aed4 thru 0x01e604 (0x3731 bytes) 000992 1744170608.956574 mapwrite 0x017549 thru 0x0221c2 (0xac7a bytes) 000993 1744170608.968311 trunc from 0x03b2d3 to 0x033718 (0x7bba bytes) 000994 1744170608.975509 mapread 0x012113 thru 0x01605f (0x3f4d bytes) 000995 1744170608.978157 mapread 0x017acf thru 0x0247de (0xcd10 bytes) 000996 1744170608.980295 write 0x0160fb thru 0x01bedd (0x5de3 bytes) 000998 1744170608.981120 write 0x024754 thru 0x024f30 (0x7dd bytes) 001000 1744170608.983110 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa0b:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 23:50:10 (1744170610) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 23:50:10 (1744170610) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7008256 .. Extending file size to 7116224 .. Extending file size to 8146688 .. Extending file size to 8502336 .. done Extending file size to 8185728 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8308192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8950656 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8941024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8954176 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9281888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9028128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 30640 30641 30643 30645 30647 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:51:14 (1744170674) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7111136 .. Extending file size to 7005824 .. Extending file size to 8634912 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7990816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8998208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8228160 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8889920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9204256 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9132672 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9140288 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9398016 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 2453 2454 2456 2457 2460 failed 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:52:17 (1744170737) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0288198 s, 72.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0725568 s, 28.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:52:21 (1744170741) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0228797 s, 91.7 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 23:52:22 (1744170742) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa57:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xdeb:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:52:24 (1744170744) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.02268 s, 92.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xded:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:52:25 (1744170745) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0213087 s, 98.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdeb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdec:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdeb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdec:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdef:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:52:27 (1744170747) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0213989 s, 98.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xded:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xded:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf1:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:52:49 (1744170769) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191311 s, 54.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdee:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdef:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:53:34 (1744170814) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0256991 s, 81.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:54:14 (1744170854) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. Creating new pool test_204f oleg452-server: Pool lustre.test_204f created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.024515 s, 85.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg452-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:55:05 (1744170905) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:55:07 (1744170907) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:55:09 (1744170909) oleg452-server: Pool lustre.test_206 created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg452-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:55:20 (1744170920) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176614 s, 59.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00944177 s, 111 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x4c9ca4af Mirror 3: 0x4c9ca4af PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:55:22 (1744170922) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:55:23 (1744170923) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:55:24 (1744170924) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.014702 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25e0000 Read this (4 bytes): BBBB 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:55:37 (1744170937) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00282278 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:55:39 (1744170939) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780131 s, 134 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa7c lmm_fid: [0x200000404:0xa7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa7c lmm_fid: [0x200000404:0xa7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdff:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836604 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 23:55:39.000000000 -0400 Modify: 2025-04-08 23:55:39.000000000 -0400 Change: 2025-04-08 23:55:39.000000000 -0400 Birth: - PASS 210a (1s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:55:40 (1744170940) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:55:41 (1744170941) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.102105 s, 401 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0116517 s, 352 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 8 23:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 8 23:55 /mnt/lustre/f211.sanity-flr PASS 211 (1s) == sanity-flr test complete, duration 675 sec ============ 23:55:43 (1744170943) === sanity-flr: start cleanup 23:55:43 (1744170943) === === sanity-flr: finish cleanup 23:55:44 (1744170944) ===