-----============= acceptance-small: sanity-flr ============----- Tue Dec 24 22:54:24 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg434-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:54:29 (1735098869) === oleg434-client.virtnet: executing check_config_client /mnt/lustre oleg434-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg434-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6cb4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6cb4800.idle_timeout=debug disable quota as required oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:54:37 (1735098877) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7008] == sanity-flr test 0a: lfs mirror create with -N option == 22:54:37 (1735098877) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:54:42 (1735098882) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:54:43 (1735098883) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:54:44 (1735098884) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:54:49 (1735098889) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:54:49 (1735098889) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 22:54:50 (1735098890) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6cb4800.stats=clear osc.lustre-OST0001-osc-ffff8800b6cb4800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:54:52 (1735098892) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:54:55 (1735098895) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:54:58 (1735098898) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:54:59 (1735098899) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 22:55:02 (1735098902) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:55:04 (1735098904) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:55:05 (1735098905) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:55:07 (1735098907) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:55:10 (1735098910) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:55:12 (1735098912) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.771104 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.124667 s, 8.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:55:16 (1735098916) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.027697 s, 114 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:55:19 (1735098919) Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:55:45 (1735098945) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.188162 s, 50.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:56:01 (1735098961) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:56:28 (1735098988) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.793216 s, 66.1 MB/s -rw-r--r-- 1 root root 52428800 Dec 24 22:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.309s user 0m0.002s sys 0m0.124s Stopping /mnt/lustre-ost1 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.217s user 0m0.000s sys 0m0.146s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg434-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg434-server oleg434-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.322s user 0m0.000s sys 0m0.133s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg434-server: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg434-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg434-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cb4800.ost_server_uuid in FULL state after 0 sec PASS 33b (26s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:56:55 (1735099015) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:56:55 (1735099015) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:56:56 (1735099016) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 22:56:57 (1735099017) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 22:56:59 (1735099019) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.160164 s, 78.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:57:03 (1735099023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111043 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105105 s, 99.8 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:57:04 (1735099024) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:57:07 (1735099027) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000821987 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00053088 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00153515 s, 667 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0010407 s, 984 kB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 22:57:10 (1735099030) create test files with size 15749462 .. b34137a16e56a91aeb7b1747eb98619f - 5b8021d4fe131cf9a9a534f659c0fc2c - 2e0fa173a2597221269cc745088a9b96 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 22:57:15 (1735099035) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:57:19 (1735099039) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:57:20 (1735099040) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237967 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00876988 s, 120 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 22:57:22 (1735099042) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0283745 s, 73.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.089465 s, 46.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0526469 s, 39.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 22:57:25 (1735099045) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 22:57:26 (1735099046) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:57:27 (1735099047) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00973467 s, 108 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 22:57:29 (1735099049) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:57:30 (1735099050) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:57:32 (1735099052) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.071503 s, 147 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:57:55 (1735099075) llite.lustre-ffff8800aa304800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0655905 s, 160 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg434-client.virtnet /mnt/lustre (opts:) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a2153000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:57:58 (1735099078) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:58:05 (1735099085) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:58:12 (1735099092) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 22:58:13 (1735099093) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.067253 s, 46.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0472281 s, 66.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 50419af0581089849d7c7f8e59b27625 - mirror 1 checksum: 335766c9c77b539ed0bdee0aba67ada4 - mirror 2 checksum: 50419af0581089849d7c7f8e59b27625 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 22:58:15 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 22:58:15 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 50419af0581089849d7c7f8e59b27625 - mirror 1 checksum: 335766c9c77b539ed0bdee0aba67ada4 - mirror 2 checksum: 335766c9c77b539ed0bdee0aba67ada4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 22:58:15 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 22:58:15 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:58:16 (1735099096) Failing mds1 on oleg434-server Stopping /mnt/lustre-mds1 (opts:) on oleg434-server 22:58:23 (1735099103) shut down facet: mds1 facet_host: oleg434-server facet_failover_host: oleg434-server Failover mds1 to oleg434-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 22:58:36 (1735099116) targets are mounted 22:58:36 (1735099116) facet_failover done oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00831383 s, 126 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:58:45 (1735099125) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:58:45 (1735099125) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:58:46 (1735099126) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00282474 s, 5.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:58:48 (1735099128) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:58:49 (1735099129) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:58:50 (1735099130) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:58:54 (1735099134) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:59:11 (1735099151) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:59:28 (1735099168) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:59:45 (1735099185) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 22:59:47 (1735099187) mirror create pid 22482 mirror split pid 22483 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22482 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22483 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 23:00:48 (1735099248) Seed set to 9249 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3e3bb 000001 1735099249.450122 trunc from 00000000 to 0x03e3ba (0x3e3bb bytes) 000009 1735099249.458186 mapread 0x01d285 thru 0x021021 (0x3d9d bytes) 000010 1735099249.466694 mapread 0x034cf1 thru 0x03e3ba (0x96ca bytes) 000014 1735099249.474191 mapwrite 0x03e2d2 thru 0x03ffff (0x1d2e bytes) 000016 1735099249.514731 trunc from 0x040000 to 0x0028a8 (0x3d757 bytes) 000020 1735099249.520179 read 0x00023c thru 0x0028a8 (0x266d bytes) 000022 1735099249.523700 MIRROR_EXTEND to 2 mirrors 000025 1735099249.659912 mapread 0x0003ef thru 0x0028a8 (0x24ba bytes) 000026 1735099249.668231 write 0x0214b4 thru 0x02608b (0x4bd8 bytes) 000027 1735099249.698661 trunc from 0x02608c to 0x03a573 (0x144e8 bytes) 000028 1735099249.713576 write 0x01b318 thru 0x02b1ee (0xfed7 bytes) 000029 1735099249.719815 write 0x00d2ab thru 0x01a294 (0xcfea bytes) 000030 1735099249.725556 MIRROR_EXTEND to 3 mirrors 000031 1735099249.911884 mapwrite 0x029215 thru 0x030bd3 (0x79bf bytes) 000033 1735099249.987443 trunc from 0x03a574 to 0x039806 (0xd6d bytes) 000034 1735099250.006887 MIRROR_EXTEND to 4 mirrors 000035 1735099250.170683 MIRROR_EXTEND to 5 mirrors 000036 1735099250.339743 write 0x0096e1 thru 0x00eb40 (0x5460 bytes) 000037 1735099250.363261 mapwrite 0x01e43c thru 0x0278b1 (0x9476 bytes) 000039 1735099250.391807 read 0x02a8e8 thru 0x02ef19 (0x4632 bytes) 000041 1735099250.392552 trunc from 0x039807 to 0x000b58 (0x38cae bytes) 000042 1735099250.398834 write 0x01904b thru 0x028129 (0xf0df bytes) 000044 1735099250.402243 MIRROR_RESYNC 5 mirrors 000045 1735099250.654482 mapwrite 0x02d511 thru 0x03286c (0x535c bytes) 000047 1735099250.697180 mapread 0x0164a7 thru 0x016ffd (0xb57 bytes) 000049 1735099250.700705 mapwrite 0x02a3f6 thru 0x030c17 (0x6822 bytes) 000050 1735099250.719888 trunc from 0x03286d to 0x015fe6 (0x1c886 bytes) 000052 1735099250.726749 write 0x004e34 thru 0x008f6d (0x413a bytes) 000054 1735099250.731294 mapwrite 0x038305 thru 0x03ffff (0x7cfb bytes) 000055 1735099250.761769 mapread 0x00576f thru 0x00a177 (0x4a09 bytes) 000058 1735099250.763159 write 0x031483 thru 0x03f4a1 (0xe01f bytes) 000059 1735099250.764093 read 0x01fb3c thru 0x02156f (0x1a34 bytes) 000061 1735099250.768281 MIRROR_EXTEND to 6 mirrors 000062 1735099250.922599 mapwrite 0x03cc98 thru 0x03dd76 (0x10df bytes) 000064 1735099250.985926 MIRROR_RESYNC 6 mirrors 000065 1735099251.268734 mapread 0x0109cb thru 0x015ebc (0x54f2 bytes) 000068 1735099251.276070 read 0x008bab thru 0x00df15 (0x536b bytes) 000069 1735099251.279985 MIRROR_EXTEND to 7 mirrors 000074 1735099251.467792 read 0x0391fd thru 0x03ffff (0x6e03 bytes) 000075 1735099251.473109 MIRROR_RESYNC 7 mirrors 000076 1735099251.580809 trunc from 0x040000 to 0x02d820 (0x127df bytes) 000077 1735099251.607253 mapread 0x000f7c thru 0x00e20b (0xd290 bytes) 000078 1735099251.614762 mapread 0x002318 thru 0x01041c (0xe105 bytes) 000080 1735099251.615557 mapread 0x001af1 thru 0x00e404 (0xc914 bytes) 000082 1735099251.616319 mapread 0x018490 thru 0x02157a (0x90eb bytes) 000085 1735099251.623381 read 0x02b0ca thru 0x02d820 (0x2757 bytes) 000086 1735099251.625395 mapread 0x026a4e thru 0x02c3e2 (0x5995 bytes) 000088 1735099251.627652 mapread 0x00fb5e thru 0x019977 (0x9e1a bytes) 000091 1735099251.630009 read 0x027a8a thru 0x02d820 (0x5d97 bytes) 000094 1735099251.630822 read 0x01f8d8 thru 0x02d58c (0xdcb5 bytes) 000095 1735099251.631181 read 0x00bcf1 thru 0x00ec96 (0x2fa6 bytes) 000096 1735099251.631471 read 0x00a017 thru 0x00f87b (0x5865 bytes) 000097 1735099251.631818 write 0x039aff thru 0x03c57c (0x2a7e bytes) 000098 1735099251.632502 read 0x00e9bc thru 0x018e43 (0xa488 bytes) 000099 1735099251.632907 mapread 0x00aab0 thru 0x016fea (0xc53b bytes) 000100 1735099251.633450 read 0x01e5d8 thru 0x027823 (0x924c bytes) 000102 1735099251.634121 mapread 0x01c0f5 thru 0x0290b7 (0xcfc3 bytes) 000103 1735099251.634628 mapread 0x01e17b thru 0x02378f (0x5615 bytes) 000104 1735099251.635042 mapwrite 0x03e7fc thru 0x03ffff (0x1804 bytes) 000107 1735099251.663025 read 0x01c109 thru 0x028bd5 (0xcacd bytes) 000108 1735099251.663565 mapread 0x009689 thru 0x00c1a1 (0x2b19 bytes) 000113 1735099251.665539 write 0x03ebcf thru 0x03ffff (0x1431 bytes) 000115 1735099251.666425 mapwrite 0x02bb27 thru 0x02f5d4 (0x3aae bytes) 000116 1735099251.689500 mapwrite 0x01ace0 thru 0x01e867 (0x3b88 bytes) 000117 1735099251.711280 MIRROR_RESYNC 7 mirrors 000119 1735099251.976481 read 0x02f806 thru 0x032be9 (0x33e4 bytes) 000120 1735099251.979029 mapread 0x015d82 thru 0x017b49 (0x1dc8 bytes) 000124 1735099251.987194 write 0x015e38 thru 0x0224a3 (0xc66c bytes) 000126 1735099252.014430 mapread 0x005079 thru 0x005f92 (0xf1a bytes) 000128 1735099252.019498 mapwrite 0x006847 thru 0x013f44 (0xd6fe bytes) 000130 1735099252.050657 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000132 1735099252.121645 read 0x03045a thru 0x03052e (0xd5 bytes) 000133 1735099252.124056 write 0x01ab2d thru 0x01e084 (0x3558 bytes) 000134 1735099252.128423 read 0x037a66 thru 0x03e1b3 (0x674e bytes) 000136 1735099252.134137 mapread 0x0181c0 thru 0x0207e2 (0x8623 bytes) 000138 1735099252.135018 read 0x035052 thru 0x037439 (0x23e8 bytes) 000142 1735099252.138185 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000143 1735099252.207893 mapwrite 0x015a83 thru 0x0174f5 (0x1a73 bytes) 000144 1735099252.236604 trunc from 0x040000 to 0x01e09d (0x21f62 bytes) truncating to largest ever: 0x3f3b9 000146 1735099252.248005 trunc from 0x01e09e to 0x03f3b8 (0x2131b bytes) 000147 1735099252.253987 mapread 0x02172f thru 0x02dae6 (0xc3b8 bytes) 000149 1735099252.260332 mapwrite 0x00af89 thru 0x013033 (0x80ab bytes) 000151 1735099252.285405 write 0x0392bc thru 0x03ffff (0x6d44 bytes) 000152 1735099252.289306 trunc from 0x040000 to 0x0018ac (0x3e753 bytes) 000153 1735099252.295386 read 0x00076c thru 0x0018ac (0x1141 bytes) 000154 1735099252.297834 mapwrite 0x026784 thru 0x03074f (0x9fcc bytes) 000155 1735099252.338990 trunc from 0x030750 to 0x035b4a (0x53fb bytes) 000157 1735099252.346494 read 0x02f1ec thru 0x035161 (0x5f76 bytes) 000158 1735099252.351223 write 0x032565 thru 0x03701c (0x4ab8 bytes) 000159 1735099252.352022 write 0x03cddc thru 0x03ffff (0x3224 bytes) 000160 1735099252.352622 write 0x0064d9 thru 0x0107e5 (0xa30d bytes) 000163 1735099252.353877 mapread 0x03c807 thru 0x03ffff (0x37f9 bytes) 000165 1735099252.354624 mapwrite 0x002030 thru 0x00664e (0x461f bytes) 000167 1735099252.380686 read 0x0183b9 thru 0x0196a9 (0x12f1 bytes) 000171 1735099252.384337 MIRROR_RESYNC 5 mirrors 000172 1735099252.596161 write 0x01f3d7 thru 0x022741 (0x336b bytes) 000173 1735099252.614129 mapread 0x02275e thru 0x02e67b (0xbf1e bytes) 000174 1735099252.619770 trunc from 0x040000 to 0x00d138 (0x32ec7 bytes) 000176 1735099252.631361 write 0x011bd3 thru 0x013425 (0x1853 bytes) 000178 1735099252.632465 trunc from 0x013426 to 0x012ead (0x578 bytes) 000179 1735099252.637601 mapread 0x00d420 thru 0x012ead (0x5a8e bytes) 000180 1735099252.644045 MIRROR_EXTEND to 6 mirrors 000183 1735099252.766898 mapread 0x00f40e thru 0x012ead (0x3aa0 bytes) 000184 1735099252.774788 read 0x0043f6 thru 0x012ead (0xeab8 bytes) 000188 1735099252.782065 read 0x00b21b thru 0x00c88d (0x1673 bytes) 000189 1735099252.783357 trunc from 0x012eae to 0x022601 (0xf754 bytes) 000190 1735099252.805651 mapwrite 0x02e6da thru 0x031836 (0x315d bytes) 000191 1735099252.836449 write 0x00e82a thru 0x01a7ef (0xbfc6 bytes) 000193 1735099252.837844 write 0x03eaaf thru 0x03ffff (0x1551 bytes) 000194 1735099252.838475 mapread 0x021ac8 thru 0x02fec7 (0xe400 bytes) 000196 1735099252.839456 mapwrite 0x03ab3c thru 0x03ffff (0x54c4 bytes) 000199 1735099252.865492 mapread 0x0054c8 thru 0x00e05a (0x8b93 bytes) 000200 1735099252.866214 mapwrite 0x035f8a thru 0x037703 (0x177a bytes) 000202 1735099252.889625 mapwrite 0x01bc7c thru 0x02459b (0x8920 bytes) 000203 1735099252.912670 mapread 0x01fbde thru 0x02d7e9 (0xdc0c bytes) 000209 1735099252.914966 write 0x008588 thru 0x00cf5f (0x49d8 bytes) 000212 1735099252.916294 read 0x024088 thru 0x027ebe (0x3e37 bytes) 000215 1735099252.919107 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000216 1735099252.980459 trunc from 0x040000 to 0x00c29b (0x33d64 bytes) 000217 1735099252.985858 read 0x009d2f thru 0x00c29b (0x256d bytes) 000220 1735099252.988782 read 0x008f2e thru 0x00c29b (0x336e bytes) 000222 1735099252.992658 write 0x02c1a5 thru 0x02f507 (0x3363 bytes) 000225 1735099252.994237 write 0x02ddf9 thru 0x02f916 (0x1b1e bytes) 000228 1735099252.995525 write 0x01f847 thru 0x022325 (0x2adf bytes) 000229 1735099253.000194 trunc from 0x02f917 to 0x03d6dd (0xddc7 bytes) 000230 1735099253.004905 write 0x025267 thru 0x025366 (0x100 bytes) 000232 1735099253.005856 mapwrite 0x02ffca thru 0x035ca6 (0x5cdd bytes) 000234 1735099253.035276 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000236 1735099253.104633 mapwrite 0x00c99e thru 0x011b46 (0x51a9 bytes) 000239 1735099253.133451 trunc from 0x03d6de to 0x002d27 (0x3a9b6 bytes) 000240 1735099253.140264 trunc from 0x002d28 to 0x00b033 (0x830c bytes) 000241 1735099253.145397 trunc from 0x00b034 to 0x02b6cb (0x20698 bytes) 000242 1735099253.150961 mapread 0x01dec9 thru 0x020b51 (0x2c89 bytes) 000244 1735099253.156825 MIRROR_EXTEND to 5 mirrors 000245 1735099253.277119 trunc from 0x02b6cc to 0x00fb7d (0x1bb4e bytes) 000247 1735099253.305690 MIRROR_RESYNC 5 mirrors 000248 1735099253.480098 mapwrite 0x012ca0 thru 0x01a66c (0x79cd bytes) 000250 1735099253.532137 read 0x00c3cc thru 0x00d80d (0x1442 bytes) 000251 1735099253.534979 mapread 0x00002c thru 0x00fb0b (0xfae0 bytes) 000253 1735099253.542437 trunc from 0x01a66d to 0x00e266 (0xc406 bytes) 000254 1735099253.550821 read 0x0093a5 thru 0x00e266 (0x4ec2 bytes) 000256 1735099253.551650 write 0x02b840 thru 0x038e03 (0xd5c4 bytes) 000257 1735099253.553036 write 0x03c2dd thru 0x03ebeb (0x290f bytes) 000258 1735099253.554316 mapwrite 0x01abb5 thru 0x02355c (0x89a8 bytes) 000259 1735099253.581166 mapwrite 0x01b038 thru 0x01b5dd (0x5a6 bytes) 000260 1735099253.597236 read 0x038949 thru 0x03d5d2 (0x4c8a bytes) 000261 1735099253.597719 mapread 0x003cd3 thru 0x00e5ce (0xa8fc bytes) 000263 1735099253.598771 mapread 0x0281ea thru 0x028d09 (0xb20 bytes) 000265 1735099253.599775 mapwrite 0x00d064 thru 0x012b9e (0x5b3b bytes) 000267 1735099253.619060 mapwrite 0x024fa0 thru 0x02dc73 (0x8cd4 bytes) 000268 1735099253.635025 mapread 0x030dd5 thru 0x0369a5 (0x5bd1 bytes) 000269 1735099253.635591 write 0x001235 thru 0x00a43a (0x9206 bytes) 000271 1735099253.636707 mapread 0x02cc40 thru 0x02eb2e (0x1eef bytes) 000272 1735099253.637296 mapwrite 0x018be5 thru 0x02220e (0x962a bytes) 000273 1735099253.656509 trunc from 0x03ebec to 0x003b5a (0x3b091 bytes) 000274 1735099253.662093 write 0x035402 thru 0x03ffff (0xabfe bytes) 000276 1735099253.663544 trunc from 0x040000 to 0x032a69 (0xd596 bytes) truncating to largest ever: 0x3fedf 000278 1735099253.669935 trunc from 0x032a6a to 0x03fede (0xd475 bytes) 000280 1735099253.675881 mapread 0x01c281 thru 0x024774 (0x84f4 bytes) 000282 1735099253.682855 mapwrite 0x00b413 thru 0x00c9fa (0x15e8 bytes) 000283 1735099253.702565 write 0x02cbe5 thru 0x02f4b6 (0x28d2 bytes) 000284 1735099253.703186 write 0x00f3f6 thru 0x0174be (0x80c9 bytes) 000286 1735099253.704171 trunc from 0x03fedf to 0x000239 (0x3fca5 bytes) 000287 1735099253.709901 trunc from 0x00023a to 0x014811 (0x145d8 bytes) 000288 1735099253.714013 read 0x00519f thru 0x0138b6 (0xe718 bytes) 000292 1735099253.718034 read 0x002286 thru 0x003129 (0xea4 bytes) 000293 1735099253.718439 mapwrite 0x02f2a7 thru 0x030603 (0x135d bytes) 000296 1735099253.740909 write 0x01e110 thru 0x01e268 (0x159 bytes) 000297 1735099253.741527 trunc from 0x030604 to 0x0376cf (0x70cc bytes) 000298 1735099253.746440 read 0x017800 thru 0x02572e (0xdf2f bytes) 000299 1735099253.747041 mapwrite 0x03fa1e thru 0x03ffff (0x5e2 bytes) 000302 1735099253.771364 trunc from 0x040000 to 0x039b36 (0x64c9 bytes) 000304 1735099253.777422 MIRROR_EXTEND to 6 mirrors 000306 1735099253.893264 mapread 0x033d22 thru 0x03704b (0x332a bytes) 000307 1735099253.900743 mapwrite 0x00a99d thru 0x01244e (0x7ab2 bytes) 000308 1735099253.956026 read 0x010461 thru 0x01e87a (0xe41a bytes) 000309 1735099253.959113 trunc from 0x039b37 to 0x01d9ad (0x1c189 bytes) 000313 1735099253.969581 read 0x010cf6 thru 0x01d9ad (0xccb8 bytes) 000314 1735099253.971337 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000315 1735099254.027066 mapwrite 0x01b65d thru 0x020296 (0x4c3a bytes) 000316 1735099254.053026 read 0x0031ff thru 0x00ea43 (0xb845 bytes) 000317 1735099254.055764 read 0x00fd4a thru 0x0118ef (0x1ba6 bytes) 000319 1735099254.058273 mapwrite 0x03a7d5 thru 0x03ffff (0x582b bytes) 000320 1735099254.080990 mapread 0x0059d7 thru 0x01370b (0xdd35 bytes) 000321 1735099254.081660 trunc from 0x040000 to 0x02727c (0x18d83 bytes) 000323 1735099254.086521 trunc from 0x02727d to 0x035e9e (0xec22 bytes) 000326 1735099254.091207 mapwrite 0x03adc6 thru 0x03e2e9 (0x3524 bytes) 000327 1735099254.118580 MIRROR_RESYNC 5 mirrors 000328 1735099254.316444 mapwrite 0x00ee1a thru 0x016c64 (0x7e4b bytes) 000330 1735099254.364911 read 0x0115a1 thru 0x0135eb (0x204b bytes) 000332 1735099254.365533 trunc from 0x03e2ea to 0x026efd (0x173ec bytes) 000333 1735099254.376979 read 0x023b11 thru 0x026efd (0x33ed bytes) 000334 1735099254.380451 mapwrite 0x0166b3 thru 0x0177f1 (0x113f bytes) 000335 1735099254.398402 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000336 1735099254.452744 trunc from 0x026efe to 0x02235e (0x4b9f bytes) 000337 1735099254.459991 MIRROR_EXTEND to 5 mirrors 000338 1735099254.569342 MIRROR_EXTEND to 6 mirrors 000342 1735099254.672387 mapwrite 0x02bb8c thru 0x031c35 (0x60aa bytes) 000343 1735099254.716057 read 0x0106ee thru 0x019480 (0x8d93 bytes) 000346 1735099254.721214 MIRROR_EXTEND to 7 mirrors 000349 1735099254.840693 mapread 0x00c4e3 thru 0x0137a0 (0x72be bytes) 000351 1735099254.849861 mapread 0x0013ec thru 0x001b74 (0x789 bytes) 000352 1735099254.854508 write 0x00db5a thru 0x0104ca (0x2971 bytes) 000354 1735099254.872823 mapread 0x022bcf thru 0x0273d4 (0x4806 bytes) 000356 1735099254.876402 write 0x02a841 thru 0x036538 (0xbcf8 bytes) 000357 1735099254.877289 write 0x0011a8 thru 0x006242 (0x509b bytes) 000358 1735099254.880016 trunc from 0x036539 to 0x02788f (0xeca9 bytes) 000359 1735099254.891164 write 0x00f2e9 thru 0x010166 (0xe7e bytes) 000361 1735099254.892133 trunc from 0x027890 to 0x01a210 (0xd67f bytes) 000362 1735099254.896635 trunc from 0x01a211 to 0x0146ea (0x5b26 bytes) 000363 1735099254.901526 write 0x005634 thru 0x007501 (0x1ece bytes) 000364 1735099254.902287 read 0x00466a thru 0x0075d8 (0x2f6f bytes) 000365 1735099254.902693 mapwrite 0x025e45 thru 0x033fbf (0xe17b bytes) 000366 1735099254.930070 read 0x022963 thru 0x02cc0a (0xa2a8 bytes) 000367 1735099254.930496 mapread 0x003ecf thru 0x00492f (0xa61 bytes) 000368 1735099254.930912 read 0x019456 thru 0x023f23 (0xaace bytes) 000369 1735099254.936283 MIRROR_EXTEND to 8 mirrors 000370 1735099255.062845 trunc from 0x033fc0 to 0x02f34e (0x4c71 bytes) 000371 1735099255.083072 read 0x02c3a7 thru 0x02d78e (0x13e8 bytes) 000374 1735099255.086333 mapread 0x00f45e thru 0x01eabe (0xf661 bytes) 000375 1735099255.094809 write 0x01c42c thru 0x0221ec (0x5dc1 bytes) 000376 1735099255.095480 mapwrite 0x03d370 thru 0x03ffff (0x2c90 bytes) 000378 1735099255.122336 write 0x0098eb thru 0x0121ff (0x8915 bytes) 000379 1735099255.125206 trunc from 0x040000 to 0x02f6b5 (0x1094a bytes) 000380 1735099255.129913 read 0x017f42 thru 0x01d5d0 (0x568f bytes) 000381 1735099255.130398 mapread 0x01bb4d thru 0x01f64a (0x3afe bytes) 000384 1735099255.131627 mapwrite 0x0331c9 thru 0x03691d (0x3755 bytes) 000385 1735099255.158443 trunc from 0x03691e to 0x00d9d4 (0x28f49 bytes) 000387 1735099255.164420 mapwrite 0x006c26 thru 0x01151e (0xa8f9 bytes) 000390 1735099255.194384 write 0x00f6fd thru 0x01d3a9 (0xdcad bytes) 000391 1735099255.195332 mapread 0x00f310 thru 0x017b7c (0x886d bytes) 000392 1735099255.195892 read 0x006f56 thru 0x00a545 (0x35f0 bytes) 000394 1735099255.196547 mapwrite 0x007605 thru 0x010445 (0x8e41 bytes) 000400 1735099255.218507 write 0x02fd57 thru 0x038cb7 (0x8f61 bytes) 000401 1735099255.219514 write 0x038cbe thru 0x03bc98 (0x2fdb bytes) 000402 1735099255.220009 read 0x0043fb thru 0x00d500 (0x9106 bytes) 000404 1735099255.220566 trunc from 0x03bc99 to 0x01e969 (0x1d32f bytes) 000408 1735099255.228774 MIRROR_RESYNC 8 mirrors 000414 1735099255.486678 trunc from 0x01e96a to 0x012ca3 (0xbcc6 bytes) 000416 1735099255.511283 trunc from 0x012ca4 to 0x012116 (0xb8d bytes) 000417 1735099255.517248 read 0x006024 thru 0x00cf7d (0x6f5a bytes) 000419 1735099255.520709 write 0x01838f thru 0x01db55 (0x57c7 bytes) 000420 1735099255.521524 write 0x014a3c thru 0x0231cc (0xe791 bytes) 000421 1735099255.524060 read 0x013115 thru 0x0217bf (0xe6ab bytes) 000422 1735099255.529862 MIRROR_RESYNC 8 mirrors 000425 1735099255.800584 mapread 0x01a3b5 thru 0x0231cc (0x8e18 bytes) 000426 1735099255.805621 mapread 0x0148d8 thru 0x016ee2 (0x260b bytes) 000427 1735099255.809466 MIRROR_RESYNC 8 mirrors 000429 1735099255.874282 write 0x0044cb thru 0x00d5e6 (0x911c bytes) 000433 1735099255.894291 trunc from 0x0231cd to 0x01239b (0x10e31 bytes) 000434 1735099255.907706 MIRROR_EXTEND to 9 mirrors 000435 1735099256.046557 trunc from 0x01239c to 0x02f4fa (0x1d15f bytes) 000436 1735099256.071615 read 0x00208f thru 0x00a596 (0x8508 bytes) 000437 1735099256.075251 write 0x029674 thru 0x02e000 (0x498d bytes) 000438 1735099256.079147 mapwrite 0x03b83a thru 0x03ffff (0x47c6 bytes) 000441 1735099256.109578 trunc from 0x040000 to 0x00213d (0x3dec2 bytes) 000443 1735099256.116608 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000445 1735099256.171657 read 0x00115f thru 0x001fc0 (0xe62 bytes) 000448 1735099256.174555 trunc from 0x00213e to 0x03faf9 (0x3d9bc bytes) 000455 1735099256.182138 read 0x00ca35 thru 0x016a24 (0x9ff0 bytes) 000458 1735099256.188524 mapwrite 0x021721 thru 0x028d6b (0x764b bytes) 000459 1735099256.215090 trunc from 0x03fafa to 0x019ad8 (0x26021 bytes) 000460 1735099256.220356 mapread 0x017f79 thru 0x019ad8 (0x1b60 bytes) 000462 1735099256.223113 MIRROR_EXTEND to 9 mirrors 000463 1735099256.354448 MIRROR_RESYNC 9 mirrors 000465 1735099256.632813 mapread 0x010663 thru 0x019ad8 (0x9476 bytes) 000467 1735099256.639669 mapwrite 0x0378f9 thru 0x039798 (0x1ea0 bytes) 000468 1735099256.683503 write 0x0244f8 thru 0x032f41 (0xea4a bytes) 000469 1735099256.689101 MIRROR_EXTEND to 10 mirrors 000470 1735099256.831034 read 0x0119a9 thru 0x01bc38 (0xa290 bytes) 000471 1735099256.835020 mapwrite 0x0110f9 thru 0x014566 (0x346e bytes) 000473 1735099256.889028 read 0x0139ad thru 0x02307b (0xf6cf bytes) 000474 1735099256.892566 read 0x031b4a thru 0x039798 (0x7c4f bytes) 000477 1735099256.893517 mapwrite 0x01eba6 thru 0x0297a1 (0xabfc bytes) 000478 1735099256.908901 write 0x01bb3a thru 0x02949a (0xd961 bytes) 000480 1735099256.910156 mapwrite 0x01583e thru 0x01ec5b (0x941e bytes) 000481 1735099256.927155 trunc from 0x039799 to 0x001182 (0x38616 bytes) 000482 1735099256.936843 mapwrite 0x008090 thru 0x00e8ce (0x683f bytes) 000483 1735099256.960900 MIRROR_EXTEND to 11 mirrors 000485 1735099257.087889 read 0x007229 thru 0x00e8ce (0x76a6 bytes) 000486 1735099257.091064 mapread 0x005e61 thru 0x00c259 (0x63f9 bytes) 000487 1735099257.094487 read 0x00bd31 thru 0x00e8ce (0x2b9e bytes) 000489 1735099257.095011 write 0x014424 thru 0x01de3b (0x9a18 bytes) 000490 1735099257.114190 mapwrite 0x031b4b thru 0x033ba0 (0x2056 bytes) 000493 1735099257.144796 mapwrite 0x00b7c7 thru 0x00fa51 (0x428b bytes) 000494 1735099257.163654 mapread 0x011b05 thru 0x01c76d (0xac69 bytes) 000495 1735099257.164294 mapwrite 0x019c93 thru 0x023793 (0x9b01 bytes) 000497 1735099257.185878 trunc from 0x033ba1 to 0x000e96 (0x32d0a bytes) 000500 1735099257.191585 write 0x028b61 thru 0x02e2fc (0x579c bytes) 000501 1735099257.192465 mapwrite 0x004d99 thru 0x014a74 (0xfcdc bytes) 000502 1735099257.233554 MIRROR_EXTEND to 12 mirrors 000503 1735099257.347763 mapwrite 0x0138f3 thru 0x01ecd6 (0xb3e4 bytes) 000505 1735099257.402579 trunc from 0x02e2fd to 0x02acbf (0x363d bytes) 000507 1735099257.412499 MIRROR_EXTEND to 13 mirrors 000508 1735099257.523433 mapwrite 0x02fa07 thru 0x03e012 (0xe60c bytes) 000509 1735099257.568796 read 0x01ead7 thru 0x02aa14 (0xbf3e bytes) 000510 1735099257.572972 MIRROR_EXTEND to 14 mirrors 000511 1735099257.693690 mapwrite 0x0358e1 thru 0x039bf8 (0x4318 bytes) 000512 1735099257.742364 mapread 0x02db27 thru 0x03bb4b (0xe025 bytes) 000514 1735099257.745547 write 0x02cfbe thru 0x02e6ff (0x1742 bytes) 000515 1735099257.746188 read 0x033d06 thru 0x03e012 (0xa30d bytes) 000517 1735099257.748471 MIRROR_RESYNC 14 mirrors 000519 1735099258.218285 mapread 0x002cac thru 0x00c7e6 (0x9b3b bytes) 000521 1735099258.225562 mapread 0x02a01e thru 0x033ea6 (0x9e89 bytes) 000522 1735099258.231283 write 0x023ffb thru 0x02d4ad (0x94b3 bytes) 000525 1735099258.252019 write 0x029af8 thru 0x02a803 (0xd0c bytes) 000527 1735099258.252623 trunc from 0x03e013 to 0x01d845 (0x207cd bytes) 000530 1735099258.258146 trunc from 0x01d846 to 0x038c67 (0x1b422 bytes) 000531 1735099258.261760 trunc from 0x038c68 to 0x022c98 (0x15fcf bytes) 000533 1735099258.267113 trunc from 0x022c99 to 0x0127af (0x104e9 bytes) 000534 1735099258.272117 trunc from 0x0127b0 to 0x038199 (0x259ea bytes) 000536 1735099258.276348 read 0x0233d8 thru 0x02f251 (0xbe7a bytes) 000537 1735099258.279124 mapwrite 0x017b6f thru 0x01c9a5 (0x4e37 bytes) 000538 1735099258.295989 mapread 0x00a2f0 thru 0x015dd2 (0xbae3 bytes) 000541 1735099258.299457 trunc from 0x03819a to 0x02bce1 (0xc4b8 bytes) 000543 1735099258.303872 mapread 0x007604 thru 0x012bac (0xb5a9 bytes) 000545 1735099258.307174 mapread 0x02b0ed thru 0x02bce1 (0xbf5 bytes) 000546 1735099258.307594 write 0x000022 thru 0x008ab4 (0x8a93 bytes) 000547 1735099258.309684 write 0x00e3de thru 0x019aae (0xb6d1 bytes) 000548 1735099258.310516 mapwrite 0x02d576 thru 0x03198c (0x4417 bytes) 000549 1735099258.332598 trunc from 0x03198d to 0x0137b1 (0x1e1db bytes) 000551 1735099258.336472 mapwrite 0x010a8a thru 0x01762f (0x6ba6 bytes) 000552 1735099258.355413 trunc from 0x017630 to 0x015cf3 (0x193c bytes) 000555 1735099258.359561 mapwrite 0x01b8bc thru 0x01e230 (0x2975 bytes) 000556 1735099258.378828 write 0x02427a thru 0x03388f (0xf616 bytes) 000558 1735099258.380185 trunc from 0x033890 to 0x02a52b (0x9364 bytes) 000559 1735099258.384004 trunc from 0x02a52c to 0x00cf7c (0x1d5af bytes) 000560 1735099258.388380 mapwrite 0x02b50a thru 0x030581 (0x5078 bytes) 000562 1735099258.409750 read 0x00ca6a thru 0x01c87e (0xfe15 bytes) 000565 1735099258.413248 read 0x02bcd2 thru 0x030581 (0x48b0 bytes) 000568 1735099258.414025 mapread 0x02bde0 thru 0x02f69b (0x38bc bytes) 000569 1735099258.415919 MIRROR_RESYNC 14 mirrors 000571 1735099258.810671 read 0x02b2d4 thru 0x030581 (0x52ae bytes) 000572 1735099258.815118 trunc from 0x030582 to 0x029d72 (0x680f bytes) 000574 1735099258.841573 mapread 0x01c1fd thru 0x029d72 (0xdb76 bytes) 000576 1735099258.850192 mapwrite 0x02ac7c thru 0x02ce1d (0x21a2 bytes) 000578 1735099258.874054 MIRROR_EXTEND to 15 mirrors 000580 1735099259.011063 mapwrite 0x011cd1 thru 0x01636c (0x469c bytes) 000582 1735099259.078666 mapwrite 0x03e147 thru 0x03e5b9 (0x473 bytes) 000583 1735099259.114290 mapread 0x02293a thru 0x022b90 (0x257 bytes) 000584 1735099259.117434 write 0x01f696 thru 0x02ec19 (0xf584 bytes) 000586 1735099259.121343 MIRROR_EXTEND to 16 mirrors 000587 1735099259.281526 read 0x037d8a thru 0x03e5b9 (0x6830 bytes) 000588 1735099259.285786 read 0x0159f6 thru 0x0205c6 (0xabd1 bytes) 000590 1735099259.292051 write 0x01911e thru 0x0258a1 (0xc784 bytes) 000592 1735099259.314527 mapread 0x000a62 thru 0x0073fa (0x6999 bytes) 000593 1735099259.326235 trunc from 0x03e5ba to 0x02b68a (0x12f2f bytes) 000594 1735099259.338279 read 0x00ce3c thru 0x01c3ae (0xf573 bytes) 000597 1735099259.343753 trunc from 0x02b68b to 0x00f516 (0x1c174 bytes) 000598 1735099259.349133 mapwrite 0x01db3b thru 0x0294e5 (0xb9ab bytes) 000604 1735099259.390743 read 0x002b8c thru 0x010bd6 (0xe04b bytes) 000609 1735099259.395139 mapread 0x001ce8 thru 0x0119cf (0xfce8 bytes) 000611 1735099259.396066 read 0x01fd06 thru 0x0283df (0x86da bytes) 000612 1735099259.396403 trunc from 0x0294e6 to 0x0353a1 (0xbebc bytes) 000614 1735099259.401455 mapread 0x0192d0 thru 0x01b7cd (0x24fe bytes) 000618 1735099259.402857 trunc from 0x0353a2 to 0x02f2f8 (0x60a9 bytes) 000619 1735099259.406847 mapwrite 0x0034f5 thru 0x0121ec (0xecf8 bytes) 000621 1735099259.426761 trunc from 0x02f2f9 to 0x003806 (0x2baf2 bytes) 000622 1735099259.431689 read 0x002996 thru 0x003806 (0xe71 bytes) 000623 1735099259.432005 write 0x03d618 thru 0x03ffff (0x29e8 bytes) 000625 1735099259.433059 write 0x004348 thru 0x006104 (0x1dbd bytes) 000626 1735099259.436425 mapwrite 0x00314a thru 0x006262 (0x3119 bytes) 000627 1735099259.457345 read 0x030362 thru 0x032a56 (0x26f5 bytes) 000629 1735099259.458096 write 0x03cf19 thru 0x03ffff (0x30e7 bytes) 000630 1735099259.458800 mapread 0x005fdd thru 0x00e3c5 (0x83e9 bytes) 000632 1735099259.459592 write 0x02de4e thru 0x031e9b (0x404e bytes) 000635 1735099259.460818 trunc from 0x040000 to 0x00e419 (0x31be6 bytes) 000637 1735099259.467271 read 0x00c45b thru 0x00df98 (0x1b3e bytes) 000638 1735099259.467694 mapread 0x00a0f2 thru 0x00e419 (0x4328 bytes) 000641 1735099259.469034 mapwrite 0x0357d1 thru 0x03ffff (0xa82f bytes) 000642 1735099259.501146 write 0x03365c thru 0x03ffff (0xc9a4 bytes) 000643 1735099259.501908 trunc from 0x040000 to 0x00907e (0x36f81 bytes) 000644 1735099259.507536 mapread 0x006c36 thru 0x008ab2 (0x1e7d bytes) 000646 1735099259.508467 mapwrite 0x00caed thru 0x014a1b (0x7f2f bytes) 000647 1735099259.546832 MIRROR_RESYNC 16 mirrors 000649 1735099260.080702 trunc from 0x014a1c to 0x00f624 (0x53f7 bytes) 000650 1735099260.109301 mapread 0x006cff thru 0x00f624 (0x8926 bytes) 000651 1735099260.115766 mapread 0x009ff9 thru 0x00f624 (0x562c bytes) 000652 1735099260.116424 write 0x01391e thru 0x02001f (0xc702 bytes) 000653 1735099260.117387 read 0x01e461 thru 0x02001f (0x1bbf bytes) 000654 1735099260.117784 mapwrite 0x027274 thru 0x0296b1 (0x243e bytes) 000656 1735099260.145232 mapwrite 0x03878b thru 0x03941f (0xc95 bytes) 000658 1735099260.169782 trunc from 0x039420 to 0x022d3b (0x166e4 bytes) 000661 1735099260.177826 write 0x02ac4b thru 0x02e619 (0x39cf bytes) 000662 1735099260.178746 write 0x024aa5 thru 0x02ef93 (0xa4ef bytes) 000663 1735099260.183103 read 0x0024b7 thru 0x007e1e (0x5968 bytes) 000664 1735099260.186760 mapwrite 0x0010be thru 0x006ac0 (0x5a03 bytes) 000666 1735099260.210432 write 0x009f80 thru 0x0139d9 (0x9a5a bytes) 000668 1735099260.211749 mapwrite 0x02dfc3 thru 0x02f5ef (0x162d bytes) 000673 1735099260.238963 mapread 0x01b215 thru 0x0274fd (0xc2e9 bytes) 000675 1735099260.240140 mapwrite 0x02ea42 thru 0x03c0ae (0xd66d bytes) 000676 1735099260.274313 mapwrite 0x02a3f1 thru 0x02d169 (0x2d79 bytes) 000677 1735099260.300541 mapread 0x021360 thru 0x02733e (0x5fdf bytes) 000678 1735099260.301719 mapwrite 0x01dd87 thru 0x029928 (0xbba2 bytes) 000683 1735099260.340748 write 0x0316bd thru 0x03f81e (0xe162 bytes) 000684 1735099260.344180 trunc from 0x03f81f to 0x02f20b (0x10613 bytes) 000685 1735099260.351429 mapwrite 0x015a39 thru 0x01f0fb (0x96c3 bytes) 000689 1735099260.374901 write 0x006ff6 thru 0x0110ad (0xa0b8 bytes) 000690 1735099260.375764 write 0x01ef84 thru 0x029ee9 (0xaf66 bytes) 000692 1735099260.376827 write 0x02b112 thru 0x033ff9 (0x8ee8 bytes) 000693 1735099260.377545 read 0x003121 thru 0x008bfc (0x5adc bytes) 000694 1735099260.379683 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000697 1735099260.453279 mapwrite 0x0140a3 thru 0x015d51 (0x1caf bytes) 000700 1735099260.474048 write 0x02cdab thru 0x031513 (0x4769 bytes) 000702 1735099260.480626 MIRROR_EXTEND to 16 mirrors 000703 1735099260.596758 trunc from 0x033ffa to 0x00b72f (0x288ca bytes) 000704 1735099260.618834 mapread 0x0057b1 thru 0x00b72f (0x5f7f bytes) 000705 1735099260.623925 read 0x005d28 thru 0x00b72f (0x5a08 bytes) 000707 1735099260.624524 mapwrite 0x001a4d thru 0x010615 (0xebc9 bytes) 000709 1735099260.663422 MIRROR_RESYNC 16 mirrors 000710 1735099261.178484 trunc from 0x010616 to 0x038757 (0x28142 bytes) 000711 1735099261.204373 mapwrite 0x00a4f3 thru 0x018c79 (0xe787 bytes) 000713 1735099261.233415 write 0x03e090 thru 0x03ffff (0x1f70 bytes) 000714 1735099261.233980 mapwrite 0x012cc5 thru 0x01ab12 (0x7e4e bytes) 000717 1735099261.253412 trunc from 0x040000 to 0x011f28 (0x2e0d7 bytes) 000718 1735099261.261006 trunc from 0x011f29 to 0x01bd30 (0x9e08 bytes) 000719 1735099261.266406 read 0x009fcc thru 0x00d8f3 (0x3928 bytes) 000720 1735099261.268573 MIRROR_RESYNC 16 mirrors 000721 1735099261.815102 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000722 1735099261.937342 mapwrite 0x01c3e7 thru 0x01e6a8 (0x22c2 bytes) 000723 1735099261.996766 mapread 0x002b19 thru 0x0068e9 (0x3dd1 bytes) 000725 1735099262.000851 trunc from 0x01e6a9 to 0x03eaaf (0x20407 bytes) 000727 1735099262.010561 MIRROR_RESYNC 15 mirrors 000730 1735099262.549492 read 0x00d7d5 thru 0x00dc4a (0x476 bytes) 000731 1735099262.551412 mapread 0x0023b2 thru 0x00927e (0x6ecd bytes) 000735 1735099262.557076 read 0x0176d7 thru 0x021dca (0xa6f4 bytes) 000736 1735099262.560260 mapwrite 0x0375ec thru 0x0395c0 (0x1fd5 bytes) 000738 1735099262.617199 mapread 0x021b28 thru 0x022b75 (0x104e bytes) 000740 1735099262.621259 read 0x031855 thru 0x03dce5 (0xc491 bytes) 000741 1735099262.625897 trunc from 0x03eab0 to 0x027dae (0x16d01 bytes) 000742 1735099262.633581 mapread 0x0262a6 thru 0x027dae (0x1b09 bytes) 000744 1735099262.634648 read 0x002fc4 thru 0x00df30 (0xaf6d bytes) 000745 1735099262.635113 mapwrite 0x03f82b thru 0x03ffff (0x7d5 bytes) 000747 1735099262.661144 mapwrite 0x03e132 thru 0x03ffff (0x1ece bytes) 000748 1735099262.684111 read 0x02bff3 thru 0x037c2f (0xbc3d bytes) 000749 1735099262.684498 read 0x0125e1 thru 0x01ef29 (0xc949 bytes) 000753 1735099262.685993 read 0x0316a3 thru 0x0332ce (0x1c2c bytes) 000754 1735099262.686368 mapread 0x031139 thru 0x03a0e0 (0x8fa8 bytes) 000755 1735099262.689043 MIRROR_RESYNC 15 mirrors 000756 1735099263.220290 mapwrite 0x029ad7 thru 0x037828 (0xdd52 bytes) 000758 1735099263.285588 read 0x013a14 thru 0x01ef8d (0xb57a bytes) 000759 1735099263.288324 trunc from 0x040000 to 0x014f99 (0x2b066 bytes) 000760 1735099263.294285 read 0x00768f thru 0x014f99 (0xd90b bytes) 000761 1735099263.297604 trunc from 0x014f9a to 0x01692a (0x1991 bytes) 000762 1735099263.302149 write 0x0241df thru 0x033661 (0xf483 bytes) 000763 1735099263.303081 trunc from 0x033662 to 0x03f41a (0xbdb9 bytes) 000764 1735099263.308972 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000765 1735099263.363027 trunc from 0x03f41b to 0x037316 (0x8104 bytes) 000766 1735099263.369456 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000768 1735099263.431790 write 0x00590b thru 0x00f32c (0x9a22 bytes) 000769 1735099263.435464 trunc from 0x037317 to 0x01dcc1 (0x19655 bytes) 000771 1735099263.440675 mapwrite 0x02cf12 thru 0x03a3bb (0xd4aa bytes) 000772 1735099263.472194 write 0x033d71 thru 0x035f33 (0x21c3 bytes) 000774 1735099263.473152 trunc from 0x03a3bc to 0x022baf (0x1780c bytes) 000776 1735099263.478877 mapread 0x01c1b8 thru 0x022baf (0x69f8 bytes) 000778 1735099263.482685 mapwrite 0x022a08 thru 0x0284cc (0x5ac5 bytes) 000779 1735099263.512881 mapread 0x00bbf0 thru 0x016739 (0xab4a bytes) 000782 1735099263.517988 mapread 0x020b11 thru 0x0253ff (0x48ef bytes) 000783 1735099263.518454 read 0x014600 thru 0x022172 (0xdb73 bytes) 000784 1735099263.518749 mapread 0x018f0b thru 0x01cbbf (0x3cb5 bytes) 000786 1735099263.522070 MIRROR_RESYNC 13 mirrors 000787 1735099263.981434 MIRROR_EXTEND to 14 mirrors 000792 1735099264.203068 mapread 0x000573 thru 0x00652d (0x5fbb bytes) 000794 1735099264.217677 mapwrite 0x009f5d thru 0x0169b5 (0xca59 bytes) 000797 1735099264.276069 trunc from 0x0284cd to 0x011978 (0x16b54 bytes) 000798 1735099264.287455 mapread 0x01169c thru 0x011978 (0x2dd bytes) 000799 1735099264.288029 trunc from 0x011979 to 0x0258cc (0x13f54 bytes) 000801 1735099264.292516 write 0x03da0c thru 0x03ffff (0x25f4 bytes) 000802 1735099264.293184 mapwrite 0x01bdd3 thru 0x029238 (0xd466 bytes) 000803 1735099264.317643 trunc from 0x040000 to 0x003c6a (0x3c395 bytes) 000804 1735099264.322822 mapwrite 0x00e343 thru 0x00f818 (0x14d6 bytes) 000805 1735099264.347698 MIRROR_RESYNC 14 mirrors 000806 1735099264.753787 trunc from 0x00f819 to 0x003c40 (0xbbd8 bytes) 000808 1735099264.780120 read 0x003197 thru 0x003c40 (0xaaa bytes) 000810 1735099264.784463 MIRROR_RESYNC 14 mirrors 000812 1735099265.154703 trunc from 0x003c41 to 0x01e6a1 (0x1aa61 bytes) 000814 1735099265.173889 trunc from 0x01e6a2 to 0x02d54c (0xeeab bytes) 000815 1735099265.179295 mapwrite 0x03cdc7 thru 0x03f363 (0x259d bytes) 000816 1735099265.207346 MIRROR_EXTEND to 15 mirrors 000817 1735099265.322559 mapwrite 0x01b00d thru 0x02510e (0xa102 bytes) 000820 1735099265.380071 read 0x036611 thru 0x03b38b (0x4d7b bytes) 000821 1735099265.383092 mapread 0x03a96e thru 0x03f363 (0x49f6 bytes) 000826 1735099265.388314 MIRROR_RESYNC 15 mirrors 000827 1735099265.862306 MIRROR_RESYNC 15 mirrors 000829 1735099265.971594 write 0x0257b5 thru 0x034b65 (0xf3b1 bytes) 000831 1735099265.999541 mapread 0x00c6dc thru 0x017d73 (0xb698 bytes) 000833 1735099266.006722 write 0x03b69e thru 0x03ffff (0x4962 bytes) 000834 1735099266.010882 trunc from 0x040000 to 0x036806 (0x97f9 bytes) 000837 1735099266.024331 mapwrite 0x017c9a thru 0x01d238 (0x559f bytes) 000840 1735099266.043354 mapwrite 0x010550 thru 0x0193d1 (0x8e82 bytes) 000842 1735099266.060819 trunc from 0x036807 to 0x03eca0 (0x849a bytes) 000843 1735099266.065657 trunc from 0x03eca1 to 0x00de37 (0x30e69 bytes) 000844 1735099266.071361 trunc from 0x00de38 to 0x02286a (0x14a33 bytes) 000846 1735099266.076031 write 0x030c68 thru 0x03906f (0x8408 bytes) 000849 1735099266.079379 MIRROR_RESYNC 15 mirrors 000850 1735099266.556556 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000851 1735099266.664777 read 0x00553b thru 0x007c53 (0x2719 bytes) 000852 1735099266.668654 MIRROR_EXTEND to 15 mirrors 000854 1735099266.836752 MIRROR_RESYNC 15 mirrors 000855 1735099266.927370 mapwrite 0x0214a4 thru 0x023536 (0x2093 bytes) 000856 1735099266.984425 write 0x02052a thru 0x02668c (0x6163 bytes) 000858 1735099266.985524 mapread 0x01eb15 thru 0x02b438 (0xc924 bytes) 000860 1735099266.989812 mapread 0x032920 thru 0x03906f (0x6750 bytes) 000861 1735099266.993178 write 0x02804f thru 0x02c67b (0x462d bytes) 000862 1735099266.993875 trunc from 0x039070 to 0x01e078 (0x1aff7 bytes) 000864 1735099267.002716 MIRROR_EXTEND to 16 mirrors 000866 1735099267.144755 write 0x009ce0 thru 0x011cb9 (0x7fda bytes) 000868 1735099267.163706 mapread 0x016080 thru 0x016635 (0x5b6 bytes) 000872 1735099267.167743 trunc from 0x01e079 to 0x01ebf3 (0xb7b bytes) 000875 1735099267.174468 read 0x01c2d0 thru 0x01ebf3 (0x2924 bytes) 000876 1735099267.175302 mapread 0x0114bc thru 0x0147dd (0x3322 bytes) 000878 1735099267.176230 read 0x0168e4 thru 0x01aeac (0x45c9 bytes) 000879 1735099267.176655 write 0x03cd86 thru 0x03eb1c (0x1d97 bytes) 000881 1735099267.178016 mapwrite 0x02df7a thru 0x02f50b (0x1592 bytes) 000883 1735099267.203430 read 0x0119d4 thru 0x01f659 (0xdc86 bytes) 000884 1735099267.203875 trunc from 0x03eb1d to 0x000f7c (0x3dba0 bytes) 000888 1735099267.213172 mapread 0x0009e2 thru 0x000f7c (0x59b bytes) 000889 1735099267.213811 write 0x029294 thru 0x02aa76 (0x17e3 bytes) 000892 1735099267.215384 write 0x00f364 thru 0x01c884 (0xd521 bytes) 000893 1735099267.220843 mapread 0x01403d thru 0x0148fa (0x8be bytes) 000894 1735099267.222059 mapwrite 0x03678c thru 0x037034 (0x8a9 bytes) 000895 1735099267.259641 read 0x0353fc thru 0x037034 (0x1c39 bytes) 000896 1735099267.261743 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000897 1735099267.320212 write 0x02c699 thru 0x02cbac (0x514 bytes) 000899 1735099267.322588 mapread 0x0165a4 thru 0x022631 (0xc08e bytes) 000901 1735099267.331290 trunc from 0x037035 to 0x013020 (0x24014 bytes) 000902 1735099267.336787 read 0x00a7d2 thru 0x00fda0 (0x55cf bytes) 000903 1735099267.340216 trunc from 0x013021 to 0x0393ff (0x263df bytes) 000904 1735099267.345827 write 0x02fd08 thru 0x039c38 (0x9f31 bytes) 000906 1735099267.351877 write 0x00cdaf thru 0x01246a (0x56bc bytes) 000907 1735099267.352566 mapread 0x02c8d8 thru 0x039c38 (0xd361 bytes) 000908 1735099267.354836 MIRROR_EXTEND to 16 mirrors 000909 1735099267.497246 trunc from 0x039c39 to 0x0237ae (0x1648a bytes) 000914 1735099267.532447 mapwrite 0x02005e thru 0x021027 (0xfca bytes) 000917 1735099267.559204 write 0x003238 thru 0x00c30e (0x90d7 bytes) 000918 1735099267.566025 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000919 1735099267.630170 write 0x00db7e thru 0x011e48 (0x42cb bytes) 000920 1735099267.634807 trunc from 0x0237af to 0x02c25a (0x8aac bytes) 000923 1735099267.642555 MIRROR_EXTEND to 16 mirrors 000924 1735099267.779147 mapread 0x021d31 thru 0x0299aa (0x7c7a bytes) 000926 1735099267.792989 mapread 0x02bc06 thru 0x02c25a (0x655 bytes) 000929 1735099267.799569 trunc from 0x02c25b to 0x0080bb (0x2419f bytes) 000930 1735099267.829830 read 0x001a0b thru 0x0080bb (0x66b1 bytes) 000931 1735099267.832772 trunc from 0x0080bc to 0x00f37e (0x72c3 bytes) 000932 1735099267.840256 write 0x008de0 thru 0x0103d5 (0x75f6 bytes) 000933 1735099267.841105 read 0x00c35f thru 0x0103d5 (0x4077 bytes) 000934 1735099267.843137 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000935 1735099267.919432 mapwrite 0x01bf49 thru 0x02af20 (0xefd8 bytes) 000938 1735099267.972799 write 0x010947 thru 0x0135a8 (0x2c62 bytes) 000940 1735099267.979117 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000942 1735099268.055038 read 0x0013e8 thru 0x0083ae (0x6fc7 bytes) 000944 1735099268.058054 mapwrite 0x02d820 thru 0x0312e4 (0x3ac5 bytes) 000945 1735099268.087079 mapread 0x01421f thru 0x0223f5 (0xe1d7 bytes) 000946 1735099268.093093 read 0x02eb84 thru 0x0312e4 (0x2761 bytes) 000947 1735099268.093684 mapwrite 0x001aaa thru 0x004a69 (0x2fc0 bytes) 000948 1735099268.110817 trunc from 0x0312e5 to 0x00a2a8 (0x2703c bytes) 000949 1735099268.117345 mapwrite 0x0206c5 thru 0x029eae (0x97ea bytes) 000952 1735099268.144050 read 0x0137c9 thru 0x022b47 (0xf37f bytes) 000954 1735099268.144793 trunc from 0x029eaf to 0x023e3b (0x6073 bytes) 000956 1735099268.149684 write 0x03292e thru 0x03930d (0x69e0 bytes) 000957 1735099268.150465 trunc from 0x03930e to 0x03f6e8 (0x63db bytes) 000958 1735099268.154378 read 0x009be0 thru 0x0172cf (0xd6f0 bytes) 000959 1735099268.154818 trunc from 0x03f6e9 to 0x0097df (0x35f09 bytes) 000960 1735099268.158553 write 0x00bb3e thru 0x017a91 (0xbf54 bytes) 000962 1735099268.159786 read 0x009151 thru 0x015f52 (0xce02 bytes) 000964 1735099268.162796 MIRROR_RESYNC 14 mirrors 000971 1735099268.571418 trunc from 0x017a92 to 0x01cd9d (0x530c bytes) 000975 1735099268.598818 trunc from 0x01cd9e to 0x03ae55 (0x1e0b8 bytes) 000976 1735099268.603669 mapwrite 0x01048f thru 0x014d6a (0x48dc bytes) 000977 1735099268.625558 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x798:0x0]: Invalid argument (22) 000980 1735099268.688398 write 0x014da2 thru 0x01a3e1 (0x5640 bytes) 000982 1735099268.693625 mapwrite 0x0282e5 thru 0x0310c3 (0x8ddf bytes) 000983 1735099268.718804 mapwrite 0x02d0b6 thru 0x02e580 (0x14cb bytes) 000984 1735099268.733700 mapwrite 0x026970 thru 0x029f76 (0x3607 bytes) 000987 1735099268.754775 MIRROR_EXTEND to 14 mirrors 000988 1735099268.903110 trunc from 0x03ae56 to 0x0155cb (0x2588a bytes) 000989 1735099268.932325 read 0x00454c thru 0x00dc91 (0x9746 bytes) 000990 1735099268.935095 trunc from 0x0155cc to 0x034a18 (0x1f44d bytes) 000991 1735099268.940560 mapwrite 0x0249f4 thru 0x02bb3e (0x714b bytes) 000996 1735099268.964321 write 0x00b923 thru 0x00e4e5 (0x2bc3 bytes) 000997 1735099268.966611 mapwrite 0x039470 thru 0x03d006 (0x3b97 bytes) 001000 1735099268.991368 read 0x009668 thru 0x00bfaa (0x2943 bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 200a: stress test ===================== 23:01:10 (1735099270) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7611968 .. Extending file size to 6904256 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7696832 .. Extending file size to 7983776 .. Extending file size to 8507904 .. Extending file size to 8968064 .. Extending file size to 8607488 .. Extending file size to 9184256 .. Extending file size to 8839200 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9212544 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 31984 31985 31987 31988 31990 done 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) 192.168.204.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:02:17 (1735099337) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 628096 .. Extending file size to 6517536 .. Extending file size to 8428480 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7040096 .. Extending file size to 8640448 .. Extending file size to 9172384 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8874624 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9100960 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 2151 2152 2153 2154 2156 done 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) 192.168.204.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:03:26 (1735099406) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210048 4864 2203136 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210048 4864 2203136 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0509103 s, 41.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.182965 s, 11.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:03:33 (1735099413) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0455927 s, 46.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 23:03:35 (1735099415) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7d5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa34:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa34:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa35:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xa37:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:03:38 (1735099418) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0497323 s, 42.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa38:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa39:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:03:40 (1735099420) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0401463 s, 52.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa38:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa38:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3b:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:03:43 (1735099423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0444837 s, 47.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa39:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa39:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3d:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:04:06 (1735099446) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396632 s, 26.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa3e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3b:0x0] } PASS 204d (40s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:04:47 (1735099487) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0386333 s, 54.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa40:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa40:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:05:27 (1735099527) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg434-server mds-ost sync done. Creating new pool test_204f oleg434-server: Pool lustre.test_204f created Adding targets to pool oleg434-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg434-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0459873 s, 45.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa41:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa41:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg434-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg434-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg434-server: Pool lustre.test_204f destroyed PASS 204f (55s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:06:22 (1735099582) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:06:24 (1735099584) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:06:27 (1735099587) oleg434-server: Pool lustre.test_206 created Adding targets to pool oleg434-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg434-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg434-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg434-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg434-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:06:40 (1735099600) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265047 s, 39.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121256 s, 86.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xd0debdb3 Mirror 3: 0xd0debdb3 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:06:43 (1735099603) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:06:45 (1735099605) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:06:46 (1735099606) Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0461434 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x10f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1740000 Read this (4 bytes): BBBB 192.168.204.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg434-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:07:02 (1735099622) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00704243 s, 1.2 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:07:05 (1735099625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172897 s, 60.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x7fa lmm_fid: [0x200000404:0x7fa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x7fa lmm_fid: [0x200000404:0x7fa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa4b:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835962 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-24 23:07:05.000000000 -0500 Modify: 2024-12-24 23:07:05.000000000 -0500 Change: 2024-12-24 23:07:06.000000000 -0500 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:07:08 (1735099628) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:07:09 (1735099629) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.331336 s, 124 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.029294 s, 140 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 24 23:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x7fc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 24 23:07 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 768 sec ============ 23:07:12 (1735099632) === sanity-flr: start cleanup 23:07:13 (1735099633) === === sanity-flr: finish cleanup 23:07:15 (1735099635) ===