-----============= acceptance-small: sanity-flr ============----- Thu Apr 17 05:30:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg345-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:30:52 (1744882252) === oleg345-client.virtnet: executing check_config_client /mnt/lustre oleg345-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg345-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012ab18800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab18800.idle_timeout=debug disable quota as required oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 05:31:00 (1744882260) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7035] == sanity-flr test 0a: lfs mirror create with -N option == 05:31:01 (1744882261) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:31:04 (1744882264) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:31:05 (1744882265) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:31:06 (1744882266) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:31:11 (1744882271) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:31:12 (1744882272) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 05:31:12 (1744882272) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012ab18800.stats=clear osc.lustre-OST0001-osc-ffff88012ab18800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:31:14 (1744882274) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:31:16 (1744882276) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:31:18 (1744882278) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:31:19 (1744882279) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x16:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range cannot extend layout with embed source fid, re-try old way lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x16:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 05:31:21 (1744882281) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:31:22 (1744882282) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:31:23 (1744882283) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:31:24 (1744882284) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:31:26 (1744882286) PASS 7 (1s) == sanity-flr test 8: extend mirror trying to use different OSTs ========================================================== 05:31:28 (1744882288) /mnt/lustre/f8.sanity-flr-1 - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25:0x0] } - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37:0x0] } PASS 8 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:31:29 (1744882289) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.253911 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302117 s, 34.7 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:31:31 (1744882291) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.0554768 s, 208 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:31:32 (1744882292) Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:31:58 (1744882318) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0553308 s, 75.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:32:07 (1744882327) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (17s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:32:24 (1744882344) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.765961 s, 68.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 17 05:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.260s user 0m0.000s sys 0m0.116s Stopping /mnt/lustre-ost1 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.253s user 0m0.000s sys 0m0.118s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg345-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg345-server oleg345-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.252s user 0m0.000s sys 0m0.102s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg345-server: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg345-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab18800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:32:45 (1744882365) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:32:46 (1744882366) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:32:47 (1744882367) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 05:32:48 (1744882368) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 05:32:49 (1744882369) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.09982 s, 126 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:32:52 (1744882372) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00918333 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102536 s, 102 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:32:53 (1744882373) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:32:56 (1744882376) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000664507 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000754691 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000688349 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000512197 s, 2.0 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 05:32:58 (1744882378) create test files with size 15731378 .. 8007e6efdb5a437d84137340c3d60283 - 1c2706358f4abac2b4886d0c4f6e6dae - ff6ce788fb4be8346921b8bc6db6cbe5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 05:33:03 (1744882383) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:33:06 (1744882386) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:33:07 (1744882387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208299 s, 50.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x3f:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x40:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x54:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00671348 s, 156 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x55:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x56:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x41:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 05:33:09 (1744882389) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0218377 s, 96.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0780037 s, 53.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0309894 s, 67.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 05:33:11 (1744882391) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 05:33:12 (1744882392) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:33:13 (1744882393) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00802519 s, 131 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 05:33:14 (1744882394) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:33:15 (1744882395) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5d:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x48:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x48:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:33:17 (1744882397) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0589479 s, 178 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:33:41 (1744882421) llite.lustre-ffff8800b38d4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0684906 s, 153 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg345-client.virtnet /mnt/lustre (opts:) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a5c93800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:33:44 (1744882424) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:33:51 (1744882431) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:33:58 (1744882438) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 05:33:58 (1744882438) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0568328 s, 55.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0417085 s, 75.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 463ed88e95a2357f6a8de0909c3f1556 - mirror 1 checksum: 8fe8abbbcdececaf655f737b283b2299 - mirror 2 checksum: 463ed88e95a2357f6a8de0909c3f1556 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 17 05:33:59 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 17 05:33:59 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 463ed88e95a2357f6a8de0909c3f1556 - mirror 1 checksum: 8fe8abbbcdececaf655f737b283b2299 - mirror 2 checksum: 8fe8abbbcdececaf655f737b283b2299 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 17 05:33:59 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 17 05:33:59 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:34:01 (1744882441) Failing mds1 on oleg345-server Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 05:34:08 (1744882448) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server Failover mds1 to oleg345-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:20 (1744882460) targets are mounted 05:34:20 (1744882460) facet_failover done oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.009262 s, 113 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:34:26 (1744882466) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:34:27 (1744882467) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:34:28 (1744882468) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00229566 s, 7.1 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:34:29 (1744882469) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:34:30 (1744882470) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:34:31 (1744882471) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:34:35 (1744882475) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:34:53 (1744882493) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:35:09 (1744882509) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:35:26 (1744882526) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 05:35:28 (1744882528) mirror create pid 23163 mirror split pid 23164 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 23163 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 23164 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 05:36:29 (1744882589) Seed set to 2590 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1744882590.269860 write 0x019a1a thru 0x024053 (0xa63a bytes) 000003 1744882590.271892 mapread 0x0137c3 thru 0x02002e (0xc86c bytes) 000004 1744882590.277764 MIRROR_EXTEND to 2 mirrors 000005 1744882590.372289 mapread 0x01d009 thru 0x024053 (0x704b bytes) 000006 1744882590.377283 mapwrite 0x035207 thru 0x03ffff (0xadf9 bytes) 000007 1744882590.413502 read 0x01879d thru 0x026af9 (0xe35d bytes) 000009 1744882590.416560 mapwrite 0x029b39 thru 0x037f66 (0xe42e bytes) 000012 1744882590.432898 mapwrite 0x0092f6 thru 0x00a259 (0xf64 bytes) truncating to largest ever: 0x3d30a 000013 1744882590.445404 trunc from 0x040000 to 0x03d309 (0x2cf6 bytes) 000014 1744882590.449289 read 0x0259c9 thru 0x029378 (0x39b0 bytes) 000015 1744882590.449540 read 0x02313b thru 0x02e150 (0xb016 bytes) 000018 1744882590.451121 MIRROR_EXTEND to 3 mirrors 000019 1744882590.545054 trunc from 0x03d30a to 0x032454 (0xaeb5 bytes) 000020 1744882590.564048 read 0x01ded0 thru 0x02dac3 (0xfbf4 bytes) 000021 1744882590.566715 mapwrite 0x01f6e1 thru 0x02cea7 (0xd7c7 bytes) 000022 1744882590.581674 read 0x021622 thru 0x030afb (0xf4da bytes) 000023 1744882590.584699 mapwrite 0x014b07 thru 0x01be4a (0x7344 bytes) 000024 1744882590.597214 trunc from 0x032455 to 0x02041c (0x12038 bytes) 000025 1744882590.601398 read 0x01bdd3 thru 0x02041c (0x464a bytes) 000027 1744882590.601963 trunc from 0x02041d to 0x021bb2 (0x1796 bytes) 000028 1744882590.606936 MIRROR_EXTEND to 4 mirrors 000029 1744882590.686980 write 0x015257 thru 0x0166e7 (0x1491 bytes) 000031 1744882590.700616 read 0x01f9b9 thru 0x021bb2 (0x21fa bytes) 000032 1744882590.702876 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000033 1744882590.744999 MIRROR_RESYNC 3 mirrors 000034 1744882590.853743 mapread 0x0080d1 thru 0x00ca59 (0x4989 bytes) 000037 1744882590.857559 mapwrite 0x0017f9 thru 0x004cce (0x34d6 bytes) 000038 1744882590.899379 read 0x012bfa thru 0x013ec8 (0x12cf bytes) 000039 1744882590.901045 mapwrite 0x01155c thru 0x0117d2 (0x277 bytes) 000040 1744882590.917352 trunc from 0x021bb3 to 0x03840c (0x1685a bytes) 000041 1744882590.926209 mapread 0x02f80f thru 0x034655 (0x4e47 bytes) 000042 1744882590.928473 read 0x018de1 thru 0x01f0ea (0x630a bytes) 000045 1744882590.931106 mapread 0x03717b thru 0x03840c (0x1292 bytes) 000046 1744882590.931442 read 0x02f0e8 thru 0x0335ad (0x44c6 bytes) 000048 1744882590.932036 write 0x014409 thru 0x0173c4 (0x2fbc bytes) 000051 1744882590.932985 read 0x034c6e thru 0x03840c (0x379f bytes) 000053 1744882590.933435 trunc from 0x03840d to 0x00f0b2 (0x2935a bytes) 000054 1744882590.938129 read 0x00beb7 thru 0x00f0b2 (0x31fc bytes) 000055 1744882590.938497 read 0x008207 thru 0x00f0b2 (0x6eac bytes) 000057 1744882590.940244 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000060 1744882590.984633 trunc from 0x00f0b3 to 0x0067fc (0x88b6 bytes) 000061 1744882590.988338 write 0x02adce thru 0x03a5fa (0xf82d bytes) 000063 1744882590.989792 read 0x027d75 thru 0x030ad5 (0x8d61 bytes) 000065 1744882590.991634 trunc from 0x03a5fb to 0x0192fd (0x212fd bytes) 000067 1744882590.995407 mapwrite 0x036ed2 thru 0x03ffff (0x912e bytes) 000068 1744882591.018203 read 0x034f08 thru 0x03a17b (0x5274 bytes) 000069 1744882591.018583 mapwrite 0x038e30 thru 0x03ffff (0x71d0 bytes) 000070 1744882591.032445 trunc from 0x040000 to 0x012eee (0x2d111 bytes) 000073 1744882591.036001 read 0x006cd3 thru 0x012655 (0xb983 bytes) 000077 1744882591.038986 mapread 0x0057f9 thru 0x012eee (0xd6f6 bytes) 000079 1744882591.041134 MIRROR_RESYNC 2 mirrors 000081 1744882591.114627 write 0x02269f thru 0x0245ef (0x1f51 bytes) 000085 1744882591.130202 trunc from 0x0245f0 to 0x001abd (0x22b32 bytes) 000087 1744882591.137761 write 0x029d5a thru 0x02a3ff (0x6a6 bytes) 000088 1744882591.139326 MIRROR_RESYNC 2 mirrors 000091 1744882591.222458 read 0x0119a0 thru 0x015079 (0x36da bytes) 000093 1744882591.226902 MIRROR_EXTEND to 3 mirrors 000095 1744882591.321135 mapwrite 0x03f281 thru 0x03ffff (0xd7f bytes) 000097 1744882591.359403 mapread 0x03b51b thru 0x03ffff (0x4ae5 bytes) 000098 1744882591.364974 MIRROR_EXTEND to 4 mirrors 000099 1744882591.454422 trunc from 0x040000 to 0x01b0fc (0x24f03 bytes) 000100 1744882591.477008 write 0x004962 thru 0x008e7c (0x451b bytes) 000101 1744882591.479780 read 0x005d09 thru 0x0106ab (0xa9a3 bytes) 000102 1744882591.482442 read 0x017cc8 thru 0x01a9ea (0x2d23 bytes) 000103 1744882591.485172 read 0x01629a thru 0x01b0fc (0x4e63 bytes) 000104 1744882591.485425 mapwrite 0x0138a5 thru 0x01b97b (0x80d7 bytes) 000105 1744882591.505476 read 0x002325 thru 0x0074c8 (0x51a4 bytes) 000107 1744882591.505934 read 0x01b4fd thru 0x01b97b (0x47f bytes) 000108 1744882591.506170 mapread 0x01a22c thru 0x01b97b (0x1750 bytes) 000110 1744882591.506649 mapwrite 0x017c1f thru 0x01db9b (0x5f7d bytes) 000111 1744882591.522665 read 0x010288 thru 0x01db9b (0xd914 bytes) 000114 1744882591.524333 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000117 1744882591.565481 trunc from 0x01db9c to 0x00951d (0x1467e bytes) 000118 1744882591.568620 trunc from 0x00951e to 0x01c7b7 (0x1329a bytes) 000119 1744882591.571631 trunc from 0x01c7b8 to 0x038c0d (0x1c456 bytes) 000121 1744882591.575038 write 0x02b777 thru 0x034660 (0x8eea bytes) 000122 1744882591.580177 MIRROR_EXTEND to 4 mirrors 000124 1744882591.688376 read 0x0145e7 thru 0x01a733 (0x614d bytes) 000125 1744882591.691126 trunc from 0x038c0e to 0x00a227 (0x2e9e6 bytes) 000127 1744882591.708099 trunc from 0x00a228 to 0x01095a (0x6733 bytes) 000128 1744882591.711446 write 0x010244 thru 0x014b1c (0x48d9 bytes) 000129 1744882591.712605 write 0x0018cf thru 0x005159 (0x388b bytes) 000130 1744882591.714656 read 0x008d2b thru 0x012990 (0x9c66 bytes) 000132 1744882591.716869 write 0x003f85 thru 0x00adbc (0x6e38 bytes) 000133 1744882591.718294 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000134 1744882591.759223 mapwrite 0x03a108 thru 0x03ffff (0x5ef8 bytes) 000137 1744882591.783271 read 0x0223af thru 0x02ded7 (0xbb29 bytes) 000138 1744882591.785812 trunc from 0x040000 to 0x01ed5c (0x212a3 bytes) 000143 1744882591.793242 trunc from 0x01ed5d to 0x018b2b (0x6231 bytes) 000144 1744882591.797491 trunc from 0x018b2c to 0x02089a (0x7d6f bytes) 000145 1744882591.801397 mapwrite 0x026e6c thru 0x03287d (0xba12 bytes) 000147 1744882591.821935 write 0x021904 thru 0x021a48 (0x145 bytes) 000148 1744882591.823713 MIRROR_RESYNC 3 mirrors 000150 1744882591.948497 trunc from 0x03287e to 0x026786 (0xc0f7 bytes) 000151 1744882591.965393 mapread 0x021f2c thru 0x022958 (0xa2d bytes) truncating to largest ever: 0x3e0bc 000154 1744882591.967948 trunc from 0x026787 to 0x03e0bb (0x17935 bytes) 000156 1744882591.971730 mapwrite 0x039be3 thru 0x03bf06 (0x2324 bytes) 000158 1744882591.986244 MIRROR_RESYNC 3 mirrors 000159 1744882592.093545 write 0x013d17 thru 0x023394 (0xf67e bytes) 000162 1744882592.110880 mapwrite 0x01c158 thru 0x01ffeb (0x3e94 bytes) 000163 1744882592.130019 trunc from 0x03e0bc to 0x013e1d (0x2a29e bytes) 000167 1744882592.137245 write 0x020e15 thru 0x02f13e (0xe32a bytes) 000168 1744882592.138358 write 0x03a32b thru 0x03f546 (0x521c bytes) 000169 1744882592.139089 write 0x02fed9 thru 0x034505 (0x462d bytes) 000172 1744882592.142171 write 0x02744b thru 0x02b352 (0x3f08 bytes) 000174 1744882592.142882 mapread 0x0078d5 thru 0x0098d9 (0x2005 bytes) 000176 1744882592.145781 trunc from 0x03f547 to 0x023811 (0x1bd35 bytes) 000178 1744882592.150032 trunc from 0x023812 to 0x02c866 (0x9055 bytes) 000179 1744882592.153755 read 0x02286d thru 0x02a61b (0x7daf bytes) 000185 1744882592.158095 read 0x029ffb thru 0x02c866 (0x286c bytes) 000187 1744882592.161322 read 0x01a918 thru 0x02a19b (0xf884 bytes) 000188 1744882592.161606 trunc from 0x02c867 to 0x03d502 (0x10c9c bytes) 000189 1744882592.165871 mapread 0x024927 thru 0x033efd (0xf5d7 bytes) 000190 1744882592.169361 trunc from 0x03d503 to 0x02d831 (0xfcd1 bytes) 000191 1744882592.175259 write 0x002dab thru 0x00bcb3 (0x8f09 bytes) 000192 1744882592.175844 mapread 0x007ccd thru 0x00e0ad (0x63e1 bytes) 000193 1744882592.176249 trunc from 0x02d832 to 0x03310c (0x58db bytes) 000195 1744882592.180181 mapread 0x02157b thru 0x026181 (0x4c07 bytes) 000200 1744882592.183222 MIRROR_EXTEND to 4 mirrors 000203 1744882592.282100 mapread 0x01ed4b thru 0x029819 (0xaacf bytes) 000204 1744882592.289098 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000205 1744882592.342813 mapread 0x01cefe thru 0x01eadd (0x1be0 bytes) 000206 1744882592.349616 write 0x0189bf thru 0x02071f (0x7d61 bytes) 000207 1744882592.364734 MIRROR_RESYNC 3 mirrors 000209 1744882592.477277 write 0x0215e0 thru 0x02c6d9 (0xb0fa bytes) 000210 1744882592.490316 read 0x007fa6 thru 0x016a46 (0xeaa1 bytes) 000211 1744882592.492659 mapread 0x012774 thru 0x01aed5 (0x8762 bytes) 000212 1744882592.496473 MIRROR_EXTEND to 4 mirrors 000213 1744882592.599383 MIRROR_RESYNC 4 mirrors 000219 1744882592.697168 mapwrite 0x011917 thru 0x015186 (0x3870 bytes) 000220 1744882592.743564 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000221 1744882592.788054 MIRROR_EXTEND to 4 mirrors 000222 1744882592.868812 read 0x0237a0 thru 0x028845 (0x50a6 bytes) 000225 1744882592.872543 trunc from 0x03310d to 0x00d82e (0x258de bytes) 000228 1744882592.890958 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000229 1744882592.934273 mapwrite 0x037389 thru 0x03a3d7 (0x304f bytes) 000230 1744882592.952943 mapread 0x02fb75 thru 0x033f77 (0x4403 bytes) 000232 1744882592.955608 write 0x0227c2 thru 0x02fd21 (0xd560 bytes) 000233 1744882592.957948 write 0x02d059 thru 0x038452 (0xb3fa bytes) 000236 1744882592.959408 mapwrite 0x02c4c0 thru 0x036ca2 (0xa7e3 bytes) 000237 1744882592.973260 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000239 1744882593.017752 mapwrite 0x0208ec thru 0x0210d8 (0x7ed bytes) 000241 1744882593.029712 read 0x007e68 thru 0x016a38 (0xebd1 bytes) 000242 1744882593.032220 mapread 0x011b68 thru 0x018fe4 (0x747d bytes) 000243 1744882593.035142 read 0x013dee thru 0x01a7d2 (0x69e5 bytes) 000244 1744882593.035550 mapwrite 0x02e7dc thru 0x03469e (0x5ec3 bytes) 000246 1744882593.049633 mapwrite 0x01f890 thru 0x01f901 (0x72 bytes) 000247 1744882593.059539 mapread 0x0039ba thru 0x00e3de (0xaa25 bytes) 000249 1744882593.060337 mapread 0x02fe20 thru 0x033657 (0x3838 bytes) 000250 1744882593.060795 write 0x002c1b thru 0x00ef44 (0xc32a bytes) 000253 1744882593.061895 read 0x019501 thru 0x025b84 (0xc684 bytes) 000255 1744882593.062382 read 0x000c0a thru 0x0040dd (0x34d4 bytes) 000256 1744882593.062604 write 0x012e36 thru 0x01b840 (0x8a0b bytes) 000257 1744882593.063157 mapwrite 0x0228b0 thru 0x02e5fb (0xbd4c bytes) 000260 1744882593.075591 write 0x01746f thru 0x01dca1 (0x6833 bytes) 000262 1744882593.076257 mapwrite 0x031e48 thru 0x036634 (0x47ed bytes) 000263 1744882593.086408 mapwrite 0x029a4d thru 0x02c973 (0x2f27 bytes) 000265 1744882593.095793 trunc from 0x03a3d8 to 0x01b093 (0x1f344 bytes) 000266 1744882593.099394 read 0x01321d thru 0x016bcc (0x39b0 bytes) 000270 1744882593.100251 write 0x02cf0b thru 0x036daa (0x9ea0 bytes) 000274 1744882593.101418 trunc from 0x036dab to 0x02f7db (0x75cf bytes) 000275 1744882593.104672 read 0x027b3e thru 0x02f7db (0x7c9e bytes) 000279 1744882593.107477 mapwrite 0x00655b thru 0x012f3f (0xc9e5 bytes) 000283 1744882593.123747 MIRROR_EXTEND to 3 mirrors 000285 1744882593.214239 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000286 1744882593.255570 write 0x033d72 thru 0x034c32 (0xec1 bytes) 000287 1744882593.257887 write 0x026b8c thru 0x03433c (0xd7b1 bytes) 000289 1744882593.260449 MIRROR_RESYNC 2 mirrors 000291 1744882593.340811 mapwrite 0x01a7a9 thru 0x01e263 (0x3abb bytes) 000293 1744882593.382679 mapwrite 0x014151 thru 0x01f80f (0xb6bf bytes) 000294 1744882593.396605 read 0x00bff1 thru 0x00d4b7 (0x14c7 bytes) 000296 1744882593.398716 write 0x0277f9 thru 0x0348d7 (0xd0df bytes) 000298 1744882593.402582 mapwrite 0x0268be thru 0x033daa (0xd4ed bytes) 000299 1744882593.415993 mapwrite 0x02ba32 thru 0x03a104 (0xe6d3 bytes) 000300 1744882593.436770 write 0x013261 thru 0x01c630 (0x93d0 bytes) 000301 1744882593.438607 mapread 0x0010c3 thru 0x0011c9 (0x107 bytes) 000302 1744882593.440397 mapwrite 0x007066 thru 0x00766b (0x606 bytes) 000304 1744882593.450177 mapread 0x002bdb thru 0x0062bf (0x36e5 bytes) 000305 1744882593.450586 trunc from 0x03a105 to 0x0065ee (0x33b16 bytes) 000307 1744882593.454199 trunc from 0x0065ef to 0x0150df (0xeaf1 bytes) 000309 1744882593.458104 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000310 1744882593.481800 mapread 0x0055ab thru 0x007d7d (0x27d3 bytes) 000312 1744882593.487810 MIRROR_EXTEND to 2 mirrors 000313 1744882593.570966 write 0x03dfbb thru 0x03ffff (0x2045 bytes) 000314 1744882593.584447 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000315 1744882593.611331 write 0x01ae03 thru 0x01bf0b (0x1109 bytes) 000319 1744882593.614888 mapwrite 0x038fad thru 0x03a98a (0x19de bytes) 000322 1744882593.631258 mapread 0x02772c thru 0x035cca (0xe59f bytes) 000326 1744882593.636213 mapwrite 0x01cb15 thru 0x027571 (0xaa5d bytes) 000327 1744882593.650609 read 0x004110 thru 0x00abf8 (0x6ae9 bytes) 000329 1744882593.652713 trunc from 0x040000 to 0x02fa80 (0x1057f bytes) 000330 1744882593.658893 mapwrite 0x0181a8 thru 0x01e543 (0x639c bytes) 000333 1744882593.671776 MIRROR_EXTEND to 2 mirrors 000334 1744882593.756519 read 0x003e63 thru 0x00ef9d (0xb13b bytes) 000336 1744882593.759444 mapwrite 0x01cfe0 thru 0x02542e (0x844f bytes) 000338 1744882593.801713 write 0x03885e thru 0x03ffff (0x77a2 bytes) 000339 1744882593.802319 read 0x03d0ce thru 0x03ffff (0x2f32 bytes) 000340 1744882593.802553 mapwrite 0x0334e0 thru 0x035bb7 (0x26d8 bytes) 000342 1744882593.816286 trunc from 0x040000 to 0x014fca (0x2b035 bytes) 000343 1744882593.823334 read 0x00c3f1 thru 0x014fca (0x8bda bytes) 000347 1744882593.824141 mapread 0x002d6f thru 0x00d8c7 (0xab59 bytes) 000349 1744882593.825792 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000350 1744882593.854156 read 0x007cf0 thru 0x014fca (0xd2db bytes) 000351 1744882593.856615 trunc from 0x014fcb to 0x032fa5 (0x1dfdb bytes) 000352 1744882593.859747 trunc from 0x032fa6 to 0x0209ce (0x125d7 bytes) 000353 1744882593.863563 mapwrite 0x01ae7b thru 0x026121 (0xb2a7 bytes) 000355 1744882593.883783 write 0x02ce94 thru 0x030588 (0x36f5 bytes) 000357 1744882593.884403 mapwrite 0x0327c1 thru 0x0371f9 (0x4a39 bytes) 000360 1744882593.900632 read 0x035868 thru 0x0371f9 (0x1992 bytes) 000361 1744882593.900926 mapread 0x0044a8 thru 0x010c9e (0xc7f7 bytes) 000364 1744882593.904438 mapread 0x01b724 thru 0x0298d6 (0xe1b3 bytes) 000365 1744882593.904848 read 0x0086b9 thru 0x016706 (0xe04e bytes) 000366 1744882593.905178 trunc from 0x0371fa to 0x007669 (0x2fb90 bytes) 000367 1744882593.908450 write 0x02bb55 thru 0x034a6d (0x8f19 bytes) 000370 1744882593.909697 mapread 0x02c27e thru 0x02dad3 (0x1856 bytes) 000371 1744882593.910091 trunc from 0x034a6e to 0x02aaad (0x9fc0 bytes) 000372 1744882593.913309 trunc from 0x02aaae to 0x00c6e2 (0x1e3cb bytes) 000378 1744882593.916934 trunc from 0x00c6e3 to 0x02fcec (0x2360a bytes) 000379 1744882593.919710 mapread 0x025af9 thru 0x02e348 (0x8850 bytes) 000381 1744882593.922938 trunc from 0x02fced to 0x0220b0 (0xdc3c bytes) 000383 1744882593.925977 mapread 0x00b397 thru 0x019779 (0xe3e3 bytes) 000385 1744882593.926832 write 0x03949d thru 0x03ffff (0x6b63 bytes) 000386 1744882593.927513 read 0x0024c2 thru 0x009d32 (0x7871 bytes) 000388 1744882593.928200 trunc from 0x040000 to 0x03b748 (0x48b7 bytes) 000389 1744882593.932353 MIRROR_EXTEND to 2 mirrors 000391 1744882594.024509 mapwrite 0x0130ef thru 0x01e346 (0xb258 bytes) 000393 1744882594.069613 mapread 0x013ac9 thru 0x0173e4 (0x391c bytes) 000395 1744882594.070464 mapread 0x015b08 thru 0x024996 (0xee8f bytes) 000396 1744882594.071085 mapwrite 0x02e137 thru 0x033a58 (0x5922 bytes) 000397 1744882594.083952 read 0x001331 thru 0x005794 (0x4464 bytes) 000398 1744882594.086251 trunc from 0x03b749 to 0x004e05 (0x36943 bytes) 000400 1744882594.093384 read 0x003985 thru 0x004e05 (0x1481 bytes) 000401 1744882594.093711 trunc from 0x004e06 to 0x02f91f (0x2ab1a bytes) 000402 1744882594.096594 mapwrite 0x00dc15 thru 0x017ab9 (0x9ea5 bytes) 000404 1744882594.111779 read 0x01e0c7 thru 0x01f78f (0x16c9 bytes) 000406 1744882594.113452 MIRROR_EXTEND to 3 mirrors 000407 1744882594.201545 read 0x015ca0 thru 0x01a37a (0x46db bytes) 000408 1744882594.204311 trunc from 0x02f920 to 0x019d95 (0x15b8a bytes) 000410 1744882594.220645 mapwrite 0x018cb6 thru 0x01ee7f (0x61ca bytes) 000412 1744882594.240856 mapread 0x018ba1 thru 0x01945b (0x8bb bytes) 000413 1744882594.242429 MIRROR_RESYNC 3 mirrors 000415 1744882594.341518 write 0x00b3bc thru 0x013ff7 (0x8c3c bytes) 000416 1744882594.354954 write 0x01e011 thru 0x0271eb (0x91db bytes) 000420 1744882594.357699 write 0x02d34b thru 0x031db2 (0x4a68 bytes) 000421 1744882594.358309 trunc from 0x031db3 to 0x02ab7e (0x7234 bytes) 000422 1744882594.365914 trunc from 0x02ab7f to 0x00c4ef (0x1e68f bytes) 000423 1744882594.369090 write 0x01d559 thru 0x021841 (0x42e9 bytes) 000424 1744882594.369825 read 0x01dd28 thru 0x01e48b (0x764 bytes) 000425 1744882594.370208 mapread 0x015493 thru 0x021841 (0xc3af bytes) 000427 1744882594.372766 mapread 0x01ed18 thru 0x021841 (0x2b2a bytes) 000429 1744882594.373374 mapread 0x01ad35 thru 0x01f5f5 (0x48c1 bytes) 000430 1744882594.374720 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000431 1744882594.417764 read 0x01b44d thru 0x021841 (0x63f5 bytes) 000433 1744882594.420344 mapwrite 0x01fa99 thru 0x020995 (0xefd bytes) 000434 1744882594.431410 mapread 0x001ac9 thru 0x004b01 (0x3039 bytes) 000435 1744882594.436358 read 0x0175b1 thru 0x021841 (0xa291 bytes) 000436 1744882594.438215 write 0x00a25a thru 0x01293d (0x86e4 bytes) 000437 1744882594.438676 write 0x00b28f thru 0x016546 (0xb2b8 bytes) 000438 1744882594.439310 mapwrite 0x00b595 thru 0x01542d (0x9e99 bytes) 000439 1744882594.452175 mapread 0x0115ca thru 0x0164c5 (0x4efc bytes) 000440 1744882594.452754 read 0x00b820 thru 0x018f71 (0xd752 bytes) 000442 1744882594.453512 mapwrite 0x010a6c thru 0x0202d8 (0xf86d bytes) 000443 1744882594.467940 mapread 0x00e7a1 thru 0x01bbab (0xd40b bytes) 000445 1744882594.468874 mapwrite 0x02bdb5 thru 0x0303b0 (0x45fc bytes) 000446 1744882594.487964 mapwrite 0x006001 thru 0x00e5b8 (0x85b8 bytes) 000447 1744882594.501556 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000449 1744882594.529292 write 0x03d33b thru 0x03ffff (0x2cc5 bytes) 000451 1744882594.530309 trunc from 0x040000 to 0x00e41f (0x31be0 bytes) 000455 1744882594.535972 read 0x007a4e thru 0x00d9fe (0x5fb1 bytes) 000456 1744882594.538772 mapwrite 0x010aa2 thru 0x016da9 (0x6308 bytes) 000459 1744882594.557391 mapwrite 0x01f2da thru 0x0216bd (0x23e4 bytes) 000460 1744882594.571586 mapwrite 0x024c0f thru 0x0305d4 (0xb9c6 bytes) 000462 1744882594.591780 trunc from 0x0305d5 to 0x031f90 (0x19bc bytes) 000463 1744882594.594683 mapread 0x02a19b thru 0x02b1c8 (0x102e bytes) 000464 1744882594.595257 write 0x01b9b2 thru 0x01fd71 (0x43c0 bytes) 000465 1744882594.595880 read 0x004503 thru 0x00ee8e (0xa98c bytes) 000466 1744882594.597984 write 0x0347c0 thru 0x03ee99 (0xa6da bytes) 000467 1744882594.598680 write 0x0041b5 thru 0x009a2c (0x5878 bytes) 000469 1744882594.599366 write 0x0026f0 thru 0x00df3c (0xb84d bytes) 000472 1744882594.600221 mapread 0x022467 thru 0x02c087 (0x9c21 bytes) 000473 1744882594.600564 mapwrite 0x036dac thru 0x03b4f5 (0x474a bytes) 000476 1744882594.613650 write 0x03e926 thru 0x03ffff (0x16da bytes) 000477 1744882594.614112 mapread 0x020f7b thru 0x0245a2 (0x3628 bytes) 000478 1744882594.614548 read 0x013125 thru 0x013864 (0x740 bytes) 000479 1744882594.614813 write 0x028726 thru 0x02e9de (0x62b9 bytes) 000481 1744882594.615621 read 0x00f143 thru 0x014d37 (0x5bf5 bytes) 000484 1744882594.616332 read 0x0305cc thru 0x03092b (0x360 bytes) 000490 1744882594.617592 read 0x03041a thru 0x03b436 (0xb01d bytes) 000491 1744882594.620426 mapread 0x011a06 thru 0x01e9c9 (0xcfc4 bytes) 000492 1744882594.620888 trunc from 0x040000 to 0x00068a (0x3f975 bytes) 000493 1744882594.624857 read 0x0002d0 thru 0x00068a (0x3bb bytes) 000498 1744882594.626200 read 0x000187 thru 0x00068a (0x504 bytes) 000500 1744882594.626656 mapwrite 0x01ce98 thru 0x02179e (0x4907 bytes) 000501 1744882594.644150 write 0x024f99 thru 0x033a55 (0xeabd bytes) 000503 1744882594.645288 mapwrite 0x01320b thru 0x017eb4 (0x4caa bytes) 000504 1744882594.658303 mapread 0x016690 thru 0x024520 (0xde91 bytes) 000505 1744882594.660322 read 0x02e908 thru 0x033a55 (0x514e bytes) 000507 1744882594.660969 mapwrite 0x0349ca thru 0x03842a (0x3a61 bytes) 000509 1744882594.680852 MIRROR_EXTEND to 2 mirrors 000512 1744882594.790635 write 0x006218 thru 0x00a6c3 (0x44ac bytes) 000514 1744882594.804241 read 0x023e36 thru 0x033077 (0xf242 bytes) 000515 1744882594.806148 mapread 0x00f6cf thru 0x01907d (0x99af bytes) 000517 1744882594.809522 read 0x027c55 thru 0x031adb (0x9e87 bytes) 000519 1744882594.809956 mapwrite 0x0202bf thru 0x02543b (0x517d bytes) 000521 1744882594.824734 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000522 1744882594.855583 write 0x01a874 thru 0x023058 (0x87e5 bytes) 000523 1744882594.858176 trunc from 0x03842b to 0x003371 (0x350b9 bytes) 000524 1744882594.865351 write 0x01b3cb thru 0x021a65 (0x669b bytes) 000525 1744882594.866162 mapwrite 0x032072 thru 0x03ffff (0xdf8e bytes) 000528 1744882594.889114 MIRROR_EXTEND to 2 mirrors 000532 1744882594.995759 trunc from 0x040000 to 0x031390 (0xec6f bytes) 000533 1744882595.012932 MIRROR_RESYNC 2 mirrors 000534 1744882595.096744 mapwrite 0x03a2e5 thru 0x03ffff (0x5d1b bytes) 000535 1744882595.127017 write 0x020d73 thru 0x0243a2 (0x3630 bytes) 000536 1744882595.129432 read 0x0244d9 thru 0x03035e (0xbe86 bytes) 000537 1744882595.131585 mapwrite 0x03562c thru 0x03703c (0x1a11 bytes) 000538 1744882595.143320 read 0x02d68b thru 0x02e833 (0x11a9 bytes) 000540 1744882595.143839 trunc from 0x040000 to 0x0081c6 (0x37e39 bytes) 000541 1744882595.148426 mapread 0x001f99 thru 0x0081c6 (0x622e bytes) 000544 1744882595.152811 MIRROR_RESYNC 2 mirrors 000546 1744882595.234304 mapwrite 0x01215d thru 0x020638 (0xe4dc bytes) 000547 1744882595.274470 write 0x03c74e thru 0x03ffff (0x38b2 bytes) 000548 1744882595.275264 mapwrite 0x0236ef thru 0x032333 (0xec45 bytes) 000549 1744882595.293789 mapread 0x008311 thru 0x01096e (0x865e bytes) 000551 1744882595.296436 write 0x0270f1 thru 0x02bd73 (0x4c83 bytes) 000555 1744882595.298838 MIRROR_RESYNC 2 mirrors 000558 1744882595.380954 trunc from 0x040000 to 0x02cd9a (0x13265 bytes) 000559 1744882595.399404 MIRROR_EXTEND to 3 mirrors 000562 1744882595.495991 read 0x00f81c thru 0x018798 (0x8f7d bytes) 000563 1744882595.499811 MIRROR_EXTEND to 4 mirrors 000564 1744882595.611545 trunc from 0x02cd9b to 0x00c86d (0x2052d bytes) 000565 1744882595.636871 write 0x01142a thru 0x012422 (0xff9 bytes) 000566 1744882595.637543 write 0x02b3a0 thru 0x02e46f (0x30d0 bytes) 000567 1744882595.639617 MIRROR_RESYNC 4 mirrors 000568 1744882595.786200 trunc from 0x02e470 to 0x034554 (0x60e5 bytes) 000569 1744882595.803085 write 0x0175ef thru 0x025363 (0xdd75 bytes) 000570 1744882595.806845 MIRROR_EXTEND to 5 mirrors 000571 1744882595.908357 trunc from 0x034555 to 0x007f1e (0x2c636 bytes) 000572 1744882595.926528 trunc from 0x007f1f to 0x01932f (0x11411 bytes) 000574 1744882595.930817 read 0x006fd4 thru 0x011c7e (0xacab bytes) 000578 1744882595.933894 write 0x011d60 thru 0x01b5cf (0x9870 bytes) 000580 1744882595.935764 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000581 1744882595.983494 mapread 0x007189 thru 0x00a4dc (0x3354 bytes) 000583 1744882595.989752 MIRROR_EXTEND to 5 mirrors 000584 1744882596.103894 mapwrite 0x005658 thru 0x0115a9 (0xbf52 bytes) 000587 1744882596.151259 trunc from 0x01b5d0 to 0x03bdc6 (0x207f7 bytes) 000588 1744882596.155481 read 0x03638b thru 0x03bdc6 (0x5a3c bytes) 000590 1744882596.157553 mapwrite 0x02d1d7 thru 0x035792 (0x85bc bytes) 000591 1744882596.172385 mapread 0x03632c thru 0x0376b4 (0x1389 bytes) 000592 1744882596.172757 write 0x02fe43 thru 0x036226 (0x63e4 bytes) 000594 1744882596.173405 read 0x009f03 thru 0x0117be (0x78bc bytes) 000596 1744882596.173980 mapread 0x034cfb thru 0x037528 (0x282e bytes) 000597 1744882596.174346 trunc from 0x03bdc7 to 0x00d6b9 (0x2e70d bytes) 000598 1744882596.179656 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000599 1744882596.229739 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000602 1744882596.273141 trunc from 0x00d6ba to 0x038acd (0x2b414 bytes) 000603 1744882596.276316 mapwrite 0x02eb1f thru 0x030980 (0x1e62 bytes) 000604 1744882596.290753 mapread 0x03289d thru 0x0379bf (0x5123 bytes) 000605 1744882596.291165 mapread 0x036724 thru 0x038acd (0x23aa bytes) 000606 1744882596.291552 write 0x0358cd thru 0x03f52a (0x9c5e bytes) 000607 1744882596.292105 trunc from 0x03f52b to 0x023ee0 (0x1b64a bytes) 000610 1744882596.296320 trunc from 0x023ee1 to 0x019743 (0xa79d bytes) 000611 1744882596.299525 mapwrite 0x035baa thru 0x038e07 (0x325e bytes) 000612 1744882596.323122 MIRROR_EXTEND to 4 mirrors 000614 1744882596.423730 mapread 0x0331dd thru 0x038e07 (0x5c2b bytes) 000617 1744882596.431320 trunc from 0x038e08 to 0x020bbc (0x1824b bytes) 000619 1744882596.458687 trunc from 0x020bbd to 0x025b33 (0x4f77 bytes) 000620 1744882596.463655 write 0x014935 thru 0x01ae87 (0x6553 bytes) 000623 1744882596.467207 trunc from 0x025b34 to 0x00133e (0x247f5 bytes) 000625 1744882596.472258 MIRROR_RESYNC 4 mirrors 000626 1744882596.601876 read 0x000dd5 thru 0x00133e (0x56a bytes) 000627 1744882596.603373 trunc from 0x00133f to 0x000ba1 (0x79d bytes) 000628 1744882596.622738 read 0x000011 thru 0x000ba1 (0xb91 bytes) 000630 1744882596.624567 read 0x0008e4 thru 0x000ba1 (0x2be bytes) 000631 1744882596.626688 MIRROR_RESYNC 4 mirrors 000633 1744882596.732465 mapread 0x00029f thru 0x000ba1 (0x903 bytes) 000635 1744882596.733002 read 0x00038f thru 0x000ba1 (0x813 bytes) 000636 1744882596.733175 read 0x0006c2 thru 0x000ba1 (0x4e0 bytes) 000638 1744882596.733506 read 0x00026c thru 0x000ba1 (0x936 bytes) 000640 1744882596.734791 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000641 1744882596.785100 mapwrite 0x037957 thru 0x0395e7 (0x1c91 bytes) 000642 1744882596.820661 mapread 0x028249 thru 0x02da72 (0x582a bytes) 000643 1744882596.823232 read 0x01a081 thru 0x026132 (0xc0b2 bytes) 000644 1744882596.825927 mapwrite 0x02e786 thru 0x039b81 (0xb3fc bytes) 000646 1744882596.847500 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000647 1744882596.898452 mapwrite 0x02874e thru 0x030dd2 (0x8685 bytes) 000649 1744882596.921589 read 0x016dae thru 0x02640c (0xf65f bytes) 000650 1744882596.924770 trunc from 0x039b82 to 0x0159ba (0x241c7 bytes) 000651 1744882596.929106 read 0x00b37d thru 0x0159ba (0xa63e bytes) 000654 1744882596.932211 write 0x03f565 thru 0x03ffff (0xa9b bytes) 000658 1744882596.933340 mapread 0x0022b4 thru 0x0118b8 (0xf605 bytes) 000662 1744882596.934750 mapwrite 0x0103dc thru 0x016740 (0x6365 bytes) 000663 1744882596.953152 write 0x01ec7a thru 0x02770b (0x8a92 bytes) 000666 1744882596.954469 mapwrite 0x00dd12 thru 0x011f15 (0x4204 bytes) 000667 1744882596.966607 write 0x0172a3 thru 0x0232ed (0xc04b bytes) 000669 1744882596.967475 write 0x018d9e thru 0x01e35c (0x55bf bytes) 000670 1744882596.968017 read 0x03630d thru 0x03ffff (0x9cf3 bytes) 000671 1744882596.969492 MIRROR_EXTEND to 3 mirrors 000672 1744882597.097561 write 0x02e2e7 thru 0x032eee (0x4c08 bytes) 000673 1744882597.112850 write 0x031514 thru 0x03d06c (0xbb59 bytes) 000677 1744882597.115546 mapwrite 0x01bb64 thru 0x029ec7 (0xe364 bytes) 000678 1744882597.136029 read 0x0048f5 thru 0x00c7a0 (0x7eac bytes) 000680 1744882597.138887 write 0x01b59f thru 0x01f967 (0x43c9 bytes) 000684 1744882597.140039 write 0x00c8ba thru 0x012937 (0x607e bytes) 000685 1744882597.141699 mapread 0x0185c0 thru 0x026153 (0xdb94 bytes) 000686 1744882597.142345 read 0x021342 thru 0x02973f (0x83fe bytes) 000687 1744882597.142733 trunc from 0x040000 to 0x01ef7c (0x21083 bytes) 000689 1744882597.148935 MIRROR_EXTEND to 4 mirrors 000690 1744882597.241574 mapread 0x01a2f6 thru 0x01dc38 (0x3943 bytes) 000691 1744882597.247577 mapwrite 0x01e26c thru 0x02c1ac (0xdf41 bytes) 000693 1744882597.287460 read 0x0070fd thru 0x00c090 (0x4f94 bytes) 000694 1744882597.289621 write 0x02f718 thru 0x03c324 (0xcc0d bytes) 000695 1744882597.290512 mapread 0x01750f thru 0x022998 (0xb48a bytes) 000696 1744882597.292335 MIRROR_RESYNC 4 mirrors 000697 1744882597.428457 mapwrite 0x00614f thru 0x014979 (0xe82b bytes) 000698 1744882597.473174 trunc from 0x03c325 to 0x034ba0 (0x7784 bytes) 000699 1744882597.481374 write 0x01f350 thru 0x026260 (0x6f11 bytes) 000700 1744882597.483865 write 0x020921 thru 0x029260 (0x8940 bytes) 000702 1744882597.486555 mapwrite 0x010eba thru 0x0191dc (0x8323 bytes) 000703 1744882597.501584 trunc from 0x034ba1 to 0x0229d3 (0x121cd bytes) 000705 1744882597.505893 write 0x00f652 thru 0x0180e7 (0x8a96 bytes) 000706 1744882597.506626 mapread 0x01cf3a thru 0x0229d3 (0x5a9a bytes) 000707 1744882597.507185 mapwrite 0x01194e thru 0x01f818 (0xdecb bytes) 000708 1744882597.524199 MIRROR_EXTEND to 5 mirrors 000712 1744882597.630441 write 0x018da8 thru 0x024e48 (0xc0a1 bytes) 000715 1744882597.645983 write 0x02edf9 thru 0x031ba5 (0x2dad bytes) 000716 1744882597.646602 read 0x020e86 thru 0x024b13 (0x3c8e bytes) 000717 1744882597.648258 MIRROR_EXTEND to 6 mirrors 000718 1744882597.741133 read 0x02185b thru 0x024272 (0x2a18 bytes) 000719 1744882597.743286 read 0x01c60a thru 0x020f27 (0x491e bytes) 000721 1744882597.747646 mapread 0x00321c thru 0x00f328 (0xc10d bytes) 000722 1744882597.750558 MIRROR_RESYNC 6 mirrors 000724 1744882597.905240 write 0x00aa1c thru 0x0120b5 (0x769a bytes) 000725 1744882597.920473 mapwrite 0x01414d thru 0x01644d (0x2301 bytes) 000726 1744882597.938925 read 0x0034d0 thru 0x007649 (0x417a bytes) 000727 1744882597.941296 read 0x015c4a thru 0x020c85 (0xb03c bytes) 000728 1744882597.945131 MIRROR_RESYNC 6 mirrors 000729 1744882598.122560 mapread 0x0282f4 thru 0x02feb8 (0x7bc5 bytes) 000732 1744882598.126506 mapread 0x02d286 thru 0x031ba5 (0x4920 bytes) 000735 1744882598.127125 read 0x003ab8 thru 0x00a28d (0x67d6 bytes) 000736 1744882598.129053 trunc from 0x031ba6 to 0x018818 (0x1938d bytes) 000737 1744882598.143530 read 0x00335b thru 0x004fdf (0x1c85 bytes) 000739 1744882598.146096 trunc from 0x018819 to 0x013506 (0x5312 bytes) 000740 1744882598.150153 trunc from 0x013507 to 0x0183eb (0x4ee5 bytes) 000741 1744882598.153604 mapread 0x0157c8 thru 0x017948 (0x2181 bytes) 000743 1744882598.158475 read 0x00703c thru 0x00d371 (0x6336 bytes) 000744 1744882598.158853 write 0x020be0 thru 0x022e74 (0x2295 bytes) 000745 1744882598.159319 trunc from 0x022e75 to 0x009bd3 (0x192a1 bytes) 000746 1744882598.162671 mapread 0x0046ec thru 0x007bc6 (0x34db bytes) 000747 1744882598.163197 mapwrite 0x007fee thru 0x00d033 (0x5046 bytes) 000750 1744882598.179884 read 0x00a4cc thru 0x00d033 (0x2b68 bytes) 000751 1744882598.180260 trunc from 0x00d034 to 0x0166bc (0x9689 bytes) 000752 1744882598.183526 mapwrite 0x0140bc thru 0x020e11 (0xcd56 bytes) 000753 1744882598.203727 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000754 1744882598.248451 MIRROR_RESYNC 5 mirrors 000756 1744882598.402054 mapwrite 0x021954 thru 0x023972 (0x201f bytes) 000757 1744882598.435746 write 0x00af1b thru 0x00f901 (0x49e7 bytes) 000758 1744882598.439550 MIRROR_EXTEND to 6 mirrors 000759 1744882598.544533 mapwrite 0x038a4a thru 0x03ffff (0x75b6 bytes) 000760 1744882598.593341 MIRROR_RESYNC 6 mirrors 000761 1744882598.755345 write 0x000581 thru 0x0097aa (0x922a bytes) 000762 1744882598.768320 mapwrite 0x039ab4 thru 0x03ffff (0x654c bytes) 000763 1744882598.784167 read 0x0214e3 thru 0x02810d (0x6c2b bytes) 000766 1744882598.788198 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000767 1744882598.833918 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000769 1744882598.873339 mapread 0x0370c9 thru 0x03ffff (0x8f37 bytes) 000770 1744882598.877736 MIRROR_RESYNC 4 mirrors 000772 1744882598.996977 mapread 0x0202f3 thru 0x02b5dc (0xb2ea bytes) 000775 1744882599.002335 read 0x02aa04 thru 0x036b2a (0xc127 bytes) 000777 1744882599.004221 mapwrite 0x00f9a8 thru 0x01ad31 (0xb38a bytes) 000778 1744882599.046718 mapwrite 0x01f983 thru 0x02d25e (0xd8dc bytes) 000779 1744882599.063211 mapwrite 0x0219b4 thru 0x02498c (0x2fd9 bytes) 000780 1744882599.073329 read 0x020886 thru 0x02a4f7 (0x9c72 bytes) 000781 1744882599.074789 MIRROR_EXTEND to 5 mirrors 000782 1744882599.167959 mapread 0x0300be thru 0x032f97 (0x2eda bytes) 000786 1744882599.175802 MIRROR_RESYNC 5 mirrors 000787 1744882599.294071 mapwrite 0x03042b thru 0x033718 (0x32ee bytes) 000788 1744882599.333516 MIRROR_RESYNC 5 mirrors 000789 1744882599.467736 mapread 0x018848 thru 0x01bbb5 (0x336e bytes) 000791 1744882599.471213 write 0x02d41b thru 0x0352af (0x7e95 bytes) 000793 1744882599.484332 mapread 0x0152c6 thru 0x024d19 (0xfa54 bytes) 000797 1744882599.488260 mapwrite 0x0390dd thru 0x03ffff (0x6f23 bytes) 000798 1744882599.503735 MIRROR_RESYNC 5 mirrors 000799 1744882599.644604 MIRROR_EXTEND to 6 mirrors 000800 1744882599.741460 trunc from 0x040000 to 0x02d4d9 (0x12b26 bytes) 000802 1744882599.757832 write 0x02daef thru 0x02df77 (0x489 bytes) 000806 1744882599.760088 write 0x02099b thru 0x0245fc (0x3c62 bytes) 000807 1744882599.762675 mapwrite 0x02145e thru 0x024142 (0x2ce5 bytes) 000808 1744882599.774868 mapread 0x01d933 thru 0x022397 (0x4a65 bytes) 000809 1744882599.775298 mapwrite 0x02a3a8 thru 0x038faf (0xec08 bytes) 000810 1744882599.794618 trunc from 0x038fb0 to 0x011c82 (0x2732d bytes) 000814 1744882599.798106 write 0x0170de thru 0x01a4de (0x3401 bytes) 000815 1744882599.798532 mapread 0x005cb8 thru 0x013270 (0xd5b9 bytes) 000817 1744882599.802035 trunc from 0x01a4df to 0x038ca6 (0x1e7c8 bytes) 000822 1744882599.807079 MIRROR_EXTEND to 7 mirrors 000823 1744882599.897645 mapwrite 0x003528 thru 0x00d881 (0xa35a bytes) 000824 1744882599.939142 trunc from 0x038ca7 to 0x018b7c (0x2012a bytes) 000825 1744882599.946271 trunc from 0x018b7d to 0x039e1a (0x2129e bytes) 000826 1744882599.949192 write 0x03b6e2 thru 0x03ffff (0x491e bytes) 000828 1744882599.949931 mapread 0x026886 thru 0x03251b (0xbc96 bytes) 000829 1744882599.953641 mapwrite 0x036aa7 thru 0x03ffff (0x9559 bytes) 000830 1744882599.966268 trunc from 0x040000 to 0x00638b (0x39c74 bytes) 000831 1744882599.969918 mapread 0x002acf thru 0x00638b (0x38bd bytes) 000833 1744882599.970853 write 0x02bb01 thru 0x03b148 (0xf648 bytes) 000835 1744882599.973324 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000836 1744882600.021021 read 0x02381d thru 0x029059 (0x583d bytes) 000838 1744882600.023001 read 0x03711c thru 0x03b148 (0x402d bytes) 000840 1744882600.026046 trunc from 0x03b149 to 0x002274 (0x38ed4 bytes) 000842 1744882600.029257 read 0x001ea1 thru 0x002274 (0x3d4 bytes) 000845 1744882600.029923 mapwrite 0x03f590 thru 0x03ffff (0xa70 bytes) 000847 1744882600.047841 trunc from 0x040000 to 0x005827 (0x3a7d8 bytes) 000850 1744882600.052907 MIRROR_RESYNC 6 mirrors 000851 1744882600.207869 MIRROR_RESYNC 6 mirrors 000852 1744882600.247434 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000853 1744882600.308616 trunc from 0x005828 to 0x00b718 (0x5ef1 bytes) 000854 1744882600.326876 write 0x038123 thru 0x03ffff (0x7edd bytes) 000855 1744882600.328657 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000856 1744882600.371744 mapread 0x02cbc8 thru 0x02d648 (0xa81 bytes) 000857 1744882600.374253 read 0x02d681 thru 0x037ac2 (0xa442 bytes) 000858 1744882600.376581 mapwrite 0x03133d thru 0x031f8b (0xc4f bytes) 000859 1744882600.388498 read 0x0017ac thru 0x00e917 (0xd16c bytes) 000861 1744882600.392795 mapread 0x00d7b3 thru 0x0164f8 (0x8d46 bytes) 000862 1744882600.393241 write 0x01f3c1 thru 0x0222fa (0x2f3a bytes) 000863 1744882600.393659 read 0x028a3f thru 0x02cd7b (0x433d bytes) 000868 1744882600.396148 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000869 1744882600.444455 read 0x02b4af thru 0x03ae6e (0xf9c0 bytes) 000870 1744882600.447598 mapwrite 0x0123f1 thru 0x01d5df (0xb1ef bytes) 000871 1744882600.466545 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000872 1744882600.508179 MIRROR_RESYNC 2 mirrors 000873 1744882600.589614 trunc from 0x040000 to 0x029931 (0x166ce bytes) 000874 1744882600.605240 trunc from 0x029932 to 0x020548 (0x93e9 bytes) 000878 1744882600.609259 write 0x00bed9 thru 0x015188 (0x92b0 bytes) 000883 1744882600.612298 read 0x01133b thru 0x019182 (0x7e48 bytes) 000887 1744882600.614315 mapread 0x00bc12 thru 0x0118be (0x5cad bytes) 000890 1744882600.615560 trunc from 0x020549 to 0x014aa5 (0xbaa3 bytes) 000891 1744882600.618748 trunc from 0x014aa6 to 0x01269d (0x2408 bytes) 000892 1744882600.621902 read 0x008d91 thru 0x01269d (0x990d bytes) 000893 1744882600.623952 read 0x009748 thru 0x00ac55 (0x150e bytes) 000894 1744882600.624239 read 0x00cd90 thru 0x00e5d3 (0x1844 bytes) 000896 1744882600.624639 trunc from 0x01269e to 0x00ce68 (0x5835 bytes) 000897 1744882600.627460 write 0x00a454 thru 0x01a050 (0xfbfd bytes) 000899 1744882600.628353 read 0x01906b thru 0x019390 (0x326 bytes) 000901 1744882600.628833 mapread 0x008aca thru 0x01463f (0xbb76 bytes) 000902 1744882600.629207 trunc from 0x01a051 to 0x02e37c (0x1432c bytes) 000904 1744882600.632679 write 0x0198ca thru 0x028509 (0xec40 bytes) 000905 1744882600.634753 write 0x030685 thru 0x03b769 (0xb0e5 bytes) 000908 1744882600.636317 read 0x0286ca thru 0x029e97 (0x17ce bytes) 000910 1744882600.638959 mapwrite 0x032807 thru 0x039463 (0x6c5d bytes) 000913 1744882600.652819 write 0x03ef29 thru 0x03ffff (0x10d7 bytes) 000914 1744882600.653289 mapwrite 0x0297d6 thru 0x031e66 (0x8691 bytes) 000915 1744882600.665669 read 0x029242 thru 0x034bad (0xb96c bytes) 000919 1744882600.666878 mapread 0x03feb8 thru 0x03ffff (0x148 bytes) 000924 1744882600.667998 mapwrite 0x022fa3 thru 0x027c73 (0x4cd1 bytes) 000926 1744882600.678743 mapwrite 0x02175a thru 0x02857e (0x6e25 bytes) 000927 1744882600.689961 write 0x01434a thru 0x0219b9 (0xd670 bytes) 000930 1744882600.691270 write 0x0063d7 thru 0x015649 (0xf273 bytes) 000931 1744882600.691836 read 0x039588 thru 0x03ffff (0x6a78 bytes) 000932 1744882600.693384 write 0x02d8c8 thru 0x038b4e (0xb287 bytes) 000934 1744882600.694191 mapread 0x03a29f thru 0x03ffff (0x5d61 bytes) 000936 1744882600.694987 mapwrite 0x02b7cf thru 0x03b59f (0xfdd1 bytes) 000937 1744882600.707446 write 0x019e8b thru 0x027a08 (0xdb7e bytes) 000940 1744882600.708559 trunc from 0x040000 to 0x00a395 (0x35c6a bytes) 000941 1744882600.712404 mapread 0x003d60 thru 0x00a395 (0x6636 bytes) 000943 1744882600.714255 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000944 1744882600.741223 mapread 0x008d78 thru 0x00a395 (0x161e bytes) 000945 1744882600.744977 write 0x00b868 thru 0x00ba9c (0x235 bytes) 000947 1744882600.746605 read 0x009801 thru 0x00ba9c (0x229c bytes) 000948 1744882600.746883 mapwrite 0x02a985 thru 0x038e00 (0xe47c bytes) 000949 1744882600.768612 mapwrite 0x00b2b1 thru 0x00e0cc (0x2e1c bytes) 000950 1744882600.780494 mapread 0x02254e thru 0x024dfc (0x28af bytes) 000951 1744882600.783754 MIRROR_EXTEND to 2 mirrors 000953 1744882600.868271 mapread 0x00d286 thru 0x019e2c (0xcba7 bytes) 000954 1744882600.873829 trunc from 0x038e01 to 0x008081 (0x30d7f bytes) 000956 1744882600.890732 MIRROR_EXTEND to 3 mirrors 000958 1744882600.969640 write 0x03ca5e thru 0x03ffff (0x35a2 bytes) 000959 1744882600.982557 mapread 0x03646a thru 0x03922f (0x2dc6 bytes) 000960 1744882600.986114 mapread 0x00758f thru 0x0090e2 (0x1b54 bytes) 000961 1744882600.988413 write 0x0169af thru 0x01bb05 (0x5157 bytes) 000962 1744882600.990469 read 0x025ff3 thru 0x02e145 (0x8153 bytes) 000964 1744882600.992743 mapwrite 0x0058bb thru 0x013f17 (0xe65d bytes) 000965 1744882601.005334 mapread 0x028a8c thru 0x031642 (0x8bb7 bytes) 000967 1744882601.007477 read 0x00032a thru 0x00b03b (0xad12 bytes) 000968 1744882601.010352 mapwrite 0x01ebfa thru 0x02ae5a (0xc261 bytes) 000969 1744882601.022304 trunc from 0x040000 to 0x01a552 (0x25aad bytes) 000970 1744882601.030758 read 0x001433 thru 0x007a0b (0x65d9 bytes) 000971 1744882601.032200 MIRROR_EXTEND to 4 mirrors 000973 1744882601.123568 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000974 1744882601.172520 mapwrite 0x02cccc thru 0x02cfc8 (0x2fd bytes) 000976 1744882601.201835 trunc from 0x02cfc9 to 0x009b67 (0x23461 bytes) 000977 1744882601.206657 trunc from 0x009b68 to 0x037fad (0x2e446 bytes) 000978 1744882601.209848 trunc from 0x037fae to 0x03a4c1 (0x2514 bytes) 000979 1744882601.212866 mapwrite 0x038341 thru 0x03fa30 (0x76f0 bytes) 000982 1744882601.233094 MIRROR_EXTEND to 4 mirrors 000984 1744882601.318607 mapread 0x03e2fd thru 0x03fa30 (0x1734 bytes) 000986 1744882601.322237 trunc from 0x03fa31 to 0x01bf2a (0x23b06 bytes) 000988 1744882601.339371 trunc from 0x01bf2b to 0x006e3b (0x150ef bytes) 000989 1744882601.342733 mapwrite 0x00c480 thru 0x014d9b (0x891c bytes) 000991 1744882601.359584 trunc from 0x014d9c to 0x039e79 (0x250de bytes) 000993 1744882601.363508 write 0x036d49 thru 0x03ffff (0x92b7 bytes) 000994 1744882601.366151 mapread 0x01e69b thru 0x027d4c (0x96b2 bytes) 000995 1744882601.370869 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9cb:0x0]: Invalid argument (22) 000996 1744882601.415695 trunc from 0x040000 to 0x03db0f (0x24f0 bytes) 000998 1744882601.418715 write 0x02976c thru 0x031a73 (0x8308 bytes) 000999 1744882601.420868 mapread 0x02593e thru 0x02cd3d (0x7400 bytes) 001000 1744882601.422914 mapread 0x00fda5 thru 0x01a5f3 (0xa84f bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 05:36:42 (1744882602) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 05:36:43 (1744882603) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3537472 .. Extending file size to 5762688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7255456 .. Extending file size to 6492160 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8051840 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8193600 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8124576 .. Extending file size to 8523072 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9023968 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9171296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9053376 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9229184 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9403840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 31051 31052 31054 31055 31058 done 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) 192.168.203.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:37:47 (1744882667) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5176480 .. Extending file size to 7519488 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8600416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8097088 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8558720 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8759296 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9315168 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8726336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8915712 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8946624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9327712 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8953088 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9343808 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 2815 2816 2818 2820 2822 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) 192.168.203.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:38:50 (1744882730) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0340161 s, 61.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0764195 s, 27.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:38:54 (1744882734) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0224066 s, 93.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 05:38:56 (1744882736) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa19:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4e:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xd4f:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:38:57 (1744882737) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.025984 s, 80.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6c:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd50:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd51:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:38:59 (1744882739) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263699 s, 79.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6d:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6e:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd52:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6d:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6e:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd52:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd53:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:39:01 (1744882741) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0213632 s, 98.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6f:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd54:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6f:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd54:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd55:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:39:23 (1744882763) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239782 s, 43.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd70:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd57:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd56:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd70:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd57:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd56:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd71:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:40:05 (1744882805) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd72:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0256699 s, 81.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd72:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd58:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd72:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd58:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:40:45 (1744882845) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg345-server mds-ost sync done. Creating new pool test_204f oleg345-server: Pool lustre.test_204f created Adding targets to pool oleg345-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg345-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0206693 s, 101 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd59:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd59:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg345-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg345-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg345-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:41:37 (1744882897) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:41:39 (1744882899) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:41:40 (1744882900) oleg345-server: Pool lustre.test_206 created Adding targets to pool oleg345-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg345-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg345-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg345-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg345-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:41:53 (1744882913) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162068 s, 64.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956515 s, 110 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xffead32d Mirror 3: 0xffead32d PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:41:55 (1744882915) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:41:55 (1744882915) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:41:56 (1744882916) Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0131862 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1450000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1450000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1450000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1000000 Read this (4 bytes): BBBB 192.168.203.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg345-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:42:09 (1744882929) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00231484 s, 3.5 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:42:10 (1744882930) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104869 s, 100 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa3e lmm_fid: [0x200000404:0xa3e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa3e lmm_fid: [0x200000404:0xa3e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd63:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836542 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-17 05:42:11.000000000 -0400 Modify: 2025-04-17 05:42:11.000000000 -0400 Change: 2025-04-17 05:42:11.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:42:12 (1744882932) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:42:13 (1744882933) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.10529 s, 389 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0141903 s, 289 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 17 05:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa40:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 17 05:42 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 689 sec ============ 05:42:15 (1744882935) === sanity-flr: start cleanup 05:42:15 (1744882935) === === sanity-flr: finish cleanup 05:42:16 (1744882936) ===