-----============= acceptance-small: sanity-flr ============----- Tue Jun 4 23:04:30 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:04:34 (1717556674) === oleg117-client.virtnet: executing check_config_client /mnt/lustre oleg117-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg117-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5ac9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5ac9800.idle_timeout=debug disable quota as required oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:04:40 (1717556680) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6941] == sanity-flr test 0a: lfs mirror create with -N option == 23:04:40 (1717556680) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:04:43 (1717556683) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:04:43 (1717556683) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:04:44 (1717556684) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:04:49 (1717556689) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:04:50 (1717556690) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:04:51 (1717556691) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5ac9800.stats=clear osc.lustre-OST0001-osc-ffff8800b5ac9800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:04:54 (1717556694) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:04:57 (1717556697) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:04:59 (1717556699) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:05:00 (1717556700) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x22:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 23:05:02 (1717556702) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:05:04 (1717556704) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:05:05 (1717556705) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:05:07 (1717556707) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:05:10 (1717556710) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:05:12 (1717556712) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.683782 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0654103 s, 16.0 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:05:16 (1717556716) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.0618097 s, 339 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:05:19 (1717556719) Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:05:45 (1717556745) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0583576 s, 71.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:05:56 (1717556756) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:06:15 (1717556775) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.583301 s, 89.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 4 23:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.187s user 0m0.000s sys 0m0.091s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.189s user 0m0.000s sys 0m0.095s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.220s user 0m0.000s sys 0m0.096s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5ac9800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:06:37 (1717556797) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:06:37 (1717556797) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:06:38 (1717556798) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:06:39 (1717556799) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 23:06:40 (1717556800) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0901338 s, 140 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:06:43 (1717556803) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00656019 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0049719 s, 211 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:06:45 (1717556805) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:06:47 (1717556807) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000688933 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000595818 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00161943 s, 632 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000487173 s, 2.1 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 23:06:50 (1717556810) create test files with size 15738581 .. 50653b5e1206d764dacba051688224cc - 107c76b8f956861163e003b6394ddc8b - 862cda5e32c582c15f1d6f2e69c48aba - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 23:06:55 (1717556815) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:06:58 (1717556818) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:07:00 (1717556820) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243705 s, 43.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x42:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00535157 s, 196 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:07:02 (1717556822) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0208901 s, 100 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0853032 s, 49.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0405437 s, 51.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 23:07:04 (1717556824) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:07:05 (1717556825) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:07:06 (1717556826) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00498938 s, 210 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:07:08 (1717556828) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:07:09 (1717556829) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:07:11 (1717556831) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0292282 s, 359 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:07:14 (1717556834) llite.lustre-ffff8800b63ad000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0512556 s, 205 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 32782 flags: 1 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 32782 flags: 1 llite.lustre-ffff88013028a800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:07:17 (1717556837) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:07:25 (1717556845) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:07:31 (1717556851) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:07:32 (1717556852) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0521407 s, 60.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0441295 s, 71.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: f87503d91d2999d40c3f68118ca98da4 - mirror 1 checksum: 5899b7cd35e0eab05d287eb1fa954346 - mirror 2 checksum: f87503d91d2999d40c3f68118ca98da4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 4 23:07:33 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 4 23:07:33 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f87503d91d2999d40c3f68118ca98da4 - mirror 1 checksum: 5899b7cd35e0eab05d287eb1fa954346 - mirror 2 checksum: 5899b7cd35e0eab05d287eb1fa954346 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 4 23:07:33 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x60:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 4 23:07:33 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x61:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:07:35 (1717556855) Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 23:07:37 (1717556857) shut down Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 23:07:50 (1717556870) targets are mounted 23:07:50 (1717556870) facet_failover done oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00510536 s, 205 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:07:55 (1717556875) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:07:56 (1717556876) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:07:57 (1717556877) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00300441 s, 5.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:07:59 (1717556879) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:08:00 (1717556880) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:08:01 (1717556881) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:08:07 (1717556887) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:08:25 (1717556905) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:08:42 (1717556922) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:08:59 (1717556939) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 23:09:01 (1717556941) mirror create pid 20525 mirror split pid 20526 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4717: 20525 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4717: 20526 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 23:10:03 (1717557003) Seed set to 7003 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1717557003.799210 mapwrite 0x024ed3 thru 0x02ace7 (0x5e15 bytes) 000002 1717557003.833987 mapread 0x0289c1 thru 0x02ace7 (0x2327 bytes) truncating to largest ever: 0x110f 000005 1717557003.834497 trunc from 0x02ace8 to 0x00110e (0x29bd9 bytes) 000008 1717557003.838043 mapwrite 0x00fae5 thru 0x01b6f8 (0xbc14 bytes) truncating to largest ever: 0x545f 000009 1717557003.868991 trunc from 0x01b6f9 to 0x00545e (0x1629a bytes) truncating to largest ever: 0x3e68d 000011 1717557003.874138 trunc from 0x00545f to 0x03e68c (0x3922e bytes) 000015 1717557003.878015 trunc from 0x03e68d to 0x00340b (0x3b281 bytes) 000016 1717557003.881416 read 0x000ca8 thru 0x00340b (0x2764 bytes) 000018 1717557003.883687 mapwrite 0x029341 thru 0x02b5a3 (0x2263 bytes) 000019 1717557003.909594 mapwrite 0x031111 thru 0x039a55 (0x8945 bytes) 000021 1717557003.931035 write 0x0226e9 thru 0x02b972 (0x928a bytes) 000026 1717557003.931931 write 0x01f283 thru 0x01ffb2 (0xd30 bytes) 000029 1717557003.934495 mapread 0x01f6c0 thru 0x02e255 (0xeb96 bytes) 000032 1717557003.935164 write 0x016269 thru 0x0231c0 (0xcf58 bytes) 000033 1717557003.937338 mapread 0x02e753 thru 0x03897b (0xa229 bytes) 000036 1717557003.939067 MIRROR_EXTEND to 2 mirrors 000037 1717557004.029778 trunc from 0x039a56 to 0x00d576 (0x2c4df bytes) 000040 1717557004.050159 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3fe3c 000041 1717557004.154651 trunc from 0x00d577 to 0x03fe3b (0x328c5 bytes) 000042 1717557004.172683 write 0x006673 thru 0x0082c1 (0x1c4f bytes) 000043 1717557004.174871 mapwrite 0x03aa16 thru 0x03ffff (0x55ea bytes) 000045 1717557004.195823 trunc from 0x040000 to 0x035f42 (0xa0bd bytes) 000046 1717557004.199703 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000047 1717557004.244637 write 0x00c144 thru 0x012fa7 (0x6e64 bytes) 000050 1717557004.247094 mapwrite 0x037216 thru 0x03ffff (0x8dea bytes) 000055 1717557004.268928 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000057 1717557004.299968 trunc from 0x040000 to 0x01d1f6 (0x22e09 bytes) 000059 1717557004.303585 read 0x019893 thru 0x01b4d1 (0x1c3f bytes) 000061 1717557004.305004 write 0x000e66 thru 0x0067f7 (0x5992 bytes) 000062 1717557004.308330 MIRROR_EXTEND to 2 mirrors 000063 1717557004.393918 mapread 0x01932d thru 0x01d1f6 (0x3eca bytes) 000065 1717557004.399773 write 0x031bd5 thru 0x033e42 (0x226e bytes) 000069 1717557004.415079 MIRROR_RESYNC 2 mirrors 000070 1717557004.500468 read 0x0123c7 thru 0x020f74 (0xebae bytes) 000072 1717557004.503048 trunc from 0x033e43 to 0x01db1f (0x16323 bytes) 000074 1717557004.521700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000075 1717557004.553054 MIRROR_EXTEND to 2 mirrors 000077 1717557004.636882 mapread 0x019f21 thru 0x01bafb (0x1bdb bytes) 000078 1717557004.640850 read 0x001218 thru 0x00c7ac (0xb595 bytes) 000079 1717557004.642652 write 0x037cd2 thru 0x03d096 (0x53c5 bytes) 000083 1717557004.657690 write 0x02516e thru 0x02e447 (0x92da bytes) 000086 1717557004.661926 write 0x0367c3 thru 0x038583 (0x1dc1 bytes) 000088 1717557004.665547 mapwrite 0x02582a thru 0x033272 (0xda49 bytes) 000089 1717557004.682590 write 0x016c91 thru 0x02251a (0xb88a bytes) 000093 1717557004.685803 read 0x00fa39 thru 0x014bb3 (0x517b bytes) 000095 1717557004.686156 mapwrite 0x00f337 thru 0x017b67 (0x8831 bytes) 000097 1717557004.701932 read 0x0010a2 thru 0x00bdfa (0xad59 bytes) 000098 1717557004.705311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000099 1717557004.740248 read 0x01a487 thru 0x0226ee (0x8268 bytes) 000101 1717557004.742986 trunc from 0x03d097 to 0x01a12b (0x22f6b bytes) 000104 1717557004.750888 mapread 0x00d9f5 thru 0x01a12b (0xc737 bytes) 000106 1717557004.757031 read 0x001a71 thru 0x0040ec (0x267c bytes) 000111 1717557004.759808 mapread 0x017c9a thru 0x01a12b (0x2492 bytes) 000112 1717557004.760124 mapread 0x011db1 thru 0x01a12b (0x837b bytes) 000114 1717557004.760614 mapwrite 0x02326c thru 0x031d5d (0xeaf2 bytes) 000118 1717557004.785081 read 0x0000f8 thru 0x007db5 (0x7cbe bytes) 000119 1717557004.785330 trunc from 0x031d5e to 0x014981 (0x1d3dc bytes) 000126 1717557004.790267 mapwrite 0x029d03 thru 0x02da74 (0x3d72 bytes) 000127 1717557004.810552 read 0x028966 thru 0x02da74 (0x510f bytes) 000128 1717557004.811568 MIRROR_EXTEND to 2 mirrors 000130 1717557004.908852 mapwrite 0x002f51 thru 0x012a59 (0xfb09 bytes) 000133 1717557004.959690 write 0x01241e thru 0x0128f4 (0x4d7 bytes) 000134 1717557004.960011 trunc from 0x02da75 to 0x03115c (0x36e8 bytes) 000135 1717557004.968696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000136 1717557005.003901 write 0x01ba9b thru 0x0215f4 (0x5b5a bytes) 000138 1717557005.006895 mapread 0x0019f0 thru 0x009931 (0x7f42 bytes) 000139 1717557005.010238 trunc from 0x03115d to 0x039de3 (0x8c87 bytes) 000141 1717557005.014190 mapread 0x014093 thru 0x01b3dd (0x734b bytes) 000142 1717557005.016838 write 0x011773 thru 0x01dfc7 (0xc855 bytes) 000143 1717557005.017242 read 0x01fc7d thru 0x02b2bc (0xb640 bytes) 000144 1717557005.019463 mapwrite 0x002ed2 thru 0x00d4e1 (0xa610 bytes) 000145 1717557005.037487 trunc from 0x039de4 to 0x038432 (0x19b1 bytes) 000146 1717557005.041485 write 0x03205e thru 0x03dbe2 (0xbb85 bytes) 000150 1717557005.044544 write 0x009190 thru 0x016cfc (0xdb6d bytes) 000151 1717557005.044960 mapwrite 0x00eb0e thru 0x0120d3 (0x35c6 bytes) 000153 1717557005.062666 write 0x01e660 thru 0x02512e (0x6acf bytes) 000154 1717557005.063010 write 0x01c75e thru 0x01d153 (0x9f6 bytes) 000155 1717557005.063253 mapread 0x02a4e8 thru 0x032585 (0x809e bytes) 000157 1717557005.063817 mapread 0x006263 thru 0x00c808 (0x65a6 bytes) 000158 1717557005.064051 write 0x0166c4 thru 0x019098 (0x29d5 bytes) 000161 1717557005.064454 write 0x009d01 thru 0x00fdca (0x60ca bytes) 000166 1717557005.065132 read 0x00a404 thru 0x0132ce (0x8ecb bytes) 000168 1717557005.066830 MIRROR_EXTEND to 2 mirrors 000170 1717557005.167461 mapwrite 0x00b204 thru 0x018fb7 (0xddb4 bytes) 000172 1717557005.213202 MIRROR_RESYNC 2 mirrors 000173 1717557005.301989 MIRROR_EXTEND to 3 mirrors 000175 1717557005.397953 mapwrite 0x029466 thru 0x038428 (0xefc3 bytes) 000177 1717557005.450655 mapread 0x00223d thru 0x004533 (0x22f7 bytes) 000178 1717557005.453377 trunc from 0x03dbe3 to 0x025a3e (0x181a4 bytes) 000179 1717557005.462863 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000180 1717557005.509481 mapread 0x008b21 thru 0x00e680 (0x5b60 bytes) 000182 1717557005.514292 trunc from 0x025a3f to 0x006140 (0x1f8fe bytes) 000183 1717557005.518517 trunc from 0x006141 to 0x03f063 (0x38f23 bytes) 000185 1717557005.521978 mapwrite 0x03a892 thru 0x03e946 (0x40b5 bytes) 000188 1717557005.541359 read 0x008413 thru 0x0165fc (0xe1ea bytes) 000190 1717557005.543950 mapwrite 0x02ce7f thru 0x02d73e (0x8c0 bytes) 000193 1717557005.561199 trunc from 0x03f064 to 0x02b258 (0x13e0b bytes) 000195 1717557005.565609 mapwrite 0x03bf97 thru 0x03fefa (0x3f64 bytes) 000197 1717557005.585741 trunc from 0x03fefb to 0x0008ff (0x3f5fb bytes) 000199 1717557005.589614 mapwrite 0x01ccbe thru 0x026fe8 (0xa32b bytes) 000201 1717557005.610936 mapwrite 0x0355b6 thru 0x037093 (0x1ade bytes) 000202 1717557005.629364 write 0x0319c7 thru 0x032b68 (0x11a2 bytes) 000203 1717557005.631003 MIRROR_RESYNC 2 mirrors 000205 1717557005.728794 mapread 0x0027e6 thru 0x00b90c (0x9127 bytes) 000206 1717557005.733932 mapread 0x015964 thru 0x01a90f (0x4fac bytes) 000207 1717557005.736555 read 0x02741e thru 0x029494 (0x2077 bytes) 000209 1717557005.737891 trunc from 0x037094 to 0x0066cb (0x309c8 bytes) 000211 1717557005.762650 read 0x003d6b thru 0x0066cb (0x2961 bytes) 000214 1717557005.765582 mapwrite 0x005a4c thru 0x013325 (0xd8da bytes) 000216 1717557005.805376 read 0x003b31 thru 0x00b19b (0x766b bytes) 000219 1717557005.807061 MIRROR_EXTEND to 3 mirrors 000220 1717557005.914636 read 0x00cede thru 0x013325 (0x6448 bytes) 000221 1717557005.920240 MIRROR_EXTEND to 4 mirrors 000223 1717557006.037288 trunc from 0x013326 to 0x01ff23 (0xcbfe bytes) 000224 1717557006.063080 mapread 0x003cc2 thru 0x00d3c6 (0x9705 bytes) 000226 1717557006.069188 mapread 0x011a34 thru 0x015661 (0x3c2e bytes) 000228 1717557006.072128 read 0x01e4a0 thru 0x01ff23 (0x1a84 bytes) 000230 1717557006.073751 write 0x01b27a thru 0x02629e (0xb025 bytes) 000231 1717557006.074085 read 0x01f853 thru 0x02629e (0x6a4c bytes) 000232 1717557006.074258 trunc from 0x02629f to 0x033ef3 (0xdc55 bytes) 000236 1717557006.078995 mapread 0x01a2cf thru 0x01eea9 (0x4bdb bytes) 000238 1717557006.080906 MIRROR_RESYNC 4 mirrors 000239 1717557006.227059 write 0x01472f thru 0x01b01a (0x68ec bytes) 000241 1717557006.245521 mapwrite 0x027e38 thru 0x035193 (0xd35c bytes) 000243 1717557006.280795 trunc from 0x035194 to 0x033b94 (0x15ff bytes) 000245 1717557006.285362 mapwrite 0x013a26 thru 0x01b713 (0x7cee bytes) 000247 1717557006.312498 write 0x023103 thru 0x02c3d6 (0x92d4 bytes) 000248 1717557006.312909 mapwrite 0x03ff45 thru 0x03ffff (0xbb bytes) 000249 1717557006.340304 trunc from 0x040000 to 0x0044bb (0x3bb44 bytes) 000250 1717557006.343948 read 0x000fbc thru 0x0019d8 (0xa1d bytes) 000252 1717557006.347115 MIRROR_EXTEND to 5 mirrors 000253 1717557006.482152 write 0x00f0ba thru 0x0136a1 (0x45e8 bytes) 000254 1717557006.504912 MIRROR_RESYNC 5 mirrors 000255 1717557006.690991 mapwrite 0x0117b1 thru 0x01d194 (0xb9e4 bytes) 000256 1717557006.729234 write 0x01c228 thru 0x02291b (0x66f4 bytes) 000261 1717557006.731671 MIRROR_EXTEND to 6 mirrors 000264 1717557006.833778 trunc from 0x02291c to 0x02fa77 (0xd15c bytes) 000266 1717557006.854228 MIRROR_RESYNC 6 mirrors 000267 1717557007.069029 write 0x00ce88 thru 0x016c50 (0x9dc9 bytes) 000268 1717557007.084187 mapwrite 0x02fb0b thru 0x03a990 (0xae86 bytes) 000269 1717557007.110663 MIRROR_EXTEND to 7 mirrors 000270 1717557007.228599 mapwrite 0x014c0f thru 0x019865 (0x4c57 bytes) 000272 1717557007.278040 read 0x0267b5 thru 0x032fcb (0xc817 bytes) 000273 1717557007.280823 mapwrite 0x034ea6 thru 0x03ee3d (0x9f98 bytes) 000274 1717557007.306302 write 0x036b83 thru 0x03ffff (0x947d bytes) 000275 1717557007.306631 mapread 0x03485b thru 0x03ffff (0xb7a5 bytes) 000276 1717557007.306884 read 0x039764 thru 0x03bfe9 (0x2886 bytes) 000277 1717557007.308132 MIRROR_RESYNC 7 mirrors 000278 1717557007.488649 MIRROR_EXTEND to 8 mirrors 000279 1717557007.628311 mapread 0x034d85 thru 0x03ffff (0xb27b bytes) 000280 1717557007.637077 mapread 0x031d40 thru 0x03b10b (0x93cc bytes) 000281 1717557007.641494 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000282 1717557007.721561 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000283 1717557007.790128 mapread 0x016bb1 thru 0x0205e9 (0x9a39 bytes) 000285 1717557007.796263 mapread 0x02d678 thru 0x036293 (0x8c1c bytes) 000286 1717557007.800154 mapwrite 0x016034 thru 0x01aa15 (0x49e2 bytes) 000287 1717557007.845351 mapread 0x02690d thru 0x02d1e1 (0x68d5 bytes) 000288 1717557007.847955 write 0x01b0b7 thru 0x026e3c (0xbd86 bytes) 000290 1717557007.848561 read 0x02a70c thru 0x038ad1 (0xe3c6 bytes) 000291 1717557007.849859 mapread 0x031e4d thru 0x03f7be (0xd972 bytes) 000293 1717557007.852658 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000294 1717557007.910662 mapwrite 0x01b5dd thru 0x01fe34 (0x4858 bytes) 000295 1717557007.936011 MIRROR_RESYNC 5 mirrors 000296 1717557008.111750 mapwrite 0x033e52 thru 0x03ffff (0xc1ae bytes) 000298 1717557008.162166 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000299 1717557008.216366 read 0x010c8b thru 0x017a1e (0x6d94 bytes) 000300 1717557008.218433 read 0x015d2a thru 0x01a7cb (0x4aa2 bytes) 000302 1717557008.222177 write 0x028b0f thru 0x033b2b (0xb01d bytes) 000303 1717557008.222562 write 0x03b551 thru 0x03cd87 (0x1837 bytes) 000305 1717557008.222863 mapwrite 0x01e07a thru 0x01e1a4 (0x12b bytes) 000306 1717557008.235520 write 0x013233 thru 0x015408 (0x21d6 bytes) 000307 1717557008.235797 trunc from 0x040000 to 0x02cbb4 (0x1344b bytes) 000308 1717557008.239176 write 0x00a0b8 thru 0x01823a (0xe183 bytes) 000310 1717557008.239824 mapread 0x00ef9e thru 0x00f363 (0x3c6 bytes) 000311 1717557008.240139 write 0x02f023 thru 0x031f15 (0x2ef3 bytes) 000312 1717557008.240467 read 0x02d41f thru 0x02f4b0 (0x2092 bytes) 000313 1717557008.242310 trunc from 0x031f16 to 0x0249e5 (0xd530 bytes) 000314 1717557008.246726 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000315 1717557008.298572 read 0x008c5e thru 0x00db57 (0x4efa bytes) 000317 1717557008.300797 mapread 0x00cb26 thru 0x0179b2 (0xae8d bytes) 000321 1717557008.305893 mapread 0x00a017 thru 0x01383c (0x9826 bytes) 000322 1717557008.306143 write 0x011e8f thru 0x01a729 (0x889b bytes) 000323 1717557008.306525 trunc from 0x0249e6 to 0x0038c2 (0x21123 bytes) 000324 1717557008.311422 write 0x036b76 thru 0x03ffff (0x948a bytes) 000326 1717557008.312122 read 0x0251ce thru 0x026209 (0x103c bytes) 000330 1717557008.316561 mapread 0x03fc73 thru 0x03ffff (0x38d bytes) 000333 1717557008.317191 mapwrite 0x03937c thru 0x03f818 (0x649d bytes) 000335 1717557008.333474 read 0x03e90c thru 0x03ffff (0x16f4 bytes) 000338 1717557008.334021 mapwrite 0x00d8c9 thru 0x019e27 (0xc55f bytes) 000340 1717557008.350913 mapread 0x018eda thru 0x0226eb (0x9812 bytes) 000341 1717557008.352541 MIRROR_EXTEND to 4 mirrors 000342 1717557008.463813 trunc from 0x040000 to 0x03bb57 (0x44a8 bytes) 000343 1717557008.485593 mapread 0x03647f thru 0x03adc1 (0x4943 bytes) 000344 1717557008.489342 read 0x029f12 thru 0x02af03 (0xff2 bytes) 000345 1717557008.490425 mapwrite 0x03f8cf thru 0x03ffff (0x731 bytes) 000347 1717557008.509482 mapwrite 0x01997a thru 0x01c711 (0x2d98 bytes) 000349 1717557008.523069 mapwrite 0x033ff2 thru 0x03fca3 (0xbcb2 bytes) 000351 1717557008.537501 write 0x039eb1 thru 0x03e12b (0x427b bytes) 000352 1717557008.537761 mapwrite 0x02f9d0 thru 0x030a1c (0x104d bytes) 000353 1717557008.553153 write 0x01c48b thru 0x01cb80 (0x6f6 bytes) 000354 1717557008.553490 write 0x00a9f2 thru 0x00be95 (0x14a4 bytes) 000357 1717557008.556531 mapread 0x025f8b thru 0x0305f8 (0xa66e bytes) 000360 1717557008.558484 trunc from 0x040000 to 0x03375d (0xc8a2 bytes) 000361 1717557008.563516 MIRROR_EXTEND to 5 mirrors 000362 1717557008.669452 read 0x0010ca thru 0x010f9c (0xfed3 bytes) 000363 1717557008.672794 trunc from 0x03375e to 0x0112b5 (0x224a8 bytes) 000364 1717557008.698097 mapwrite 0x007d0b thru 0x00d61f (0x5915 bytes) 000365 1717557008.719411 trunc from 0x0112b6 to 0x00ffba (0x12fb bytes) 000366 1717557008.725217 MIRROR_EXTEND to 6 mirrors 000367 1717557008.827378 trunc from 0x00ffbb to 0x018b8d (0x8bd3 bytes) 000368 1717557008.850553 write 0x02d4ad thru 0x03938d (0xbee1 bytes) 000369 1717557008.851265 write 0x0332bc thru 0x036257 (0x2f9c bytes) 000370 1717557008.851564 mapwrite 0x0035a6 thru 0x006f0e (0x3969 bytes) 000371 1717557008.874444 read 0x006b60 thru 0x01046e (0x990f bytes) 000372 1717557008.874754 mapwrite 0x0066a0 thru 0x00b709 (0x506a bytes) 000374 1717557008.895006 mapwrite 0x01307e thru 0x01e800 (0xb783 bytes) 000375 1717557008.916726 read 0x021db7 thru 0x02bfc5 (0xa20f bytes) 000377 1717557008.919508 trunc from 0x03938e to 0x000a82 (0x3890b bytes) 000379 1717557008.925044 trunc from 0x000a83 to 0x0211b8 (0x20736 bytes) 000380 1717557008.928390 read 0x01edd1 thru 0x0211b8 (0x23e8 bytes) 000381 1717557008.931939 write 0x015d91 thru 0x01e61e (0x888e bytes) 000382 1717557008.933412 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000388 1717557008.992500 read 0x00f237 thru 0x018f65 (0x9d2f bytes) 000389 1717557008.995577 mapread 0x004ee9 thru 0x005d8c (0xea4 bytes) 000390 1717557008.999082 MIRROR_EXTEND to 6 mirrors 000391 1717557009.124251 write 0x01532f thru 0x01bb9c (0x686e bytes) 000392 1717557009.144127 write 0x006c87 thru 0x0167d7 (0xfb51 bytes) 000393 1717557009.147214 mapread 0x012059 thru 0x0154c8 (0x3470 bytes) 000394 1717557009.147538 trunc from 0x0211b9 to 0x014799 (0xca1f bytes) 000395 1717557009.153243 write 0x037277 thru 0x03a67b (0x3405 bytes) 000396 1717557009.153718 read 0x00077b thru 0x00b265 (0xaaeb bytes) 000398 1717557009.155614 trunc from 0x03a67c to 0x02fe4d (0xa82e bytes) 000402 1717557009.159788 trunc from 0x02fe4e to 0x00aef9 (0x24f54 bytes) 000403 1717557009.164528 mapwrite 0x0236fc thru 0x02f190 (0xba95 bytes) 000405 1717557009.191175 trunc from 0x02f191 to 0x01682b (0x18965 bytes) 000408 1717557009.195903 read 0x001b7a thru 0x009b5f (0x7fe6 bytes) 000411 1717557009.196377 mapwrite 0x00093a thru 0x005f43 (0x560a bytes) 000412 1717557009.213773 write 0x01f8c3 thru 0x02d562 (0xdca0 bytes) 000414 1717557009.215769 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000415 1717557009.275640 mapread 0x01fe67 thru 0x024c2a (0x4dc4 bytes) 000417 1717557009.281007 mapwrite 0x00e911 thru 0x013f7a (0x566a bytes) 000418 1717557009.306785 mapwrite 0x03cf23 thru 0x03ffff (0x30dd bytes) 000419 1717557009.335347 mapwrite 0x01f7e3 thru 0x02169b (0x1eb9 bytes) 000422 1717557009.350711 write 0x03e7ed thru 0x03ffff (0x1813 bytes) 000423 1717557009.351014 trunc from 0x040000 to 0x004ee0 (0x3b11f bytes) 000424 1717557009.355890 MIRROR_RESYNC 5 mirrors 000426 1717557009.533764 mapwrite 0x00a5a6 thru 0x00b8cd (0x1328 bytes) 000428 1717557009.576956 mapread 0x0070a0 thru 0x00b8cd (0x482e bytes) 000429 1717557009.579466 read 0x00a594 thru 0x00b8cd (0x133a bytes) 000430 1717557009.579621 read 0x00b5dc thru 0x00b8cd (0x2f2 bytes) 000431 1717557009.581031 MIRROR_EXTEND to 6 mirrors 000432 1717557009.676571 mapwrite 0x0069bb thru 0x010c95 (0xa2db bytes) 000434 1717557009.726890 trunc from 0x010c96 to 0x01cd48 (0xc0b3 bytes) 000435 1717557009.731986 mapread 0x005cdb thru 0x012c63 (0xcf89 bytes) 000436 1717557009.734553 mapread 0x010c78 thru 0x01cd48 (0xc0d1 bytes) 000437 1717557009.736094 mapwrite 0x02e948 thru 0x032489 (0x3b42 bytes) 000438 1717557009.764251 trunc from 0x03248a to 0x02438e (0xe0fb bytes) 000439 1717557009.769233 write 0x02d811 thru 0x0370fe (0x98ee bytes) 000441 1717557009.769896 mapwrite 0x0028e8 thru 0x003834 (0xf4d bytes) 000442 1717557009.790489 mapwrite 0x03ace1 thru 0x03ffff (0x531f bytes) 000444 1717557009.826372 mapread 0x009424 thru 0x0134b6 (0xa093 bytes) 000448 1717557009.827198 mapwrite 0x03cda1 thru 0x03ffff (0x325f bytes) 000449 1717557009.844575 write 0x036149 thru 0x03ffff (0x9eb7 bytes) 000450 1717557009.845003 mapwrite 0x001425 thru 0x004b89 (0x3765 bytes) 000451 1717557009.860545 mapwrite 0x038d02 thru 0x03ffff (0x72fe bytes) 000452 1717557009.875552 trunc from 0x040000 to 0x00f726 (0x308d9 bytes) 000453 1717557009.879255 write 0x01877e thru 0x01bf29 (0x37ac bytes) 000458 1717557009.880026 mapread 0x01991b thru 0x01bf29 (0x260f bytes) 000459 1717557009.880354 write 0x02d03f thru 0x0305c0 (0x3582 bytes) 000460 1717557009.880617 write 0x02d999 thru 0x035251 (0x78b9 bytes) 000461 1717557009.882180 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000462 1717557009.936311 mapread 0x012848 thru 0x014cb2 (0x246b bytes) 000463 1717557009.941344 read 0x0165d9 thru 0x021a9c (0xb4c4 bytes) 000464 1717557009.943691 mapread 0x01be38 thru 0x021dfa (0x5fc3 bytes) 000466 1717557009.944115 trunc from 0x035252 to 0x021397 (0x13eba bytes) 000467 1717557009.948102 write 0x009a65 thru 0x016805 (0xcda1 bytes) 000468 1717557009.950419 trunc from 0x021398 to 0x03433a (0x12fa3 bytes) 000474 1717557009.955954 write 0x0249c3 thru 0x0303ee (0xba2c bytes) 000480 1717557009.960942 trunc from 0x03433b to 0x015c52 (0x1e6e8 bytes) 000482 1717557009.965424 mapwrite 0x000423 thru 0x00ecc2 (0xe8a0 bytes) 000483 1717557009.992748 trunc from 0x015c53 to 0x0146bf (0x1593 bytes) 000484 1717557009.997329 mapwrite 0x028451 thru 0x035b01 (0xd6b1 bytes) 000485 1717557010.025773 MIRROR_EXTEND to 6 mirrors 000486 1717557010.129812 mapread 0x0042b4 thru 0x00707f (0x2dcc bytes) 000488 1717557010.137237 mapread 0x0300db thru 0x033dcf (0x3cf5 bytes) 000489 1717557010.141652 read 0x0133ed thru 0x01ead3 (0xb6e7 bytes) 000490 1717557010.144767 write 0x03891f thru 0x03ffff (0x76e1 bytes) 000491 1717557010.158831 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000493 1717557010.210245 MIRROR_RESYNC 5 mirrors 000494 1717557010.363637 MIRROR_RESYNC 5 mirrors 000495 1717557010.413268 write 0x01f5cf thru 0x02f1c3 (0xfbf5 bytes) 000496 1717557010.429552 mapwrite 0x00ea4d thru 0x01b84b (0xcdff bytes) 000497 1717557010.451202 mapwrite 0x01ab39 thru 0x0227f0 (0x7cb8 bytes) 000498 1717557010.465905 mapwrite 0x01f422 thru 0x02a80b (0xb3ea bytes) 000503 1717557010.480046 mapwrite 0x015af2 thru 0x01f6f1 (0x9c00 bytes) 000505 1717557010.495500 mapwrite 0x039c88 thru 0x03ffff (0x6378 bytes) 000506 1717557010.510544 mapread 0x03c7b6 thru 0x03ffff (0x384a bytes) 000507 1717557010.510845 read 0x03a114 thru 0x03ffff (0x5eec bytes) 000508 1717557010.510978 write 0x025d78 thru 0x02b947 (0x5bd0 bytes) 000510 1717557010.511316 read 0x032091 thru 0x03ffff (0xdf6f bytes) 000511 1717557010.511511 read 0x003208 thru 0x0101ec (0xcfe5 bytes) 000513 1717557010.513981 read 0x03fcc9 thru 0x03ffff (0x337 bytes) 000514 1717557010.514251 mapread 0x036eb4 thru 0x03ffff (0x914c bytes) 000515 1717557010.514590 trunc from 0x040000 to 0x0339af (0xc650 bytes) 000517 1717557010.519987 read 0x02917b thru 0x033683 (0xa509 bytes) 000518 1717557010.520193 mapwrite 0x020cf5 thru 0x025da5 (0x50b1 bytes) 000519 1717557010.537684 mapwrite 0x02c2f5 thru 0x03281a (0x6526 bytes) 000521 1717557010.554518 read 0x005dfc thru 0x009588 (0x378d bytes) 000522 1717557010.554708 mapwrite 0x000528 thru 0x0008bc (0x395 bytes) 000523 1717557010.570044 write 0x0099b9 thru 0x016f9e (0xd5e6 bytes) 000524 1717557010.570509 read 0x02e738 thru 0x02eee1 (0x7aa bytes) 000526 1717557010.570865 mapread 0x02481c thru 0x025d80 (0x1565 bytes) 000528 1717557010.571307 trunc from 0x0339b0 to 0x0071d5 (0x2c7da bytes) 000530 1717557010.578148 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000531 1717557010.623177 write 0x00e6d8 thru 0x017496 (0x8dbf bytes) 000532 1717557010.623698 write 0x00ac4a thru 0x016c25 (0xbfdc bytes) 000534 1717557010.625320 mapread 0x008097 thru 0x00c2c8 (0x4232 bytes) 000535 1717557010.628420 trunc from 0x017497 to 0x028f7b (0x11ae5 bytes) 000536 1717557010.631916 write 0x015e0e thru 0x0252e6 (0xf4d9 bytes) 000537 1717557010.633724 trunc from 0x028f7c to 0x0320f1 (0x9176 bytes) 000538 1717557010.637171 mapread 0x00bc4f thru 0x00bce6 (0x98 bytes) 000539 1717557010.637708 mapread 0x019aee thru 0x0205bb (0x6ace bytes) 000540 1717557010.638021 trunc from 0x0320f2 to 0x019956 (0x1879b bytes) 000541 1717557010.641794 mapread 0x0087c4 thru 0x009890 (0x10cd bytes) 000542 1717557010.642115 read 0x018e44 thru 0x019956 (0xb13 bytes) 000543 1717557010.642317 mapread 0x0006a9 thru 0x00eaf5 (0xe44d bytes) 000544 1717557010.644320 mapwrite 0x00a8f8 thru 0x0145c6 (0x9ccf bytes) 000547 1717557010.662369 mapwrite 0x00fabb thru 0x01b6d9 (0xbc1f bytes) 000548 1717557010.685838 MIRROR_RESYNC 4 mirrors 000549 1717557010.847520 MIRROR_RESYNC 4 mirrors 000550 1717557010.883174 read 0x0125ab thru 0x01b6d9 (0x912f bytes) 000551 1717557010.885412 write 0x02ebb8 thru 0x03114a (0x2593 bytes) 000556 1717557010.900662 read 0x00ffb4 thru 0x010df6 (0xe43 bytes) 000562 1717557010.904885 trunc from 0x03114b to 0x039c70 (0x8b26 bytes) 000563 1717557010.915133 read 0x010dd2 thru 0x018a93 (0x7cc2 bytes) 000565 1717557010.915533 mapwrite 0x017597 thru 0x01a3b4 (0x2e1e bytes) 000567 1717557010.934442 trunc from 0x039c71 to 0x036f15 (0x2d5b bytes) 000568 1717557010.938450 mapread 0x0056df thru 0x0102ea (0xac0c bytes) 000569 1717557010.940085 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000571 1717557010.990026 mapread 0x00f56b thru 0x01dfb9 (0xea4f bytes) 000573 1717557010.996955 mapwrite 0x033b72 thru 0x03b826 (0x7cb5 bytes) 000574 1717557011.020774 mapread 0x034112 thru 0x037133 (0x3022 bytes) 000575 1717557011.021116 write 0x02e491 thru 0x039872 (0xb3e2 bytes) 000577 1717557011.021662 trunc from 0x03b827 to 0x00e239 (0x2d5ed bytes) 000580 1717557011.025995 write 0x025870 thru 0x035183 (0xf914 bytes) 000581 1717557011.026508 trunc from 0x035184 to 0x02a5e0 (0xaba3 bytes) 000584 1717557011.030522 trunc from 0x02a5e1 to 0x0077d6 (0x22e0a bytes) 000585 1717557011.035590 trunc from 0x0077d7 to 0x00069a (0x713c bytes) 000586 1717557011.040599 mapwrite 0x00d3d0 thru 0x01991e (0xc54f bytes) 000587 1717557011.071387 mapwrite 0x031544 thru 0x031d3f (0x7fc bytes) 000588 1717557011.093136 trunc from 0x031d40 to 0x029a74 (0x82cb bytes) 000591 1717557011.096782 mapread 0x013a10 thru 0x020ff9 (0xd5ea bytes) 000592 1717557011.099345 write 0x036306 thru 0x03bfbb (0x5cb6 bytes) 000595 1717557011.101176 MIRROR_RESYNC 3 mirrors 000596 1717557011.218206 mapread 0x018f76 thru 0x022e34 (0x9ebf bytes) 000597 1717557011.221800 mapwrite 0x013385 thru 0x01aab8 (0x7734 bytes) 000599 1717557011.270430 mapwrite 0x0178f4 thru 0x023d8c (0xc499 bytes) 000600 1717557011.290372 mapwrite 0x008edf thru 0x009cba (0xddc bytes) 000601 1717557011.310231 mapwrite 0x021d0d thru 0x02283c (0xb30 bytes) 000605 1717557011.325827 read 0x035446 thru 0x03bfbb (0x6b76 bytes) 000606 1717557011.328285 write 0x005720 thru 0x011593 (0xbe74 bytes) 000607 1717557011.328660 write 0x0380f8 thru 0x03ba09 (0x3912 bytes) 000609 1717557011.329022 trunc from 0x03bfbc to 0x023ee0 (0x180db bytes) 000610 1717557011.337849 read 0x010f17 thru 0x01e09f (0xd189 bytes) 000611 1717557011.338066 mapwrite 0x0065e7 thru 0x01028c (0x9ca6 bytes) 000613 1717557011.354421 write 0x02db66 thru 0x030d6a (0x3205 bytes) 000614 1717557011.354843 write 0x01dd19 thru 0x01f6bd (0x19a5 bytes) 000617 1717557011.355325 mapread 0x00d02a thru 0x00d4e7 (0x4be bytes) 000621 1717557011.355958 mapwrite 0x03763b thru 0x03ffff (0x89c5 bytes) 000622 1717557011.381865 write 0x0169d3 thru 0x01800b (0x1639 bytes) 000625 1717557011.382399 read 0x00501a thru 0x00ac24 (0x5c0b bytes) 000626 1717557011.382550 mapread 0x02813c thru 0x02f582 (0x7447 bytes) 000628 1717557011.383007 read 0x02f9d0 thru 0x032ab6 (0x30e7 bytes) 000629 1717557011.383203 write 0x028ae9 thru 0x02bfe7 (0x34ff bytes) 000630 1717557011.383465 mapread 0x00dc95 thru 0x00f5a4 (0x1910 bytes) 000631 1717557011.383671 write 0x00c207 thru 0x011035 (0x4e2f bytes) 000633 1717557011.383954 mapread 0x03304f thru 0x03d82f (0xa7e1 bytes) 000634 1717557011.384235 write 0x025673 thru 0x02d8ac (0x823a bytes) 000636 1717557011.384634 mapwrite 0x0161fa thru 0x022267 (0xc06e bytes) 000639 1717557011.403790 trunc from 0x040000 to 0x024174 (0x1be8b bytes) 000640 1717557011.410070 write 0x0376ee thru 0x03ffff (0x8912 bytes) 000641 1717557011.410445 mapwrite 0x02b3b3 thru 0x03940a (0xe058 bytes) 000642 1717557011.433076 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000644 1717557011.481117 mapread 0x02f31c thru 0x0330ad (0x3d92 bytes) 000645 1717557011.484992 trunc from 0x040000 to 0x001667 (0x3e998 bytes) 000646 1717557011.488405 mapwrite 0x008a59 thru 0x0115b7 (0x8b5f bytes) 000648 1717557011.514849 trunc from 0x0115b8 to 0x00b9c0 (0x5bf7 bytes) 000652 1717557011.518799 trunc from 0x00b9c1 to 0x02aa2e (0x1f06e bytes) 000653 1717557011.522121 trunc from 0x02aa2f to 0x0113fc (0x19632 bytes) 000654 1717557011.525972 mapread 0x00e8a5 thru 0x0113fc (0x2b58 bytes) 000658 1717557011.530070 MIRROR_RESYNC 2 mirrors 000659 1717557011.619983 write 0x0313a9 thru 0x03efbb (0xdc13 bytes) 000661 1717557011.636207 mapwrite 0x02581b thru 0x033d7f (0xe565 bytes) 000662 1717557011.661878 mapwrite 0x0190b6 thru 0x020cc8 (0x7c13 bytes) 000664 1717557011.681780 trunc from 0x03efbc to 0x01eecb (0x200f0 bytes) 000665 1717557011.691985 read 0x00f191 thru 0x01d236 (0xe0a6 bytes) 000667 1717557011.693888 write 0x0387f6 thru 0x03f5db (0x6de6 bytes) 000669 1717557011.694375 trunc from 0x03f5dc to 0x01e456 (0x21185 bytes) 000670 1717557011.697526 mapwrite 0x0299e9 thru 0x034095 (0xa6ad bytes) 000672 1717557011.721867 write 0x0287ab thru 0x032165 (0x99bb bytes) 000674 1717557011.722355 mapwrite 0x006119 thru 0x014cef (0xebd7 bytes) 000676 1717557011.740540 write 0x034a83 thru 0x03df00 (0x947e bytes) 000678 1717557011.741165 mapread 0x022659 thru 0x0263af (0x3d57 bytes) 000679 1717557011.741418 mapwrite 0x01a61c thru 0x02064c (0x6031 bytes) 000682 1717557011.759647 MIRROR_RESYNC 2 mirrors 000683 1717557011.846532 trunc from 0x03df01 to 0x009c63 (0x3429d bytes) 000684 1717557011.864195 mapread 0x00694a thru 0x008560 (0x1c17 bytes) 000685 1717557011.868534 trunc from 0x009c64 to 0x02fb1d (0x25eba bytes) 000687 1717557011.872889 write 0x03e985 thru 0x03ffff (0x167b bytes) 000688 1717557011.874680 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000690 1717557011.911250 read 0x016ac3 thru 0x0261db (0xf719 bytes) 000691 1717557011.914484 mapwrite 0x02484c thru 0x02d5f6 (0x8dab bytes) 000692 1717557011.939475 mapwrite 0x02a576 thru 0x037967 (0xd3f2 bytes) 000694 1717557011.956134 trunc from 0x040000 to 0x0193af (0x26c50 bytes) 000696 1717557011.959874 trunc from 0x0193b0 to 0x006710 (0x12c9f bytes) 000697 1717557011.963076 read 0x0009f6 thru 0x00574b (0x4d56 bytes) 000698 1717557011.965072 mapread 0x000e67 thru 0x006710 (0x58aa bytes) 000700 1717557011.965548 trunc from 0x006711 to 0x034b59 (0x2e449 bytes) 000704 1717557011.968797 trunc from 0x034b5a to 0x0053eb (0x2f76e bytes) 000705 1717557011.971834 trunc from 0x0053ec to 0x01d164 (0x17d79 bytes) 000706 1717557011.974498 read 0x0049da thru 0x004fa1 (0x5c8 bytes) 000708 1717557011.974861 mapread 0x014016 thru 0x01d164 (0x914f bytes) 000709 1717557011.977937 mapread 0x011c70 thru 0x015cfd (0x408e bytes) 000712 1717557011.978453 mapwrite 0x038b9e thru 0x038fb1 (0x414 bytes) 000714 1717557011.998783 mapwrite 0x00d61c thru 0x01baea (0xe4cf bytes) 000715 1717557012.013487 mapread 0x00779c thru 0x011443 (0x9ca8 bytes) 000717 1717557012.014122 trunc from 0x038fb2 to 0x03305c (0x5f55 bytes) 000718 1717557012.017851 write 0x029bbf thru 0x035293 (0xb6d5 bytes) 000722 1717557012.018530 write 0x0213a9 thru 0x0219c9 (0x621 bytes) 000723 1717557012.018776 trunc from 0x035294 to 0x03a192 (0x4eff bytes) 000727 1717557012.022247 read 0x030582 thru 0x03a0b7 (0x9b36 bytes) 000729 1717557012.025196 mapwrite 0x0286bc thru 0x02c3d5 (0x3d1a bytes) 000731 1717557012.042385 mapread 0x02c811 thru 0x03053b (0x3d2b bytes) 000732 1717557012.042693 read 0x01a82e thru 0x025d6b (0xb53e bytes) 000734 1717557012.042988 read 0x01c2e8 thru 0x025e54 (0x9b6d bytes) 000736 1717557012.043238 trunc from 0x03a193 to 0x02e980 (0xb812 bytes) 000739 1717557012.046972 mapwrite 0x023d7a thru 0x02abab (0x6e32 bytes) 000740 1717557012.062660 trunc from 0x02e981 to 0x00ba14 (0x22f6c bytes) 000742 1717557012.067331 MIRROR_EXTEND to 2 mirrors 000743 1717557012.148149 trunc from 0x00ba15 to 0x03633a (0x2a926 bytes) 000744 1717557012.167469 mapwrite 0x009fbe thru 0x014a75 (0xaab8 bytes) 000746 1717557012.194758 read 0x017897 thru 0x02628e (0xe9f8 bytes) 000747 1717557012.197085 mapwrite 0x01e2b2 thru 0x01e2e2 (0x31 bytes) 000749 1717557012.208732 mapwrite 0x026f28 thru 0x03031c (0x93f5 bytes) 000750 1717557012.225350 mapwrite 0x00166f thru 0x00b396 (0x9d28 bytes) 000751 1717557012.241986 read 0x02449a thru 0x02542a (0xf91 bytes) 000752 1717557012.242182 mapread 0x033826 thru 0x03633a (0x2b15 bytes) 000754 1717557012.244136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000755 1717557012.277351 mapwrite 0x0131f9 thru 0x022a0f (0xf817 bytes) 000758 1717557012.296261 mapread 0x032bc1 thru 0x03633a (0x377a bytes) 000759 1717557012.298681 write 0x01c4e7 thru 0x01cd0e (0x828 bytes) 000760 1717557012.298928 mapwrite 0x02caa4 thru 0x03c5e2 (0xfb3f bytes) 000761 1717557012.321645 trunc from 0x03c5e3 to 0x023ec8 (0x1871a bytes) 000762 1717557012.325368 trunc from 0x023ec9 to 0x03a01c (0x16154 bytes) 000763 1717557012.328519 trunc from 0x03a01d to 0x011b1f (0x284fd bytes) 000765 1717557012.332194 trunc from 0x011b20 to 0x0210ed (0xf5ce bytes) 000766 1717557012.335293 trunc from 0x0210ee to 0x022d11 (0x1c24 bytes) 000769 1717557012.338317 mapread 0x00efbc thru 0x01ee35 (0xfe7a bytes) 000770 1717557012.342243 write 0x000ee9 thru 0x006807 (0x591f bytes) 000771 1717557012.344155 mapread 0x002441 thru 0x002755 (0x315 bytes) 000772 1717557012.344354 trunc from 0x022d12 to 0x00e252 (0x14abf bytes) 000773 1717557012.347262 write 0x03b575 thru 0x03ffff (0x4a8b bytes) 000774 1717557012.347652 read 0x025c8c thru 0x02ed5f (0x90d4 bytes) 000776 1717557012.349696 write 0x01c91d thru 0x0208d9 (0x3fbd bytes) 000781 1717557012.352413 write 0x00d645 thru 0x01178e (0x414a bytes) 000782 1717557012.353814 mapread 0x03eed1 thru 0x03ffff (0x112f bytes) 000785 1717557012.354432 mapwrite 0x02f755 thru 0x039334 (0x9be0 bytes) 000788 1717557012.374156 mapread 0x035abf thru 0x03ffff (0xa541 bytes) 000789 1717557012.374475 write 0x022745 thru 0x02a898 (0x8154 bytes) 000790 1717557012.374809 read 0x0052e3 thru 0x008e38 (0x3b56 bytes) 000791 1717557012.374998 write 0x010eda thru 0x014d17 (0x3e3e bytes) 000795 1717557012.377139 write 0x00a8a6 thru 0x01238c (0x7ae7 bytes) 000797 1717557012.377550 mapread 0x0158a4 thru 0x015efd (0x65a bytes) 000798 1717557012.377770 mapwrite 0x0170e9 thru 0x023725 (0xc63d bytes) 000803 1717557012.396005 write 0x027e17 thru 0x0282cb (0x4b5 bytes) 000804 1717557012.396297 mapread 0x02d968 thru 0x033c11 (0x62aa bytes) 000808 1717557012.396966 trunc from 0x040000 to 0x010c62 (0x2f39d bytes) 000811 1717557012.401754 mapread 0x0103de thru 0x010c62 (0x885 bytes) 000813 1717557012.402345 read 0x002e5e thru 0x010c62 (0xde05 bytes) 000816 1717557012.402984 mapread 0x009325 thru 0x009904 (0x5e0 bytes) 000817 1717557012.403252 read 0x006cd9 thru 0x010c62 (0x9f8a bytes) 000820 1717557012.403788 write 0x036899 thru 0x0396f2 (0x2e5a bytes) 000822 1717557012.404312 read 0x0027e5 thru 0x00e297 (0xbab3 bytes) 000825 1717557012.404799 write 0x021041 thru 0x02986b (0x882b bytes) 000826 1717557012.408088 trunc from 0x0396f3 to 0x033775 (0x5f7d bytes) 000827 1717557012.411293 mapread 0x0146d3 thru 0x019f92 (0x58c0 bytes) 000828 1717557012.411632 trunc from 0x033776 to 0x0048a5 (0x2eed0 bytes) 000829 1717557012.414959 mapread 0x003aee thru 0x0048a5 (0xdb8 bytes) 000830 1717557012.416313 MIRROR_EXTEND to 2 mirrors 000831 1717557012.494375 read 0x000e91 thru 0x0048a5 (0x3a15 bytes) 000832 1717557012.496385 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000834 1717557012.527381 mapread 0x0042f1 thru 0x0048a5 (0x5b5 bytes) 000835 1717557012.528802 mapwrite 0x02b5dc thru 0x03ad6c (0xf791 bytes) 000838 1717557012.558800 write 0x03a6a9 thru 0x03ffff (0x5957 bytes) 000839 1717557012.559177 read 0x00be6b thru 0x00d86e (0x1a04 bytes) 000844 1717557012.561182 mapwrite 0x006ed7 thru 0x00bde0 (0x4f0a bytes) 000845 1717557012.575744 mapwrite 0x00b58d thru 0x013f22 (0x8996 bytes) 000846 1717557012.590799 read 0x0113df thru 0x01cec0 (0xbae2 bytes) 000848 1717557012.593289 trunc from 0x040000 to 0x01accf (0x25330 bytes) 000850 1717557012.596786 write 0x004218 thru 0x00f599 (0xb382 bytes) 000852 1717557012.597193 trunc from 0x01acd0 to 0x02a9c3 (0xfcf4 bytes) 000853 1717557012.601205 MIRROR_EXTEND to 2 mirrors 000854 1717557012.706863 read 0x02104b thru 0x02a0ac (0x9062 bytes) 000855 1717557012.709235 write 0x03da02 thru 0x03ef44 (0x1543 bytes) 000857 1717557012.722800 read 0x007c67 thru 0x016911 (0xecab bytes) 000858 1717557012.726402 mapread 0x0262de thru 0x027dad (0x1ad0 bytes) 000859 1717557012.727527 MIRROR_EXTEND to 3 mirrors 000860 1717557012.827057 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000862 1717557012.868608 mapwrite 0x019333 thru 0x022001 (0x8ccf bytes) 000863 1717557012.890783 write 0x00ca48 thru 0x018368 (0xb921 bytes) 000864 1717557012.893308 trunc from 0x03ef45 to 0x028e78 (0x160cc bytes) 000865 1717557012.900676 read 0x013f8b thru 0x02303c (0xf0b2 bytes) 000867 1717557012.901118 write 0x03e17c thru 0x03ffff (0x1e84 bytes) 000868 1717557012.901528 read 0x0031bf thru 0x00520c (0x204e bytes) 000869 1717557012.904463 mapwrite 0x03098d thru 0x03cb4d (0xc1c1 bytes) 000870 1717557012.920327 trunc from 0x040000 to 0x015002 (0x2affd bytes) 000871 1717557012.923831 read 0x012b3a thru 0x015002 (0x24c9 bytes) 000874 1717557012.924301 mapwrite 0x036ef4 thru 0x03ffff (0x910c bytes) 000875 1717557012.947458 mapwrite 0x03e1ca thru 0x03ffff (0x1e36 bytes) 000877 1717557012.960384 read 0x00ebec thru 0x0121ee (0x3603 bytes) 000882 1717557012.960944 read 0x02bf10 thru 0x039aab (0xdb9c bytes) 000883 1717557012.961093 write 0x0079cb thru 0x015701 (0xdd37 bytes) 000884 1717557012.961418 mapread 0x0016a2 thru 0x00c0fe (0xaa5d bytes) 000885 1717557012.961704 trunc from 0x040000 to 0x0249c0 (0x1b63f bytes) 000886 1717557012.965458 mapread 0x0165c9 thru 0x0244e8 (0xdf20 bytes) 000889 1717557012.966141 trunc from 0x0249c1 to 0x020978 (0x4048 bytes) 000891 1717557012.969462 mapwrite 0x00dc11 thru 0x010106 (0x24f6 bytes) 000892 1717557012.984634 write 0x02c3f2 thru 0x03a996 (0xe5a5 bytes) 000893 1717557012.985141 trunc from 0x03a997 to 0x00f21b (0x2b77b bytes) 000894 1717557012.989684 MIRROR_RESYNC 2 mirrors 000895 1717557013.071151 read 0x00a3a4 thru 0x00f21b (0x4e78 bytes) 000897 1717557013.073067 mapread 0x0044af thru 0x00cc0d (0x875f bytes) 000899 1717557013.076667 trunc from 0x00f21c to 0x01f843 (0x10628 bytes) 000900 1717557013.090467 trunc from 0x01f844 to 0x0263fa (0x6bb7 bytes) 000902 1717557013.094171 read 0x0142c3 thru 0x01e406 (0xa144 bytes) 000903 1717557013.097081 trunc from 0x0263fb to 0x002fc0 (0x2343a bytes) 000905 1717557013.100450 read 0x001edc thru 0x002fc0 (0x10e5 bytes) 000907 1717557013.100759 trunc from 0x002fc1 to 0x00d525 (0xa565 bytes) 000910 1717557013.103831 read 0x00b599 thru 0x00d525 (0x1f8d bytes) 000911 1717557013.107025 MIRROR_EXTEND to 3 mirrors 000912 1717557013.190748 read 0x00d371 thru 0x00d525 (0x1b5 bytes) 000913 1717557013.191523 mapwrite 0x02c3d5 thru 0x02ebbd (0x27e9 bytes) 000915 1717557013.226928 write 0x0256e8 thru 0x031382 (0xbc9b bytes) 000916 1717557013.230650 MIRROR_EXTEND to 4 mirrors 000917 1717557013.319018 mapwrite 0x03a27d thru 0x03ed4f (0x4ad3 bytes) 000918 1717557013.355627 trunc from 0x03ed50 to 0x03e4ca (0x885 bytes) 000919 1717557013.359212 mapread 0x02f5dd thru 0x03582d (0x6251 bytes) 000921 1717557013.361462 trunc from 0x03e4cb to 0x01f527 (0x1efa3 bytes) 000922 1717557013.364558 mapwrite 0x007584 thru 0x00934f (0x1dcc bytes) 000925 1717557013.381195 read 0x00951c thru 0x0107e6 (0x72cb bytes) 000926 1717557013.382714 MIRROR_RESYNC 4 mirrors 000928 1717557013.510857 mapwrite 0x025617 thru 0x02aa68 (0x5452 bytes) 000929 1717557013.547715 trunc from 0x02aa69 to 0x00c295 (0x1e7d3 bytes) 000930 1717557013.553531 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000932 1717557013.605687 trunc from 0x00c296 to 0x012863 (0x65ce bytes) 000934 1717557013.608971 read 0x00fd95 thru 0x012863 (0x2acf bytes) 000935 1717557013.610523 read 0x00e63b thru 0x012863 (0x4229 bytes) 000937 1717557013.612953 trunc from 0x012864 to 0x027ad8 (0x15275 bytes) 000938 1717557013.616251 mapwrite 0x01057c thru 0x01941d (0x8ea2 bytes) 000939 1717557013.635759 mapread 0x018791 thru 0x027ad8 (0xf348 bytes) 000940 1717557013.636101 mapread 0x0017c0 thru 0x005141 (0x3982 bytes) 000941 1717557013.637525 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000942 1717557013.684548 MIRROR_RESYNC 2 mirrors 000945 1717557013.770300 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x58c:0x0]: Invalid argument (22) 000946 1717557013.804642 trunc from 0x027ad9 to 0x0265f1 (0x14e7 bytes) 000949 1717557013.814770 mapwrite 0x02b6e6 thru 0x03979d (0xe0b8 bytes) 000950 1717557013.843545 mapwrite 0x0003c7 thru 0x00e574 (0xe1ae bytes) 000953 1717557013.863493 read 0x0050ff thru 0x00d226 (0x8128 bytes) 000954 1717557013.863743 mapwrite 0x03b830 thru 0x03f922 (0x40f3 bytes) 000958 1717557013.885030 write 0x00058e thru 0x00ad2c (0xa79f bytes) 000959 1717557013.885423 read 0x027300 thru 0x02febd (0x8bbe bytes) 000962 1717557013.885833 mapwrite 0x006633 thru 0x00ac79 (0x4647 bytes) 000963 1717557013.900601 read 0x003797 thru 0x00b932 (0x819c bytes) 000964 1717557013.900780 mapread 0x011434 thru 0x0115e8 (0x1b5 bytes) 000965 1717557013.903581 mapwrite 0x001373 thru 0x00c4be (0xb14c bytes) 000969 1717557013.918413 trunc from 0x03f923 to 0x00899e (0x36f84 bytes) 000972 1717557013.922478 mapread 0x0027b0 thru 0x00899e (0x61ef bytes) 000973 1717557013.922788 read 0x005f33 thru 0x00899e (0x2a6c bytes) 000977 1717557013.923264 trunc from 0x00899f to 0x029ff0 (0x21652 bytes) 000978 1717557013.926027 trunc from 0x029ff1 to 0x01f8d7 (0xa719 bytes) 000979 1717557013.928578 trunc from 0x01f8d8 to 0x022eb3 (0x35dc bytes) 000980 1717557013.932368 MIRROR_EXTEND to 2 mirrors 000985 1717557014.019485 write 0x01fc6f thru 0x02adab (0xb13d bytes) 000986 1717557014.034482 read 0x01fdfd thru 0x022526 (0x272a bytes) 000987 1717557014.034730 mapwrite 0x02d13f thru 0x030187 (0x3049 bytes) 000990 1717557014.061339 mapread 0x002f09 thru 0x0099b9 (0x6ab1 bytes) 000991 1717557014.063756 mapread 0x02cc7a thru 0x02ed5d (0x20e4 bytes) 000992 1717557014.064044 trunc from 0x030188 to 0x03637f (0x61f8 bytes) 000993 1717557014.067112 mapread 0x00042d thru 0x003b53 (0x3727 bytes) 000994 1717557014.069395 write 0x0237a6 thru 0x02f667 (0xbec2 bytes) 000996 1717557014.069918 write 0x00ba61 thru 0x00fb67 (0x4107 bytes) 000997 1717557014.070105 mapread 0x02f038 thru 0x03637f (0x7348 bytes) 000998 1717557014.072044 read 0x02015b thru 0x02886d (0x8713 bytes) 000999 1717557014.072194 read 0x005ef0 thru 0x011c69 (0xbd7a bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 200a: stress test ===================== 23:10:15 (1717557015) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8676864 .. Extending file size to 6061888 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7388960 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7784512 .. Extending file size to 7845056 .. Extending file size to 7975264 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8128832 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9322368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8505632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8797536 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8831808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8958912 .. done Extending file size to 9312512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 30800 30801 30803 30804 30807 done 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:11:18 (1717557078) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 907936 .. Extending file size to 2013280 .. failed Extending file size to 8931200 .. Extending file size to 5956864 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6792256 .. Extending file size to 7315040 .. Extending file size to 8001792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8107392 .. done Extending file size to 8944960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9002336 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9022272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9041152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 1770 1771 1773 1775 1777 done 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:12:22 (1717557142) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0241285 s, 86.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0610375 s, 34.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 200c (3s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:12:26 (1717557146) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0216615 s, 96.8 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 23:12:27 (1717557147) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5cd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad3:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xacd:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:12:29 (1717557149) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0198389 s, 106 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xace:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xacf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:12:31 (1717557151) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0173766 s, 121 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad1:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:12:32 (1717557152) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0154066 s, 136 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad3:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:12:55 (1717557175) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146076 s, 71.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xad9:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:13:40 (1717557220) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0147027 s, 143 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xada:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xada:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:14:20 (1717557260) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. Creating new pool oleg117-server: Pool lustre.test_204f created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0193151 s, 109 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xadb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xad8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xadb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg117-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 23:15:11 (1717557311) PASS 205 (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:15:13 (1717557313) oleg117-server: Pool lustre.test_206 created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg117-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:15:25 (1717557325) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144739 s, 72.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00498879 s, 210 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x6783b203 Mirror 3: 0x6783b203 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:15:27 (1717557327) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:15:28 (1717557328) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:15:29 (1717557329) Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0131045 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2090000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1380000 Read this (4 bytes): BBBB 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:15:43 (1717557343) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00204307 s, 4.0 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:15:44 (1717557344) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00446589 s, 235 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5f1 lmm_fid: [0x200000404:0x5f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x5f1 lmm_fid: [0x200000404:0x5f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xae1:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835441 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-06-04 23:15:44.000000000 -0400 Modify: 2024-06-04 23:15:44.000000000 -0400 Change: 2024-06-04 23:15:45.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:15:46 (1717557346) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test complete, duration 676 sec ============ 23:15:46 (1717557346) === sanity-flr: start cleanup 23:15:47 (1717557347) === === sanity-flr: finish cleanup 23:15:47 (1717557347) ===