-----============= acceptance-small: sanity-flr ============----- Wed Jul 24 23:55:21 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:55:25 (1721879725) === oleg136-client.virtnet: executing check_config_client /mnt/lustre oleg136-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg136-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aa36000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aa36000.idle_timeout=debug disable quota as required oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:55:31 (1721879731) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6951] == sanity-flr test 0a: lfs mirror create with -N option == 23:55:31 (1721879731) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:55:34 (1721879734) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:55:35 (1721879735) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:55:36 (1721879736) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:55:43 (1721879743) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:55:44 (1721879744) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:55:45 (1721879745) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012aa36000.stats=clear osc.lustre-OST0001-osc-ffff88012aa36000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:55:47 (1721879747) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:55:50 (1721879750) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:55:52 (1721879752) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:55:53 (1721879753) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x22:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 23:55:56 (1721879756) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:55:58 (1721879758) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:55:59 (1721879759) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:56:01 (1721879761) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:56:03 (1721879763) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:56:05 (1721879765) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.556176 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0610841 s, 17.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:56:08 (1721879768) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0163608 s, 128 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:56:10 (1721879770) Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:56:35 (1721879795) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.17736 s, 65.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:56:45 (1721879805) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:57:06 (1721879826) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.774732 s, 67.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 24 23:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.198s user 0m0.000s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.180s user 0m0.000s sys 0m0.136s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.239s user 0m0.000s sys 0m0.168s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aa36000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:57:26 (1721879846) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:57:27 (1721879847) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:57:28 (1721879848) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:57:29 (1721879849) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 23:57:31 (1721879851) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.126525 s, 99.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:57:34 (1721879854) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00919619 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00958711 s, 109 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:57:36 (1721879856) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:57:39 (1721879859) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000672698 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000646319 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000685901 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000683582 s, 1.5 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:57:41 (1721879861) create test files with size 15753818 .. 7517320a6ebdb692fe9cb270ed76aba9 - ec7f3edb7a90f680725239e52b45c20e - fa4c1f81dc2564708d8dd62fe1b484a9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 23:57:47 (1721879867) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:57:51 (1721879871) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:57:52 (1721879872) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319438 s, 32.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00863965 s, 121 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:57:54 (1721879874) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0345235 s, 60.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.100769 s, 41.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0528106 s, 39.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:57:57 (1721879877) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:57:58 (1721879878) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:57:59 (1721879879) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00855028 s, 123 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:58:01 (1721879881) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:58:02 (1721879882) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:58:04 (1721879884) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0788514 s, 133 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:58:07 (1721879887) llite.lustre-ffff8800a9d88000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0780589 s, 134 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88013687a000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:58:10 (1721879890) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:58:17 (1721879897) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:58:24 (1721879904) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:58:25 (1721879905) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0664843 s, 47.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0442519 s, 71.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 5d78555c0cf6e5721b4bbef851606ebf - mirror 1 checksum: 5e22eee5e174fe1c95c2050c391348b1 - mirror 2 checksum: 5d78555c0cf6e5721b4bbef851606ebf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 24 23:58:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 24 23:58:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5d78555c0cf6e5721b4bbef851606ebf - mirror 1 checksum: 5e22eee5e174fe1c95c2050c391348b1 - mirror 2 checksum: 5e22eee5e174fe1c95c2050c391348b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 24 23:58:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 24 23:58:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:58:28 (1721879908) Failing mds1 on oleg136-server Stopping /mnt/lustre-mds1 (opts:) on oleg136-server 23:58:30 (1721879910) shut down Failover mds1 to oleg136-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 23:58:42 (1721879922) targets are mounted 23:58:42 (1721879922) facet_failover done oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823665 s, 127 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:58:47 (1721879927) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:58:47 (1721879927) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:58:48 (1721879928) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00213223 s, 7.7 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:58:50 (1721879930) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:58:51 (1721879931) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:58:51 (1721879931) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:58:55 (1721879935) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:59:13 (1721879953) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:59:29 (1721879969) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:59:46 (1721879986) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 23:59:47 (1721879987) mirror create pid 21518 mirror split pid 21519 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 21518 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 21519 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 00:00:49 (1721880049) Seed set to 49 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x5fc3 000004 1721880049.750172 trunc from 00000000 to 0x005fc2 (0x5fc3 bytes) 000005 1721880049.754074 trunc from 0x005fc3 to 0x005ac0 (0x502 bytes) 000006 1721880049.757525 read 0x005772 thru 0x005ac0 (0x34f bytes) 000007 1721880049.759181 read 0x00023c thru 0x001c51 (0x1a16 bytes) 000008 1721880049.761558 read 0x0017ba thru 0x005ac0 (0x4307 bytes) 000009 1721880049.761827 mapwrite 0x016da8 thru 0x01ee9d (0x80f6 bytes) 000010 1721880049.801882 mapwrite 0x033a00 thru 0x03aa0e (0x700f bytes) 000012 1721880049.826587 MIRROR_EXTEND to 2 mirrors 000014 1721880049.940966 read 0x03025a thru 0x03aa0e (0xa7b5 bytes) 000016 1721880049.946242 MIRROR_RESYNC 2 mirrors 000017 1721880049.982084 mapread 0x026d4c thru 0x030ade (0x9d93 bytes) 000020 1721880049.989285 write 0x0299a1 thru 0x034145 (0xa7a5 bytes) truncating to largest ever: 0x39d6f 000022 1721880050.008723 trunc from 0x03aa0f to 0x039d6e (0xca0 bytes) 000023 1721880050.017391 write 0x032d25 thru 0x03f549 (0xc825 bytes) 000024 1721880050.020269 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3f6a9 000025 1721880050.129991 trunc from 0x03f54a to 0x03f6a8 (0x15f bytes) 000026 1721880050.146959 MIRROR_EXTEND to 3 mirrors 000028 1721880050.260173 mapread 0x00d821 thru 0x00fb13 (0x22f3 bytes) 000029 1721880050.267350 mapread 0x01d5ed thru 0x01e765 (0x1179 bytes) 000035 1721880050.280856 read 0x0064ed thru 0x010225 (0x9d39 bytes) 000037 1721880050.283896 trunc from 0x03f6a9 to 0x036e5f (0x8849 bytes) 000040 1721880050.299605 mapwrite 0x008f76 thru 0x00f85b (0x68e6 bytes) 000042 1721880050.318064 mapwrite 0x02650c thru 0x0335d2 (0xd0c7 bytes) 000043 1721880050.338711 write 0x024346 thru 0x02fd2f (0xb9ea bytes) 000044 1721880050.339517 read 0x031dd4 thru 0x036e5f (0x508c bytes) 000046 1721880050.340259 write 0x006ad3 thru 0x00c2fc (0x582a bytes) 000048 1721880050.344136 mapread 0x030433 thru 0x036e5f (0x6a2d bytes) 000050 1721880050.346360 MIRROR_RESYNC 3 mirrors 000052 1721880050.464458 mapwrite 0x03123f thru 0x03284e (0x1610 bytes) 000054 1721880050.505077 read 0x0277cb thru 0x031516 (0x9d4c bytes) 000056 1721880050.506986 mapwrite 0x023a31 thru 0x02c43c (0x8a0c bytes) 000057 1721880050.522522 MIRROR_EXTEND to 4 mirrors 000059 1721880050.614421 mapread 0x0325fe thru 0x0350c6 (0x2ac9 bytes) 000061 1721880050.620737 write 0x0320b3 thru 0x0347f7 (0x2745 bytes) 000062 1721880050.636817 write 0x03d5d0 thru 0x03ffff (0x2a30 bytes) 000063 1721880050.638869 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000068 1721880050.684637 mapwrite 0x0307ff thru 0x03095d (0x15f bytes) 000069 1721880050.697371 mapread 0x02e82f thru 0x03670c (0x7ede bytes) 000070 1721880050.701286 mapwrite 0x012087 thru 0x020902 (0xe87c bytes) 000071 1721880050.719268 write 0x0121a0 thru 0x016c0b (0x4a6c bytes) 000073 1721880050.720215 mapread 0x01a0a3 thru 0x0270ba (0xd018 bytes) 000074 1721880050.722994 trunc from 0x040000 to 0x02cea5 (0x1315a bytes) 000078 1721880050.730342 mapread 0x024a45 thru 0x02cea5 (0x8461 bytes) 000080 1721880050.730987 mapread 0x013645 thru 0x020ea2 (0xd85e bytes) 000081 1721880050.731372 read 0x018619 thru 0x01bed4 (0x38bc bytes) 000082 1721880050.731626 mapread 0x0143aa thru 0x016a33 (0x268a bytes) 000083 1721880050.732145 trunc from 0x02cea6 to 0x02651b (0x698a bytes) 000085 1721880050.736669 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000086 1721880050.780399 trunc from 0x02651c to 0x034542 (0xe027 bytes) 000091 1721880050.784207 mapread 0x013606 thru 0x018b88 (0x5583 bytes) 000092 1721880050.788658 trunc from 0x034543 to 0x00ca06 (0x27b3c bytes) 000093 1721880050.793767 MIRROR_EXTEND to 3 mirrors 000094 1721880050.893626 mapread 0x00a87f thru 0x00ca06 (0x2188 bytes) 000096 1721880050.898840 mapwrite 0x030932 thru 0x038e93 (0x8562 bytes) 000097 1721880050.932917 write 0x02d3fa thru 0x036e9b (0x9aa2 bytes) 000100 1721880050.933847 write 0x032576 thru 0x038ff5 (0x6a80 bytes) 000101 1721880050.935507 MIRROR_EXTEND to 4 mirrors 000102 1721880051.030900 trunc from 0x038ff6 to 0x005a89 (0x3356c bytes) 000105 1721880051.052375 MIRROR_EXTEND to 5 mirrors 000108 1721880051.151508 trunc from 0x005a8a to 0x03df43 (0x384ba bytes) 000111 1721880051.167702 trunc from 0x03df44 to 0x003e9b (0x3a0a8 bytes) 000112 1721880051.172146 read 0x002520 thru 0x003e9b (0x197c bytes) 000115 1721880051.174314 trunc from 0x003e9c to 0x006541 (0x26a6 bytes) 000116 1721880051.177679 mapwrite 0x029a4d thru 0x0374fd (0xdab1 bytes) 000119 1721880051.202111 write 0x00d1ca thru 0x01a647 (0xd47e bytes) 000121 1721880051.206508 mapread 0x025182 thru 0x02bcb6 (0x6b35 bytes) 000122 1721880051.207012 mapread 0x02123a thru 0x027452 (0x6219 bytes) 000125 1721880051.211050 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000127 1721880051.271724 write 0x005a21 thru 0x011cf3 (0xc2d3 bytes) 000129 1721880051.276153 trunc from 0x0374fe to 0x035ce5 (0x1818 bytes) 000131 1721880051.282434 write 0x00b168 thru 0x011a60 (0x68f9 bytes) 000132 1721880051.283159 read 0x012f74 thru 0x01d755 (0xa7e2 bytes) 000134 1721880051.287566 mapwrite 0x01fa28 thru 0x02741e (0x79f7 bytes) 000135 1721880051.312097 trunc from 0x035ce6 to 0x0034da (0x3280b bytes) 000136 1721880051.318495 MIRROR_RESYNC 4 mirrors 000137 1721880051.479466 mapwrite 0x02ae56 thru 0x03ae50 (0xfffb bytes) 000139 1721880051.527859 mapwrite 0x0011e7 thru 0x00ec7b (0xda95 bytes) 000140 1721880051.551331 write 0x013f03 thru 0x01562c (0x172a bytes) 000141 1721880051.554923 write 0x035fb1 thru 0x03ffff (0xa04f bytes) 000143 1721880051.555925 read 0x010e3b thru 0x015f74 (0x513a bytes) 000144 1721880051.556306 trunc from 0x040000 to 0x03652a (0x9ad5 bytes) 000146 1721880051.562333 write 0x020420 thru 0x0225ab (0x218c bytes) 000147 1721880051.565801 mapread 0x00a97e thru 0x015f2b (0xb5ae bytes) 000148 1721880051.566497 mapread 0x0221d8 thru 0x02aea9 (0x8cd2 bytes) 000152 1721880051.570511 MIRROR_EXTEND to 5 mirrors 000153 1721880051.690744 write 0x0251a8 thru 0x02ab62 (0x59bb bytes) 000154 1721880051.710407 read 0x016d04 thru 0x01bf4c (0x5249 bytes) 000155 1721880051.713401 read 0x0159d4 thru 0x01ff3d (0xa56a bytes) 000156 1721880051.718785 mapwrite 0x00ebce thru 0x01ddf9 (0xf22c bytes) 000157 1721880051.739830 read 0x028ab6 thru 0x03361a (0xab65 bytes) 000159 1721880051.742002 MIRROR_EXTEND to 6 mirrors 000160 1721880051.857416 trunc from 0x03652b to 0x012560 (0x23fca bytes) 000161 1721880051.882294 trunc from 0x012561 to 0x029f06 (0x179a6 bytes) 000164 1721880051.888557 mapread 0x012b63 thru 0x0191a5 (0x6643 bytes) 000166 1721880051.895209 write 0x00a194 thru 0x00d90b (0x3778 bytes) 000168 1721880051.898223 mapwrite 0x00fc35 thru 0x01a0db (0xa4a7 bytes) 000169 1721880051.916966 trunc from 0x029f07 to 0x02e57a (0x4674 bytes) 000170 1721880051.921024 trunc from 0x02e57b to 0x0019c3 (0x2cbb7 bytes) 000172 1721880051.926455 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000174 1721880051.976809 write 0x01cec9 thru 0x021523 (0x465b bytes) 000175 1721880051.977826 mapread 0x0211f9 thru 0x021523 (0x32b bytes) 000176 1721880051.978413 write 0x024f68 thru 0x028e64 (0x3efd bytes) 000177 1721880051.979155 mapwrite 0x010dcf thru 0x01f107 (0xe339 bytes) 000178 1721880052.011263 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000179 1721880052.062052 trunc from 0x028e65 to 0x0308cd (0x7a69 bytes) 000180 1721880052.066161 read 0x01e558 thru 0x02b314 (0xcdbd bytes) 000181 1721880052.069072 mapread 0x013ec9 thru 0x016cf1 (0x2e29 bytes) 000182 1721880052.073745 read 0x013abc thru 0x01a3ad (0x68f2 bytes) 000185 1721880052.074481 mapread 0x00f440 thru 0x01e6d0 (0xf291 bytes) 000186 1721880052.077233 read 0x02479a thru 0x0250ae (0x915 bytes) 000187 1721880052.077585 mapwrite 0x025839 thru 0x03360c (0xddd4 bytes) 000189 1721880052.099300 trunc from 0x03360d to 0x0196ac (0x19f60 bytes) 000190 1721880052.103410 mapread 0x011d19 thru 0x0196ac (0x7994 bytes) 000193 1721880052.104524 write 0x0218c5 thru 0x028992 (0x70ce bytes) 000196 1721880052.108124 read 0x00e293 thru 0x0183ba (0xa128 bytes) 000197 1721880052.109672 MIRROR_RESYNC 4 mirrors 000201 1721880052.259218 mapread 0x01bbb1 thru 0x02220f (0x665f bytes) 000202 1721880052.264348 mapwrite 0x00743d thru 0x00b55c (0x4120 bytes) 000204 1721880052.313801 read 0x000c43 thru 0x007334 (0x66f2 bytes) 000207 1721880052.316075 mapread 0x02489f thru 0x028992 (0x40f4 bytes) 000208 1721880052.319366 mapread 0x00b878 thru 0x0116a7 (0x5e30 bytes) 000210 1721880052.322278 mapwrite 0x02da12 thru 0x030aae (0x309d bytes) 000212 1721880052.350690 trunc from 0x030aaf to 0x015b24 (0x1af8a bytes) 000213 1721880052.356244 read 0x00213a thru 0x00e1f0 (0xc0b7 bytes) 000216 1721880052.357224 write 0x01f90d thru 0x021866 (0x1f5a bytes) 000217 1721880052.357924 read 0x01145f thru 0x01a5c3 (0x9165 bytes) 000221 1721880052.361870 mapread 0x01ca6f thru 0x021866 (0x4df8 bytes) 000222 1721880052.362483 write 0x00ca0f thru 0x00efec (0x25de bytes) 000224 1721880052.363441 mapwrite 0x006bdb thru 0x01459f (0xd9c5 bytes) 000225 1721880052.385082 read 0x0207cd thru 0x021866 (0x109a bytes) 000226 1721880052.385453 write 0x02fa5b thru 0x031e78 (0x241e bytes) 000228 1721880052.386404 write 0x03cf64 thru 0x03d9c8 (0xa65 bytes) 000230 1721880052.389135 MIRROR_EXTEND to 5 mirrors 000232 1721880052.497848 trunc from 0x03d9c9 to 0x03d283 (0x745 bytes) 000235 1721880052.516180 read 0x017a78 thru 0x019637 (0x1bc0 bytes) 000236 1721880052.517818 mapwrite 0x03f8a2 thru 0x03ffff (0x75e bytes) 000241 1721880052.536594 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000242 1721880052.582702 trunc from 0x040000 to 0x0158b5 (0x2a74a bytes) 000243 1721880052.587659 MIRROR_RESYNC 4 mirrors 000244 1721880052.712720 write 0x0289dd thru 0x02c0e8 (0x370c bytes) 000247 1721880052.726691 write 0x022abe thru 0x02f43f (0xc982 bytes) 000248 1721880052.728513 read 0x0177c3 thru 0x021228 (0x9a66 bytes) 000250 1721880052.730772 mapread 0x004cec thru 0x01381b (0xeb30 bytes) 000251 1721880052.735231 read 0x01a684 thru 0x01fdde (0x575b bytes) 000252 1721880052.735495 mapread 0x01957a thru 0x01c519 (0x2fa0 bytes) 000253 1721880052.736005 write 0x0043ac thru 0x013546 (0xf19b bytes) 000254 1721880052.736665 mapwrite 0x025621 thru 0x029f27 (0x4907 bytes) 000255 1721880052.751726 read 0x0115ec thru 0x011641 (0x56 bytes) 000256 1721880052.753625 MIRROR_EXTEND to 5 mirrors 000257 1721880052.848280 trunc from 0x02f440 to 0x035d85 (0x6946 bytes) 000258 1721880052.867383 trunc from 0x035d86 to 0x013cfe (0x22087 bytes) 000259 1721880052.872158 trunc from 0x013cff to 0x00535e (0xe9a0 bytes) 000260 1721880052.876038 mapwrite 0x025302 thru 0x0278f0 (0x25ef bytes) 000261 1721880052.906237 mapread 0x0040ae thru 0x0088e4 (0x4837 bytes) 000262 1721880052.909771 trunc from 0x0278f1 to 0x02bd24 (0x4434 bytes) 000264 1721880052.913598 mapread 0x02169f thru 0x02bd24 (0xa686 bytes) 000266 1721880052.916217 mapread 0x013fea thru 0x021e5a (0xde71 bytes) 000267 1721880052.921934 MIRROR_EXTEND to 6 mirrors 000269 1721880053.017051 mapread 0x01af87 thru 0x0251e1 (0xa25b bytes) 000270 1721880053.024207 trunc from 0x02bd25 to 0x0082ef (0x23a35 bytes) 000271 1721880053.042145 mapwrite 0x001a8f thru 0x006945 (0x4eb7 bytes) 000272 1721880053.061153 trunc from 0x0082f0 to 0x002c9b (0x5654 bytes) 000273 1721880053.068312 mapread 0x0022a8 thru 0x002c9b (0x9f4 bytes) 000276 1721880053.069509 mapread 0x001289 thru 0x002c9b (0x1a13 bytes) 000278 1721880053.070317 mapread 0x00013c thru 0x000acd (0x992 bytes) 000281 1721880053.072897 read 0x0001b4 thru 0x002c9b (0x2ae8 bytes) 000283 1721880053.073403 mapwrite 0x032730 thru 0x036aee (0x43bf bytes) 000285 1721880053.098167 mapwrite 0x00784a thru 0x00d2a0 (0x5a57 bytes) 000289 1721880053.119178 read 0x006dc8 thru 0x00bb66 (0x4d9f bytes) 000290 1721880053.119595 mapread 0x02b801 thru 0x030fe0 (0x57e0 bytes) 000292 1721880053.120340 mapwrite 0x0182dc thru 0x019f84 (0x1ca9 bytes) 000293 1721880053.137445 trunc from 0x036aef to 0x00004f (0x36a9f bytes) 000294 1721880053.141099 mapwrite 0x02726d thru 0x028955 (0x16e9 bytes) 000296 1721880053.162114 mapread 0x020e24 thru 0x028955 (0x7b32 bytes) 000297 1721880053.169300 mapread 0x00601b thru 0x00873a (0x2720 bytes) 000298 1721880053.172707 trunc from 0x028956 to 0x004a88 (0x23ecd bytes) 000299 1721880053.176133 trunc from 0x004a89 to 0x019a6f (0x14fe7 bytes) 000301 1721880053.179912 mapread 0x0102f6 thru 0x012387 (0x2092 bytes) 000302 1721880053.183776 mapread 0x010e6c thru 0x019a6f (0x8c04 bytes) 000303 1721880053.190680 write 0x01cde4 thru 0x02cd75 (0xff92 bytes) 000308 1721880053.192451 write 0x00c087 thru 0x017d8b (0xbd05 bytes) 000309 1721880053.195668 MIRROR_RESYNC 6 mirrors 000312 1721880053.376816 trunc from 0x02cd76 to 0x025530 (0x7845 bytes) 000313 1721880053.393397 write 0x007eaa thru 0x016377 (0xe4ce bytes) 000314 1721880053.396308 read 0x004f07 thru 0x01097e (0xba78 bytes) 000315 1721880053.397895 read 0x003a32 thru 0x00fc53 (0xc222 bytes) 000316 1721880053.400899 mapwrite 0x0210a4 thru 0x0248e7 (0x3844 bytes) 000318 1721880053.417806 mapwrite 0x032905 thru 0x032caf (0x3ab bytes) 000320 1721880053.439461 trunc from 0x032cb0 to 0x03c7b5 (0x9b06 bytes) 000322 1721880053.444096 read 0x01ecbb thru 0x0257a9 (0x6aef bytes) 000324 1721880053.444839 mapread 0x019123 thru 0x01bb7b (0x2a59 bytes) 000326 1721880053.445624 mapwrite 0x039859 thru 0x03e20e (0x49b6 bytes) 000327 1721880053.465315 mapwrite 0x02d41d thru 0x037b41 (0xa725 bytes) 000328 1721880053.481795 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000329 1721880053.525798 mapwrite 0x02af03 thru 0x02eb5b (0x3c59 bytes) 000330 1721880053.543012 read 0x00d25e thru 0x014826 (0x75c9 bytes) 000331 1721880053.545359 mapwrite 0x00c9bb thru 0x018106 (0xb74c bytes) 000332 1721880053.565038 write 0x0246c0 thru 0x029ca0 (0x55e1 bytes) 000333 1721880053.567826 read 0x007904 thru 0x013483 (0xbb80 bytes) 000334 1721880053.568270 read 0x02bf74 thru 0x03ad94 (0xee21 bytes) 000335 1721880053.572368 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000337 1721880053.620111 mapwrite 0x028d7c thru 0x02f8f8 (0x6b7d bytes) 000338 1721880053.638300 trunc from 0x03e20f to 0x012333 (0x2bedb bytes) 000339 1721880053.641946 trunc from 0x012334 to 0x007188 (0xb1ab bytes) 000341 1721880053.645215 trunc from 0x007189 to 0x003d43 (0x3445 bytes) 000343 1721880053.648938 mapwrite 0x015d52 thru 0x02093b (0xabea bytes) 000344 1721880053.670119 write 0x039273 thru 0x03d59b (0x4329 bytes) 000345 1721880053.671789 MIRROR_RESYNC 4 mirrors 000348 1721880053.801291 write 0x02a240 thru 0x02c63a (0x23fb bytes) 000349 1721880053.818575 MIRROR_EXTEND to 5 mirrors 000351 1721880053.921968 mapread 0x03a4b5 thru 0x03d59b (0x30e7 bytes) 000352 1721880053.927554 trunc from 0x03d59c to 0x009e4a (0x33751 bytes) 000353 1721880053.948954 read 0x006cff thru 0x009e4a (0x314c bytes) 000355 1721880053.951224 mapread 0x000eb8 thru 0x00730d (0x6456 bytes) 000357 1721880053.955789 mapread 0x000344 thru 0x0016b1 (0x136e bytes) 000358 1721880053.956472 read 0x0040b1 thru 0x009e4a (0x5d9a bytes) 000364 1721880053.958314 write 0x021fd4 thru 0x028542 (0x656f bytes) 000367 1721880053.959591 write 0x001c01 thru 0x00a5f8 (0x89f8 bytes) 000368 1721880053.961762 trunc from 0x028543 to 0x029bad (0x166b bytes) 000369 1721880053.968255 MIRROR_EXTEND to 6 mirrors 000371 1721880054.089601 mapread 0x00d749 thru 0x010aea (0x33a2 bytes) 000372 1721880054.098067 mapread 0x0282fe thru 0x029bad (0x18b0 bytes) 000373 1721880054.104122 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000374 1721880054.177908 trunc from 0x029bae to 0x03230c (0x875f bytes) 000376 1721880054.193512 mapwrite 0x01e467 thru 0x02c3d7 (0xdf71 bytes) 000377 1721880054.216061 mapread 0x0122b7 thru 0x019270 (0x6fba bytes) 000379 1721880054.219172 read 0x016b0e thru 0x01733a (0x82d bytes) 000381 1721880054.219760 mapwrite 0x01dcfc thru 0x020d81 (0x3086 bytes) 000383 1721880054.234442 read 0x02d5ea thru 0x03230c (0x4d23 bytes) 000384 1721880054.237543 mapread 0x01ab7e thru 0x0236ca (0x8b4d bytes) 000389 1721880054.238933 mapwrite 0x025e2c thru 0x02ea1e (0x8bf3 bytes) 000391 1721880054.255900 MIRROR_EXTEND to 6 mirrors 000392 1721880054.363462 mapread 0x02e713 thru 0x03230c (0x3bfa bytes) 000393 1721880054.371067 MIRROR_EXTEND to 7 mirrors 000394 1721880054.473113 read 0x030261 thru 0x03230c (0x20ac bytes) 000395 1721880054.475653 write 0x02756e thru 0x0280ba (0xb4d bytes) 000396 1721880054.492680 trunc from 0x03230d to 0x01441d (0x1deef bytes) 000397 1721880054.500649 write 0x011beb thru 0x01e1fb (0xc611 bytes) 000398 1721880054.503519 write 0x0306e5 thru 0x032a92 (0x23ae bytes) 000399 1721880054.505376 MIRROR_EXTEND to 8 mirrors 000400 1721880054.615180 MIRROR_EXTEND to 9 mirrors 000402 1721880054.712884 read 0x02447e thru 0x02abe4 (0x6767 bytes) 000403 1721880054.715944 mapread 0x00fea6 thru 0x01c6e7 (0xc842 bytes) 000404 1721880054.723307 write 0x004b8a thru 0x013b85 (0xeffc bytes) 000405 1721880054.741139 trunc from 0x032a93 to 0x02b415 (0x767d bytes) 000406 1721880054.748934 read 0x02a7b6 thru 0x02b415 (0xc60 bytes) 000407 1721880054.751924 mapread 0x0077af thru 0x016064 (0xe8b6 bytes) 000408 1721880054.753900 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000409 1721880054.801795 write 0x03df5c thru 0x03ffff (0x20a4 bytes) 000410 1721880054.802667 trunc from 0x040000 to 0x02a08f (0x15f70 bytes) 000411 1721880054.806193 trunc from 0x02a090 to 0x033d0b (0x9c7c bytes) 000412 1721880054.809990 mapread 0x023fda thru 0x02f48b (0xb4b2 bytes) 000413 1721880054.816915 mapwrite 0x021125 thru 0x022d7f (0x1c5b bytes) 000414 1721880054.830740 trunc from 0x033d0c to 0x03ba68 (0x7d5d bytes) 000416 1721880054.835891 read 0x005c72 thru 0x010055 (0xa3e4 bytes) 000419 1721880054.838875 write 0x01a1d9 thru 0x02677d (0xc5a5 bytes) 000422 1721880054.843695 MIRROR_EXTEND to 9 mirrors 000424 1721880054.965354 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000425 1721880055.025107 mapread 0x013c37 thru 0x01ca08 (0x8dd2 bytes) 000426 1721880055.031220 mapwrite 0x022352 thru 0x02a6d0 (0x837f bytes) 000427 1721880055.076485 write 0x01793a thru 0x017a35 (0xfc bytes) 000429 1721880055.079075 trunc from 0x03ba69 to 0x02a93e (0x1112a bytes) 000431 1721880055.086246 read 0x00991f thru 0x011657 (0x7d39 bytes) 000433 1721880055.088920 mapread 0x01e9e8 thru 0x01f63b (0xc54 bytes) 000434 1721880055.090972 MIRROR_RESYNC 8 mirrors 000437 1721880055.326774 MIRROR_RESYNC 8 mirrors 000439 1721880055.381211 mapread 0x027e43 thru 0x02a93e (0x2afc bytes) 000440 1721880055.386023 read 0x0043f7 thru 0x01422c (0xfe36 bytes) 000445 1721880055.389416 read 0x016645 thru 0x01f05c (0x8a18 bytes) 000446 1721880055.391999 write 0x00d517 thru 0x019097 (0xbb81 bytes) 000447 1721880055.408077 read 0x01a4b3 thru 0x0234dc (0x902a bytes) 000448 1721880055.410776 write 0x029fe0 thru 0x03795f (0xd980 bytes) 000449 1721880055.411509 write 0x0083d4 thru 0x00c4d2 (0x40ff bytes) 000450 1721880055.412131 trunc from 0x037960 to 0x020b23 (0x16e3c bytes) 000453 1721880055.421293 mapread 0x00b0ea thru 0x0137bb (0x86d2 bytes) 000454 1721880055.422000 mapread 0x018c93 thru 0x020b23 (0x7e91 bytes) 000456 1721880055.422926 trunc from 0x020b24 to 0x016469 (0xa6ba bytes) 000457 1721880055.427144 write 0x02d138 thru 0x032c1d (0x5ae6 bytes) 000458 1721880055.428058 mapread 0x007287 thru 0x012cf5 (0xba6f bytes) 000459 1721880055.428710 mapread 0x00aaaa thru 0x01a090 (0xf5e7 bytes) 000462 1721880055.435441 trunc from 0x032c1e to 0x00c199 (0x26a84 bytes) 000463 1721880055.439414 trunc from 0x00c19a to 0x018c03 (0xca6a bytes) 000466 1721880055.444299 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000468 1721880055.494418 mapwrite 0x019063 thru 0x01923a (0x1d8 bytes) 000469 1721880055.514652 write 0x015a29 thru 0x01c8f3 (0x6ecb bytes) 000472 1721880055.517217 trunc from 0x01c8f4 to 0x03c391 (0x1fa9e bytes) 000473 1721880055.521246 trunc from 0x03c392 to 0x017e97 (0x244fa bytes) 000474 1721880055.524407 mapread 0x00ace8 thru 0x00d4dd (0x27f6 bytes) 000477 1721880055.527923 mapwrite 0x00ea48 thru 0x01c1c6 (0xd77f bytes) 000478 1721880055.547305 read 0x018632 thru 0x01bb42 (0x3511 bytes) 000480 1721880055.548109 read 0x0170ab thru 0x01c1c6 (0x511c bytes) 000483 1721880055.549022 trunc from 0x01c1c7 to 0x0197b9 (0x2a0d bytes) 000485 1721880055.553732 MIRROR_EXTEND to 8 mirrors 000487 1721880055.653853 mapwrite 0x010d11 thru 0x011fee (0x12de bytes) 000488 1721880055.701433 read 0x00b4ac thru 0x015b78 (0xa6cd bytes) 000489 1721880055.701780 read 0x0140a0 thru 0x0150f6 (0x1057 bytes) 000490 1721880055.702084 read 0x0143c6 thru 0x017893 (0x34ce bytes) 000492 1721880055.702469 read 0x0072f7 thru 0x016add (0xf7e7 bytes) 000493 1721880055.704683 mapread 0x018193 thru 0x0197b9 (0x1627 bytes) 000495 1721880055.705254 mapwrite 0x027e3d thru 0x029f97 (0x215b bytes) 000496 1721880055.729823 mapread 0x01bcfd thru 0x026fe3 (0xb2e7 bytes) 000497 1721880055.730553 trunc from 0x029f98 to 0x0125ff (0x17998 bytes) 000498 1721880055.735408 mapread 0x00efe0 thru 0x00f3eb (0x40c bytes) 000500 1721880055.738107 MIRROR_EXTEND to 9 mirrors 000502 1721880055.840522 trunc from 0x012600 to 0x0017ee (0x10e11 bytes) 000505 1721880055.861347 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000508 1721880055.907217 mapwrite 0x0399c7 thru 0x03ffff (0x6639 bytes) 000509 1721880055.929979 mapwrite 0x030d23 thru 0x0316e1 (0x9bf bytes) 000510 1721880055.943947 mapwrite 0x0382d9 thru 0x03ffff (0x7d27 bytes) 000511 1721880055.959285 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000512 1721880056.005175 mapread 0x01bb28 thru 0x028b33 (0xd00c bytes) 000513 1721880056.011505 mapread 0x03fed8 thru 0x03ffff (0x128 bytes) 000514 1721880056.013679 write 0x007e82 thru 0x00d4b6 (0x5635 bytes) 000515 1721880056.016399 mapread 0x00bd1a thru 0x0106e7 (0x49ce bytes) 000516 1721880056.017019 mapread 0x03ad3c thru 0x03ffff (0x52c4 bytes) 000517 1721880056.017553 read 0x001c6c thru 0x0075ef (0x5984 bytes) 000518 1721880056.017950 read 0x015533 thru 0x01eb2d (0x95fb bytes) 000519 1721880056.020587 trunc from 0x040000 to 0x01c733 (0x238cc bytes) 000520 1721880056.024407 trunc from 0x01c734 to 0x01b0b2 (0x1681 bytes) 000521 1721880056.029246 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000524 1721880056.080419 mapwrite 0x02923f thru 0x02f8c2 (0x6684 bytes) 000525 1721880056.105773 write 0x00b999 thru 0x00d7d4 (0x1e3c bytes) 000528 1721880056.108491 mapread 0x01e154 thru 0x01fda9 (0x1c56 bytes) 000529 1721880056.110898 write 0x00f9cd thru 0x019954 (0x9f88 bytes) 000530 1721880056.111497 write 0x03e7ca thru 0x03ffff (0x1836 bytes) 000535 1721880056.112821 trunc from 0x040000 to 0x02bd70 (0x1428f bytes) 000537 1721880056.116600 mapwrite 0x00d6cd thru 0x00fc72 (0x25a6 bytes) 000538 1721880056.132418 trunc from 0x02bd71 to 0x03d629 (0x118b9 bytes) 000540 1721880056.136547 write 0x0114f8 thru 0x011b3d (0x646 bytes) 000541 1721880056.137211 read 0x0338bb thru 0x03d629 (0x9d6f bytes) 000542 1721880056.139951 write 0x025933 thru 0x02d8fa (0x7fc8 bytes) 000544 1721880056.141846 read 0x032365 thru 0x0348f9 (0x2595 bytes) 000545 1721880056.144973 MIRROR_RESYNC 6 mirrors 000547 1721880056.313829 trunc from 0x03d62a to 0x001e84 (0x3b7a5 bytes) 000548 1721880056.330973 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000549 1721880056.373750 MIRROR_RESYNC 5 mirrors 000550 1721880056.493087 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000551 1721880056.543174 trunc from 0x001e85 to 0x020029 (0x1e1a5 bytes) 000552 1721880056.560021 read 0x00d572 thru 0x00f2eb (0x1d7a bytes) 000553 1721880056.561473 mapwrite 0x00019a thru 0x00a2c6 (0xa12d bytes) 000554 1721880056.579790 read 0x00a32f thru 0x0128e0 (0x85b2 bytes) 000557 1721880056.583079 write 0x027389 thru 0x028e2c (0x1aa4 bytes) 000558 1721880056.583672 write 0x02f0b6 thru 0x02f4dc (0x427 bytes) 000559 1721880056.584092 write 0x017b0b thru 0x020f53 (0x9449 bytes) 000560 1721880056.584763 write 0x00ac9a thru 0x0176d1 (0xca38 bytes) 000561 1721880056.585328 trunc from 0x02f4dd to 0x03cea3 (0xd9c7 bytes) 000565 1721880056.590318 write 0x0395a9 thru 0x03ffff (0x6a57 bytes) 000569 1721880056.594920 mapread 0x008c67 thru 0x017edf (0xf279 bytes) 000573 1721880056.596611 mapread 0x033f0c thru 0x0363a5 (0x249a bytes) 000580 1721880056.599109 write 0x01ab6b thru 0x026948 (0xbdde bytes) 000581 1721880056.600063 write 0x009aef thru 0x010647 (0x6b59 bytes) 000582 1721880056.600810 mapwrite 0x030e69 thru 0x03803f (0x71d7 bytes) 000584 1721880056.618459 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000585 1721880056.657807 read 0x01f296 thru 0x025a27 (0x6792 bytes) 000587 1721880056.660168 read 0x03254a thru 0x03ffff (0xdab6 bytes) 000588 1721880056.663343 read 0x0324cb thru 0x03a9fa (0x8530 bytes) 000591 1721880056.664047 write 0x01b31f thru 0x01d8cf (0x25b1 bytes) 000592 1721880056.664588 mapwrite 0x012e8d thru 0x0140c2 (0x1236 bytes) 000593 1721880056.677510 write 0x00a6c3 thru 0x01854e (0xde8c bytes) 000594 1721880056.679697 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000598 1721880056.721946 write 0x00d104 thru 0x0179f7 (0xa8f4 bytes) 000599 1721880056.724512 mapwrite 0x01266b thru 0x01d05e (0xa9f4 bytes) 000602 1721880056.741072 mapread 0x02b1a5 thru 0x034e4b (0x9ca7 bytes) 000605 1721880056.745803 mapwrite 0x0275bb thru 0x02be47 (0x488d bytes) 000607 1721880056.759154 read 0x010ebe thru 0x01dfe3 (0xd126 bytes) 000608 1721880056.759666 read 0x038e3f thru 0x03df5e (0x5120 bytes) 000611 1721880056.763227 write 0x01742c thru 0x0268bc (0xf491 bytes) 000612 1721880056.764102 mapread 0x0294a5 thru 0x032c93 (0x97ef bytes) 000614 1721880056.764742 write 0x035ef6 thru 0x03d3c6 (0x74d1 bytes) 000615 1721880056.765191 mapread 0x035ae1 thru 0x03f36d (0x988d bytes) 000619 1721880056.766285 trunc from 0x040000 to 0x015f0c (0x2a0f3 bytes) 000620 1721880056.771007 trunc from 0x015f0d to 0x036f38 (0x2102c bytes) 000621 1721880056.775004 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000624 1721880056.802145 mapwrite 0x02d689 thru 0x02dd97 (0x70f bytes) 000625 1721880056.815166 mapread 0x016047 thru 0x016b6e (0xb28 bytes) 000626 1721880056.817354 trunc from 0x036f39 to 0x00a4ad (0x2ca8b bytes) 000627 1721880056.820749 mapwrite 0x023436 thru 0x02bae5 (0x86b0 bytes) 000629 1721880056.840692 trunc from 0x02bae6 to 0x0390c4 (0xd5df bytes) 000630 1721880056.844117 write 0x002c3c thru 0x003e59 (0x121e bytes) 000631 1721880056.846687 trunc from 0x0390c5 to 0x018b28 (0x2059c bytes) 000632 1721880056.849940 read 0x012771 thru 0x016c63 (0x44f3 bytes) 000634 1721880056.852368 write 0x03be8e thru 0x03ffff (0x4172 bytes) 000635 1721880056.853125 read 0x03945c thru 0x03ffff (0x6ba4 bytes) 000636 1721880056.855921 write 0x00f785 thru 0x01adca (0xb646 bytes) 000640 1721880056.858738 MIRROR_EXTEND to 2 mirrors 000642 1721880056.959868 mapwrite 0x0378a7 thru 0x03b7ec (0x3f46 bytes) 000644 1721880057.006349 read 0x0064dd thru 0x015f84 (0xfaa8 bytes) 000646 1721880057.009716 write 0x011b5f thru 0x01de4c (0xc2ee bytes) 000647 1721880057.011650 mapwrite 0x001fe7 thru 0x00845b (0x6475 bytes) 000648 1721880057.026306 mapread 0x0116ed thru 0x01ddb2 (0xc6c6 bytes) 000649 1721880057.026843 mapwrite 0x0325e9 thru 0x03ffff (0xda17 bytes) 000653 1721880057.041023 write 0x02565e thru 0x02d4b6 (0x7e59 bytes) 000654 1721880057.044284 read 0x01feda thru 0x02c63f (0xc766 bytes) 000656 1721880057.044908 write 0x01dafc thru 0x024828 (0x6d2d bytes) 000658 1721880057.045848 mapwrite 0x010022 thru 0x01a310 (0xa2ef bytes) 000661 1721880057.062395 write 0x01a414 thru 0x01f052 (0x4c3f bytes) 000664 1721880057.065477 MIRROR_RESYNC 2 mirrors 000666 1721880057.151944 mapwrite 0x009927 thru 0x00a203 (0x8dd bytes) 000667 1721880057.194238 MIRROR_EXTEND to 3 mirrors 000670 1721880057.298006 read 0x023a79 thru 0x030fdc (0xd564 bytes) 000671 1721880057.301693 trunc from 0x040000 to 0x02fedb (0x10124 bytes) 000672 1721880057.321248 trunc from 0x02fedc to 0x02c87e (0x365d bytes) 000679 1721880057.327202 mapread 0x008664 thru 0x014fc5 (0xc962 bytes) 000680 1721880057.334684 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000681 1721880057.372629 trunc from 0x02c87f to 0x0384d8 (0xbc5a bytes) 000682 1721880057.375734 write 0x03b133 thru 0x03ffff (0x4ecd bytes) 000684 1721880057.376439 write 0x01cd2b thru 0x028771 (0xba47 bytes) 000685 1721880057.378848 write 0x0211fa thru 0x025ada (0x48e1 bytes) 000686 1721880057.380435 MIRROR_RESYNC 2 mirrors 000687 1721880057.465559 write 0x03137e thru 0x03afd5 (0x9c58 bytes) 000689 1721880057.478848 trunc from 0x040000 to 0x03b8d8 (0x4727 bytes) 000690 1721880057.486498 trunc from 0x03b8d9 to 0x01ab2b (0x20dad bytes) 000693 1721880057.491397 MIRROR_EXTEND to 3 mirrors 000696 1721880057.577717 write 0x00c98b thru 0x0110ea (0x4760 bytes) 000697 1721880057.591674 mapwrite 0x0171a8 thru 0x024252 (0xd0ab bytes) 000699 1721880057.616183 write 0x01ff59 thru 0x021142 (0x11ea bytes) 000700 1721880057.616904 mapread 0x01f713 thru 0x024252 (0x4b40 bytes) 000701 1721880057.617501 trunc from 0x024253 to 0x014515 (0xfd3d bytes) 000702 1721880057.621202 write 0x03e741 thru 0x03ecf9 (0x5b9 bytes) 000704 1721880057.621979 trunc from 0x03ecfa to 0x000bce (0x3e12b bytes) 000705 1721880057.625232 mapread 0x0008d9 thru 0x000bce (0x2f6 bytes) 000706 1721880057.626679 trunc from 0x000bcf to 0x02080e (0x1fc40 bytes) 000709 1721880057.630484 trunc from 0x02080f to 0x03e30e (0x1db00 bytes) 000711 1721880057.633965 mapwrite 0x00ea74 thru 0x0153f2 (0x697f bytes) 000716 1721880057.652546 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000717 1721880057.693040 read 0x018c7e thru 0x01c6f7 (0x3a7a bytes) 000718 1721880057.695048 trunc from 0x03e30f to 0x00c72b (0x31be3 bytes) 000719 1721880057.698106 mapread 0x007827 thru 0x00c72b (0x4f05 bytes) 000720 1721880057.702066 read 0x0029f9 thru 0x00316b (0x773 bytes) 000721 1721880057.703701 trunc from 0x00c72c to 0x027eae (0x1b783 bytes) 000722 1721880057.706869 mapread 0x00d934 thru 0x016789 (0x8e56 bytes) 000723 1721880057.710409 mapread 0x00db6c thru 0x0164e5 (0x897a bytes) 000724 1721880057.710834 mapread 0x011a2d thru 0x01ff4a (0xe51e bytes) 000725 1721880057.711240 mapwrite 0x0195e0 thru 0x01e7f4 (0x5215 bytes) 000727 1721880057.724386 mapread 0x00ea60 thru 0x015c75 (0x7216 bytes) 000728 1721880057.725038 mapread 0x008cf2 thru 0x018a14 (0xfd23 bytes) 000730 1721880057.725840 read 0x0232d2 thru 0x027eae (0x4bdd bytes) 000732 1721880057.726311 read 0x026c0b thru 0x027d97 (0x118d bytes) 000734 1721880057.726970 trunc from 0x027eaf to 0x01c99b (0xb513 bytes) 000735 1721880057.730931 read 0x002d4f thru 0x003adb (0xd8d bytes) 000736 1721880057.731205 mapread 0x016b59 thru 0x01c99b (0x5e43 bytes) 000737 1721880057.731625 read 0x0174ae thru 0x01c99b (0x54ee bytes) 000738 1721880057.731869 mapread 0x010ef3 thru 0x01c99b (0xbaa9 bytes) 000739 1721880057.732215 mapread 0x004fc3 thru 0x009e47 (0x4e85 bytes) 000743 1721880057.733241 mapwrite 0x008f0a thru 0x00a747 (0x183e bytes) 000744 1721880057.748378 MIRROR_EXTEND to 3 mirrors 000745 1721880057.835234 MIRROR_EXTEND to 4 mirrors 000746 1721880057.925646 write 0x022b9c thru 0x026810 (0x3c75 bytes) 000747 1721880057.939112 trunc from 0x026811 to 0x01d603 (0x920d bytes) 000749 1721880057.946301 read 0x004e8f thru 0x00986c (0x49de bytes) 000752 1721880057.948706 trunc from 0x01d604 to 0x0254fa (0x7ef7 bytes) 000753 1721880057.951910 read 0x02338b thru 0x0254fa (0x2170 bytes) 000754 1721880057.955113 write 0x03a1e0 thru 0x03ffff (0x5e20 bytes) 000755 1721880057.955787 trunc from 0x040000 to 0x017f89 (0x28076 bytes) 000756 1721880057.959328 read 0x002c9b thru 0x004b40 (0x1ea6 bytes) 000757 1721880057.959779 mapwrite 0x01b44f thru 0x01d3e3 (0x1f95 bytes) 000759 1721880057.977716 trunc from 0x01d3e4 to 0x02f998 (0x125b5 bytes) 000760 1721880057.981584 mapread 0x02d66c thru 0x02f998 (0x232d bytes) 000762 1721880057.984746 trunc from 0x02f999 to 0x006b0e (0x28e8a bytes) 000763 1721880057.987939 mapread 0x0056f0 thru 0x006b0e (0x141f bytes) 000764 1721880057.988372 mapwrite 0x008734 thru 0x00ff28 (0x77f5 bytes) 000765 1721880058.010556 trunc from 0x00ff29 to 0x001b52 (0xe3d6 bytes) 000766 1721880058.013963 mapread 0x0016ed thru 0x001b52 (0x466 bytes) 000768 1721880058.014498 mapread 0x0015a8 thru 0x001b52 (0x5ab bytes) 000770 1721880058.014990 mapread 0x0009c6 thru 0x001b52 (0x118d bytes) 000771 1721880058.015292 mapwrite 0x03c69b thru 0x03ffff (0x3965 bytes) 000773 1721880058.032477 mapwrite 0x00fdcd thru 0x019f82 (0xa1b6 bytes) 000775 1721880058.057267 read 0x02a05b thru 0x02a563 (0x509 bytes) 000776 1721880058.061244 trunc from 0x040000 to 0x02563a (0x1a9c5 bytes) 000777 1721880058.065218 mapread 0x01eb30 thru 0x02563a (0x6b0b bytes) 000779 1721880058.065963 mapread 0x023b77 thru 0x02563a (0x1ac4 bytes) 000780 1721880058.066312 mapread 0x005262 thru 0x00dd26 (0x8ac5 bytes) 000781 1721880058.066832 mapread 0x00c2f0 thru 0x00dde5 (0x1af6 bytes) 000783 1721880058.068944 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000787 1721880058.115274 trunc from 0x02563b to 0x00675a (0x1eee0 bytes) 000788 1721880058.119260 mapwrite 0x00f716 thru 0x01cacc (0xd3b7 bytes) 000790 1721880058.146785 write 0x017f6b thru 0x027075 (0xf10b bytes) 000794 1721880058.148622 read 0x018f13 thru 0x020343 (0x7431 bytes) 000796 1721880058.149192 read 0x01611b thru 0x0237f4 (0xd6da bytes) 000798 1721880058.149758 read 0x01ba12 thru 0x027075 (0xb664 bytes) 000799 1721880058.151830 MIRROR_RESYNC 3 mirrors 000802 1721880058.274355 read 0x01be7b thru 0x01e7dc (0x2962 bytes) 000803 1721880058.276183 mapread 0x018f98 thru 0x01e6bc (0x5725 bytes) 000813 1721880058.282959 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000814 1721880058.338211 MIRROR_EXTEND to 3 mirrors 000815 1721880058.438895 mapwrite 0x008d3e thru 0x00cc44 (0x3f07 bytes) 000816 1721880058.488501 mapread 0x00c09a thru 0x019e6a (0xddd1 bytes) 000820 1721880058.491978 write 0x01beda thru 0x02acd4 (0xedfb bytes) 000822 1721880058.492982 trunc from 0x02acd5 to 0x037251 (0xc57d bytes) 000823 1721880058.501891 mapread 0x01ee23 thru 0x0266e0 (0x78be bytes) 000824 1721880058.502364 write 0x036bbf thru 0x03931f (0x2761 bytes) 000830 1721880058.506466 mapwrite 0x00a34e thru 0x01265b (0x830e bytes) 000833 1721880058.522881 mapread 0x027c1e thru 0x0288e8 (0xccb bytes) 000835 1721880058.523631 read 0x01a36b thru 0x0272d8 (0xcf6e bytes) 000840 1721880058.524999 mapwrite 0x039dd0 thru 0x03ffff (0x6230 bytes) 000841 1721880058.545449 mapwrite 0x0307c2 thru 0x03a2c0 (0x9aff bytes) 000844 1721880058.561972 mapwrite 0x009193 thru 0x00dd52 (0x4bc0 bytes) 000845 1721880058.575356 write 0x00da9c thru 0x01a8d0 (0xce35 bytes) 000846 1721880058.576458 mapwrite 0x028e83 thru 0x02bd73 (0x2ef1 bytes) 000852 1721880058.595318 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000853 1721880058.646201 mapwrite 0x01e67b thru 0x0232f9 (0x4c7f bytes) 000856 1721880058.664637 trunc from 0x040000 to 0x020e62 (0x1f19d bytes) 000858 1721880058.670442 MIRROR_RESYNC 2 mirrors 000860 1721880058.759860 write 0x0315c0 thru 0x03c4f6 (0xaf37 bytes) 000861 1721880058.774965 read 0x01b66a thru 0x020b39 (0x54d0 bytes) 000862 1721880058.778720 MIRROR_RESYNC 2 mirrors 000865 1721880058.873653 trunc from 0x03c4f7 to 0x00fdc3 (0x2c733 bytes) 000866 1721880058.891871 write 0x029bfe thru 0x02b8c1 (0x1cc4 bytes) 000868 1721880058.893078 mapread 0x027437 thru 0x02b8c1 (0x448b bytes) 000872 1721880058.897087 mapread 0x0159fb thru 0x016d66 (0x136c bytes) 000873 1721880058.899872 write 0x03bfab thru 0x03ffff (0x4055 bytes) 000874 1721880058.900498 read 0x03987b thru 0x03ffff (0x6785 bytes) 000875 1721880058.902014 write 0x03cb4f thru 0x03f08c (0x253e bytes) 000876 1721880058.902447 mapread 0x038863 thru 0x03ffff (0x779d bytes) 000877 1721880058.904707 mapwrite 0x007553 thru 0x00aea6 (0x3954 bytes) 000880 1721880058.922750 mapwrite 0x024a68 thru 0x0288cb (0x3e64 bytes) 000881 1721880058.938986 mapread 0x022e42 thru 0x02fbfe (0xcdbd bytes) 000882 1721880058.941089 write 0x01003c thru 0x01f2c9 (0xf28e bytes) 000884 1721880058.942016 mapwrite 0x0182bd thru 0x022577 (0xa2bb bytes) 000885 1721880058.957577 write 0x0302bc thru 0x0343b7 (0x40fc bytes) 000887 1721880058.958456 mapwrite 0x0141a9 thru 0x023cf5 (0xfb4d bytes) 000890 1721880058.979208 MIRROR_EXTEND to 3 mirrors 000892 1721880059.097921 write 0x03422c thru 0x034bd2 (0x9a7 bytes) 000894 1721880059.112742 mapwrite 0x0298b3 thru 0x02cddb (0x3529 bytes) 000895 1721880059.128928 mapwrite 0x00ff76 thru 0x01aa82 (0xab0d bytes) 000898 1721880059.147978 write 0x0072a5 thru 0x00a634 (0x3390 bytes) 000899 1721880059.150749 write 0x01b870 thru 0x01fc9b (0x442c bytes) 000901 1721880059.151408 trunc from 0x040000 to 0x01fb69 (0x20496 bytes) 000902 1721880059.156857 read 0x01a3a1 thru 0x01fb69 (0x57c9 bytes) 000903 1721880059.157389 write 0x0109f3 thru 0x012c3b (0x2249 bytes) 000907 1721880059.159221 write 0x020468 thru 0x02fa6a (0xf603 bytes) 000908 1721880059.160410 read 0x02d1fd thru 0x02fa6a (0x286e bytes) 000909 1721880059.162405 MIRROR_EXTEND to 4 mirrors 000910 1721880059.279823 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000911 1721880059.325516 mapread 0x00c236 thru 0x011d4e (0x5b19 bytes) 000916 1721880059.331568 mapwrite 0x01ee61 thru 0x022def (0x3f8f bytes) 000918 1721880059.352751 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000920 1721880059.398568 MIRROR_EXTEND to 3 mirrors 000921 1721880059.499234 write 0x03e13c thru 0x03ffff (0x1ec4 bytes) 000922 1721880059.513810 write 0x02c69f thru 0x03b2dc (0xec3e bytes) 000923 1721880059.516713 mapread 0x022dfb thru 0x028188 (0x538e bytes) 000926 1721880059.520803 MIRROR_EXTEND to 4 mirrors 000928 1721880059.645657 mapwrite 0x03c474 thru 0x03ffff (0x3b8c bytes) 000929 1721880059.692406 read 0x013ae3 thru 0x01bc2c (0x814a bytes) 000931 1721880059.695354 mapread 0x019d0a thru 0x028ab5 (0xedac bytes) 000934 1721880059.701914 MIRROR_RESYNC 4 mirrors 000935 1721880059.841755 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000936 1721880059.929311 mapread 0x01b8d4 thru 0x01e90e (0x303b bytes) 000941 1721880059.955587 mapwrite 0x03182b thru 0x03dc21 (0xc3f7 bytes) 000942 1721880060.011827 read 0x01ee99 thru 0x020e74 (0x1fdc bytes) 000943 1721880060.013755 mapwrite 0x01fb48 thru 0x023886 (0x3d3f bytes) 000944 1721880060.029677 trunc from 0x040000 to 0x038cfc (0x7303 bytes) 000946 1721880060.039364 mapread 0x02ee1d thru 0x030ce7 (0x1ecb bytes) 000947 1721880060.040160 mapwrite 0x0025c4 thru 0x003d31 (0x176e bytes) 000951 1721880060.058252 read 0x012e52 thru 0x022ce3 (0xfe92 bytes) 000952 1721880060.061731 mapwrite 0x008f5f thru 0x00d419 (0x44bb bytes) 000954 1721880060.077719 read 0x0196ec thru 0x01a444 (0xd59 bytes) 000958 1721880060.078782 write 0x031a73 thru 0x03416f (0x26fd bytes) 000961 1721880060.079821 trunc from 0x038cfd to 0x00a3a8 (0x2e954 bytes) 000965 1721880060.086698 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000966 1721880060.135688 read 0x00539c thru 0x00a3a8 (0x500d bytes) 000967 1721880060.138162 mapread 0x0002b1 thru 0x00a3a8 (0xa0f8 bytes) 000968 1721880060.143504 trunc from 0x00a3a9 to 0x032185 (0x27ddd bytes) 000969 1721880060.147703 write 0x02f077 thru 0x039cd5 (0xac5f bytes) 000971 1721880060.150937 read 0x003b50 thru 0x005921 (0x1dd2 bytes) 000972 1721880060.151242 read 0x02e4e2 thru 0x03128c (0x2dab bytes) 000973 1721880060.151560 trunc from 0x039cd6 to 0x00c371 (0x2d964 bytes) 000974 1721880060.155857 mapread 0x007eff thru 0x00c371 (0x4473 bytes) 000975 1721880060.160202 MIRROR_RESYNC 2 mirrors 000976 1721880060.270209 trunc from 0x00c372 to 0x02949e (0x1d12d bytes) 000978 1721880060.291142 read 0x0156b6 thru 0x0232e3 (0xdc2e bytes) 000979 1721880060.294234 read 0x006cf4 thru 0x00edac (0x80b9 bytes) 000980 1721880060.297963 trunc from 0x02949f to 0x005e39 (0x23665 bytes) 000983 1721880060.304220 mapwrite 0x00fe19 thru 0x011443 (0x162b bytes) 000984 1721880060.328461 read 0x002cf6 thru 0x004757 (0x1a62 bytes) 000986 1721880060.330368 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000987 1721880060.359316 read 0x0071e9 thru 0x007385 (0x19d bytes) 000990 1721880060.361359 mapread 0x000660 thru 0x00f66d (0xf00e bytes) 000992 1721880060.369118 read 0x00809d thru 0x00d26e (0x51d2 bytes) 000993 1721880060.369437 mapwrite 0x03f9a7 thru 0x03ffff (0x659 bytes) 000996 1721880060.393809 MIRROR_EXTEND to 2 mirrors 000997 1721880060.516816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x80e:0x0]: Invalid argument (22) 000998 1721880060.562151 mapwrite 0x00f14e thru 0x013608 (0x44bb bytes) 001000 1721880060.585792 write 0x0140dc thru 0x0181b3 (0x40d8 bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 200a: stress test ===================== 00:01:01 (1721880061) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre2 Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5349248 .. Extending file size to 6664512 .. done Extending file size to 7324416 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9122656 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7739584 .. Extending file size to 8881120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9289632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9289184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 893 894 896 898 900 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) 192.168.201.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:02:05 (1721880125) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre2 Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2259008 .. Extending file size to 4186688 .. done Extending file size to 9028160 .. Extending file size to 5873184 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6781344 .. Extending file size to 7646336 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8314592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8839968 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8997344 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195264 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9118944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 3906 3907 3909 3911 3913 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) 192.168.201.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:03:08 (1721880188) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4992 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0295431 s, 71.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0962406 s, 21.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:03:12 (1721880192) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.033832 s, 62.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 00:03:14 (1721880194) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x85a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcfd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcfd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcfe:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcf8:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:03:16 (1721880196) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0251367 s, 83.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:03:17 (1721880197) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.025129 s, 83.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd00:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd01:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd00:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd01:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfc:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:03:19 (1721880199) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0225492 s, 93.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd02:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd02:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcfe:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:03:42 (1721880222) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166507 s, 63.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd03:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd00:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd03:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd00:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd04:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:04:23 (1721880263) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd01:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd02:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0264757 s, 79.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd01:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd02:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd05:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd01:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd02:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd05:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:05:08 (1721880308) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. Creating new pool oleg136-server: Pool lustre.test_204f created Adding targets to pool oleg136-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg136-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0234663 s, 89.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd03:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd06:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd03:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd06:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg136-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg136-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg136-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (58s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 00:06:06 (1721880366) PASS 205 (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:06:08 (1721880368) oleg136-server: Pool lustre.test_206 created Adding targets to pool oleg136-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg136-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg136-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg136-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg136-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:06:21 (1721880381) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159699 s, 65.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00894108 s, 117 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3ed1981d Mirror 3: 0x3ed1981d PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:06:23 (1721880383) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:06:24 (1721880384) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:06:25 (1721880385) Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0276416 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12a0000 Read this (4 bytes): BBBB 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:06:39 (1721880399) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00243104 s, 3.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:06:40 (1721880400) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123125 s, 85.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x87e lmm_fid: [0x200000404:0x87e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x87e lmm_fid: [0x200000404:0x87e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd0d:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836094 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:06:40.000000000 -0400 Modify: 2024-07-25 00:06:40.000000000 -0400 Change: 2024-07-25 00:06:41.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:06:42 (1721880402) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 681 sec ============ 00:06:43 (1721880403) === sanity-flr: start cleanup 00:06:43 (1721880403) === === sanity-flr: finish cleanup 00:06:44 (1721880404) ===