-----============= acceptance-small: sanity-flr ============----- Mon Dec 2 13:25:08 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:25:14 (1733163914) === oleg159-client.virtnet: executing check_config_client /mnt/lustre oleg159-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg159-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6f66000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6f66000.idle_timeout=debug disable quota as required oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:25:26 (1733163926) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6965] == sanity-flr test 0a: lfs mirror create with -N option == 13:25:27 (1733163927) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:25:35 (1733163935) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:25:37 (1733163937) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:25:38 (1733163938) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:25:49 (1733163949) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:25:50 (1733163950) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 13:25:52 (1733163952) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6f66000.stats=clear osc.lustre-OST0001-osc-ffff8800b6f66000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:25:56 (1733163956) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:26:00 (1733163960) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:26:04 (1733163964) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:26:06 (1733163966) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 13:26:09 (1733163969) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:26:12 (1733163972) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:26:14 (1733163974) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:26:17 (1733163977) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:26:21 (1733163981) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:26:24 (1733163984) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.739852 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119438 s, 8.8 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:26:29 (1733163989) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0445533 s, 118 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:26:33 (1733163993) Stopping /mnt/lustre-ost1 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg159-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg159-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:27:07 (1733164027) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.182336 s, 40.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg159-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:27:22 (1733164042) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg159-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg159-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:27:46 (1733164066) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.04588 s, 50.1 MB/s -rw-r--r-- 1 root root 52428800 Dec 2 13:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.385s user 0m0.000s sys 0m0.141s Stopping /mnt/lustre-ost1 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.289s user 0m0.001s sys 0m0.130s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg159-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg159-server oleg159-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.299s user 0m0.000s sys 0m0.125s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg159-server: oleg159-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg159-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f66000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:28:13 (1733164093) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:28:15 (1733164095) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:28:16 (1733164096) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:28:17 (1733164097) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 13:28:19 (1733164099) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.197843 s, 63.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:28:24 (1733164104) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108981 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012339 s, 85.0 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:28:26 (1733164106) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:28:29 (1733164109) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00171183 s, 598 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00132983 s, 770 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00189401 s, 541 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000941157 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 13:28:33 (1733164113) create test files with size 15736191 .. f81754893faae4119cfb6702023cd21f - 24a994e495d9cc5e59a4bde8d43c2fd7 - 3b417927e15c77295be49c0330a6b728 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg159-client.virtnet /mnt/lustre (opts:) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 13:28:40 (1733164120) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:28:46 (1733164126) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:28:48 (1733164128) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0485014 s, 21.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168436 s, 62.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 13:28:51 (1733164131) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0523163 s, 40.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.117295 s, 35.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0808413 s, 25.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 13:28:55 (1733164135) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:28:56 (1733164136) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:28:58 (1733164138) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158475 s, 66.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 13:29:00 (1733164140) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:29:01 (1733164141) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:29:05 (1733164145) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0895928 s, 117 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg159-client.virtnet /mnt/lustre (opts:) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:29:08 (1733164148) llite.lustre-ffff8800a1936000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0810344 s, 129 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg159-client.virtnet /mnt/lustre (opts:) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a2f44000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:29:12 (1733164152) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:29:23 (1733164163) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:29:33 (1733164173) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:29:34 (1733164174) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0937456 s, 33.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0568853 s, 55.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 42e4d5b0ec7f3706a05b747883cce82b - mirror 1 checksum: 9cd8314087bdace7f1d2a95f0b24b9f9 - mirror 2 checksum: 42e4d5b0ec7f3706a05b747883cce82b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 2 13:29:35 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Dec 2 13:29:35 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 42e4d5b0ec7f3706a05b747883cce82b - mirror 1 checksum: 9cd8314087bdace7f1d2a95f0b24b9f9 - mirror 2 checksum: 9cd8314087bdace7f1d2a95f0b24b9f9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 2 13:29:35 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Dec 2 13:29:35 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:29:38 (1733164178) Failing mds1 on oleg159-server Stopping /mnt/lustre-mds1 (opts:) on oleg159-server 13:29:40 (1733164180) shut down Failover mds1 to oleg159-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 13:29:54 (1733164194) targets are mounted 13:29:54 (1733164194) facet_failover done oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111655 s, 93.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:30:00 (1733164200) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:30:01 (1733164201) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:30:02 (1733164202) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00456434 s, 3.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:30:05 (1733164205) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:30:06 (1733164206) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:30:07 (1733164207) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:30:14 (1733164214) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:30:32 (1733164232) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:30:50 (1733164250) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:31:07 (1733164267) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 13:31:09 (1733164269) mirror create pid 22315 mirror split pid 22316 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 22315 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 22316 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 13:32:11 (1733164331) Seed set to 4331 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x120cb 000001 1733164331.748742 trunc from 00000000 to 0x0120ca (0x120cb bytes) 000003 1733164331.755743 mapwrite 0x01bf36 thru 0x024896 (0x8961 bytes) 000006 1733164331.823501 read 0x01ece4 thru 0x024896 (0x5bb3 bytes) truncating to largest ever: 0x272da 000007 1733164331.824074 trunc from 0x024897 to 0x0272d9 (0x2a43 bytes) 000009 1733164331.830160 read 0x01c3b7 thru 0x026e4a (0xaa94 bytes) 000010 1733164331.838275 MIRROR_EXTEND to 2 mirrors 000011 1733164331.992604 MIRROR_RESYNC 2 mirrors 000013 1733164332.044577 write 0x026993 thru 0x02d0d6 (0x6744 bytes) 000014 1733164332.070174 mapread 0x017949 thru 0x01a68d (0x2d45 bytes) 000015 1733164332.075719 mapwrite 0x0181f4 thru 0x01d8e2 (0x56ef bytes) 000016 1733164332.108526 write 0x00ec4e thru 0x01621e (0x75d1 bytes) 000019 1733164332.116928 MIRROR_RESYNC 2 mirrors 000021 1733164332.280863 read 0x025b28 thru 0x02d0d6 (0x75af bytes) 000022 1733164332.286857 write 0x03ec3e thru 0x03ffff (0x13c2 bytes) 000023 1733164332.314080 MIRROR_EXTEND to 3 mirrors 000024 1733164332.544475 trunc from 0x040000 to 0x01046c (0x2fb93 bytes) 000025 1733164332.582730 write 0x019d85 thru 0x021f47 (0x81c3 bytes) 000027 1733164332.583812 trunc from 0x021f48 to 0x00990c (0x1863b bytes) 000028 1733164332.595469 read 0x000f67 thru 0x00990c (0x89a6 bytes) truncating to largest ever: 0x340a3 000029 1733164332.599890 trunc from 0x00990d to 0x0340a2 (0x2a796 bytes) 000030 1733164332.607712 trunc from 0x0340a3 to 0x02b624 (0x8a7e bytes) truncating to largest ever: 0x3b529 000031 1733164332.615044 trunc from 0x02b625 to 0x03b528 (0xff04 bytes) 000034 1733164332.623891 mapread 0x003f7e thru 0x007008 (0x308b bytes) 000035 1733164332.624661 mapread 0x027622 thru 0x02d42b (0x5e0a bytes) 000036 1733164332.637509 MIRROR_EXTEND to 4 mirrors 000037 1733164332.818270 MIRROR_RESYNC 4 mirrors 000038 1733164333.021779 trunc from 0x03b529 to 0x01ca23 (0x1eb05 bytes) 000040 1733164333.056768 mapwrite 0x011b51 thru 0x01806c (0x651c bytes) 000041 1733164333.097191 MIRROR_RESYNC 4 mirrors 000043 1733164333.308674 mapwrite 0x019162 thru 0x0272a7 (0xe146 bytes) 000045 1733164333.376671 read 0x002995 thru 0x01085e (0xdeca bytes) 000046 1733164333.381861 mapwrite 0x002160 thru 0x00729b (0x513c bytes) 000047 1733164333.405229 mapread 0x019402 thru 0x01f2ef (0x5eee bytes) 000048 1733164333.406126 mapread 0x014f0a thru 0x01b36b (0x6462 bytes) 000049 1733164333.406738 write 0x002b72 thru 0x005f59 (0x33e8 bytes) 000053 1733164333.408295 read 0x00e7d4 thru 0x01be3f (0xd66c bytes) 000055 1733164333.415095 MIRROR_EXTEND to 5 mirrors 000057 1733164333.608792 mapread 0x01a8a4 thru 0x02475b (0x9eb8 bytes) 000059 1733164333.630431 write 0x014f66 thru 0x01a961 (0x59fc bytes) 000061 1733164333.665220 MIRROR_RESYNC 5 mirrors 000065 1733164333.913489 trunc from 0x0272a8 to 0x009ae3 (0x1d7c4 bytes) 000066 1733164333.951091 mapwrite 0x004497 thru 0x00bcaa (0x7814 bytes) 000067 1733164333.989675 read 0x00b2e4 thru 0x00bcaa (0x9c7 bytes) 000068 1733164333.990089 write 0x00e549 thru 0x01085e (0x2316 bytes) 000069 1733164333.990842 write 0x01fbeb thru 0x02f87d (0xfc93 bytes) 000070 1733164333.992762 trunc from 0x02f87e to 0x0134c1 (0x1c3bc bytes) 000071 1733164334.000799 read 0x00d741 thru 0x00dacb (0x38b bytes) 000072 1733164334.004985 trunc from 0x0134c2 to 0x01854d (0x508c bytes) 000074 1733164334.014841 write 0x03c0c8 thru 0x03ffff (0x3f38 bytes) 000077 1733164334.017434 trunc from 0x040000 to 0x00d1ae (0x32e51 bytes) 000078 1733164334.025954 mapread 0x007acf thru 0x00d1ae (0x56e0 bytes) 000079 1733164334.026689 read 0x00bc7c thru 0x00d1ae (0x1533 bytes) 000080 1733164334.027090 read 0x00a6ec thru 0x00d1ae (0x2ac3 bytes) 000081 1733164334.027486 mapread 0x00510b thru 0x00d1ae (0x80a4 bytes) 000083 1733164334.028340 read 0x0025ce thru 0x008179 (0x5bac bytes) 000085 1733164334.028918 mapread 0x004ec5 thru 0x006b0d (0x1c49 bytes) 000086 1733164334.029440 write 0x03d002 thru 0x03eef3 (0x1ef2 bytes) 000089 1733164334.033003 MIRROR_RESYNC 5 mirrors 000091 1733164334.268818 mapwrite 0x03328c thru 0x0334f4 (0x269 bytes) 000092 1733164334.327627 trunc from 0x03eef4 to 0x0048b5 (0x3a63e bytes) 000094 1733164334.339840 mapread 0x00382c thru 0x0048b5 (0x108a bytes) 000096 1733164334.342718 trunc from 0x0048b6 to 0x03ac11 (0x3635c bytes) 000097 1733164334.348474 trunc from 0x03ac12 to 0x026130 (0x14ae1 bytes) 000098 1733164334.353456 write 0x034c6c thru 0x035771 (0xb06 bytes) 000100 1733164334.356653 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000101 1733164334.417053 MIRROR_RESYNC 4 mirrors 000102 1733164334.592380 write 0x007399 thru 0x011e36 (0xaa9e bytes) 000104 1733164334.614551 trunc from 0x035772 to 0x02065a (0x15117 bytes) 000106 1733164334.628499 write 0x03b9c2 thru 0x03ffff (0x463e bytes) 000107 1733164334.629349 trunc from 0x040000 to 0x00e879 (0x31786 bytes) 000108 1733164334.634367 write 0x0176f8 thru 0x01eda0 (0x76a9 bytes) 000112 1733164334.636052 mapread 0x005663 thru 0x0069fa (0x1398 bytes) 000114 1733164334.639936 mapwrite 0x02c028 thru 0x037f9a (0xbf73 bytes) 000115 1733164334.674764 mapwrite 0x0168a7 thru 0x02642b (0xfb85 bytes) 000118 1733164334.704113 mapread 0x0007e2 thru 0x00ec6c (0xe48b bytes) 000119 1733164334.707268 trunc from 0x037f9b to 0x0058e4 (0x326b6 bytes) 000120 1733164334.715022 MIRROR_EXTEND to 5 mirrors 000121 1733164334.836010 read 0x002fc0 thru 0x0058e4 (0x2925 bytes) 000123 1733164334.840510 mapwrite 0x02c4ff thru 0x03c392 (0xfe94 bytes) 000124 1733164334.902011 write 0x02e900 thru 0x037d57 (0x9458 bytes) 000125 1733164334.902884 trunc from 0x03c393 to 0x01b7b6 (0x20bdc bytes) 000129 1733164334.911861 trunc from 0x01b7b7 to 0x023afd (0x8347 bytes) 000130 1733164334.917883 read 0x001fb0 thru 0x009bc4 (0x7c15 bytes) 000131 1733164334.925122 MIRROR_EXTEND to 6 mirrors 000132 1733164335.096033 write 0x0078f7 thru 0x008a21 (0x112b bytes) 000134 1733164335.119621 mapwrite 0x001968 thru 0x002b11 (0x11aa bytes) 000136 1733164335.157345 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000137 1733164335.236303 read 0x021f7e thru 0x023afd (0x1b80 bytes) 000138 1733164335.239523 read 0x01b80f thru 0x01dbf5 (0x23e7 bytes) 000139 1733164335.247401 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000141 1733164335.325184 read 0x00a880 thru 0x00cd2f (0x24b0 bytes) 000142 1733164335.329739 MIRROR_EXTEND to 5 mirrors 000147 1733164335.496032 read 0x00260a thru 0x00c032 (0x9a29 bytes) 000148 1733164335.505238 MIRROR_RESYNC 5 mirrors 000149 1733164335.773397 mapwrite 0x011f2d thru 0x021dd7 (0xfeab bytes) 000151 1733164335.854049 read 0x01fdf7 thru 0x023afd (0x3d07 bytes) 000152 1733164335.854530 trunc from 0x023afe to 0x01448a (0xf673 bytes) 000153 1733164335.872169 trunc from 0x01448b to 0x006612 (0xde78 bytes) 000156 1733164335.881425 read 0x002d54 thru 0x006612 (0x38bf bytes) 000158 1733164335.888385 MIRROR_RESYNC 5 mirrors 000159 1733164336.173766 trunc from 0x006613 to 0x0085d5 (0x1fc3 bytes) 000163 1733164336.205471 write 0x039f4a thru 0x03ffff (0x60b6 bytes) 000165 1733164336.206798 read 0x006331 thru 0x0122d4 (0xbfa4 bytes) 000166 1733164336.211092 write 0x026e19 thru 0x02b571 (0x4759 bytes) 000168 1733164336.215096 mapwrite 0x0054ca thru 0x00d867 (0x839e bytes) 000169 1733164336.252868 mapread 0x0058f3 thru 0x00fb63 (0xa271 bytes) 000170 1733164336.253476 trunc from 0x040000 to 0x028c5c (0x173a3 bytes) 000171 1733164336.261751 mapread 0x01065f thru 0x01683f (0x61e1 bytes) 000173 1733164336.265384 read 0x01ac7d thru 0x01dae2 (0x2e66 bytes) 000174 1733164336.266168 write 0x01209c thru 0x01740f (0x5374 bytes) 000175 1733164336.266918 mapread 0x001a07 thru 0x005640 (0x3c3a bytes) 000176 1733164336.267690 read 0x024753 thru 0x028c5c (0x450a bytes) 000177 1733164336.270947 mapread 0x010cdc thru 0x015bcb (0x4ef0 bytes) 000178 1733164336.271670 write 0x019462 thru 0x022f45 (0x9ae4 bytes) 000179 1733164336.272463 mapread 0x0213e3 thru 0x028c5c (0x787a bytes) 000181 1733164336.277566 write 0x0075d1 thru 0x010ce4 (0x9714 bytes) 000184 1733164336.279012 write 0x037925 thru 0x03d223 (0x58ff bytes) 000185 1733164336.281481 MIRROR_EXTEND to 6 mirrors 000187 1733164336.467865 mapwrite 0x001670 thru 0x01039c (0xed2d bytes) 000189 1733164336.552138 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000192 1733164336.618867 write 0x00f325 thru 0x014bb1 (0x588d bytes) 000194 1733164336.622969 mapread 0x0057d4 thru 0x00be3e (0x666b bytes) 000195 1733164336.627454 read 0x027188 thru 0x02c555 (0x53ce bytes) 000197 1733164336.631626 write 0x03914c thru 0x039dc4 (0xc79 bytes) 000198 1733164336.635832 write 0x03b179 thru 0x03ea09 (0x3891 bytes) 000203 1733164336.639990 MIRROR_RESYNC 5 mirrors 000204 1733164336.893086 mapwrite 0x013184 thru 0x01fc09 (0xca86 bytes) 000205 1733164336.966343 trunc from 0x03ea0a to 0x03ae6b (0x3b9e bytes) 000206 1733164336.979587 mapread 0x00a221 thru 0x016585 (0xc365 bytes) truncating to largest ever: 0x3e702 000208 1733164336.984034 trunc from 0x03ae6c to 0x03e701 (0x3896 bytes) 000209 1733164336.988993 trunc from 0x03e702 to 0x004c78 (0x39a89 bytes) 000210 1733164336.994172 write 0x003f37 thru 0x00f499 (0xb563 bytes) 000213 1733164336.997956 trunc from 0x00f49a to 0x002a8d (0xca0c bytes) 000214 1733164337.003168 read 0x001a0a thru 0x002a8d (0x1084 bytes) 000217 1733164337.004357 trunc from 0x002a8e to 0x00a06d (0x75e0 bytes) 000219 1733164337.010625 read 0x001e34 thru 0x00a06d (0x823a bytes) 000220 1733164337.016095 mapwrite 0x0290c6 thru 0x02a0c0 (0xffb bytes) 000223 1733164337.044856 mapread 0x0250aa thru 0x02a0c0 (0x5017 bytes) 000226 1733164337.046097 write 0x021a90 thru 0x02ca1e (0xaf8f bytes) 000227 1733164337.046980 mapwrite 0x027fef thru 0x037dbf (0xfdd1 bytes) 000228 1733164337.078124 write 0x014f87 thru 0x024938 (0xf9b2 bytes) 000229 1733164337.079264 read 0x031a85 thru 0x0362c5 (0x4841 bytes) 000230 1733164337.079706 write 0x02e99f thru 0x03d31e (0xe980 bytes) 000231 1733164337.080700 mapread 0x000166 thru 0x007700 (0x759b bytes) 000237 1733164337.082812 mapwrite 0x0375d7 thru 0x03e75c (0x7186 bytes) 000238 1733164337.117598 read 0x00e7fc thru 0x0159ca (0x71cf bytes) 000241 1733164337.119061 mapread 0x01172e thru 0x01bd65 (0xa638 bytes) 000244 1733164337.120406 trunc from 0x03e75d to 0x02a326 (0x14436 bytes) 000245 1733164337.128406 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000246 1733164337.192899 mapwrite 0x023405 thru 0x031c2a (0xe826 bytes) 000247 1733164337.236677 MIRROR_RESYNC 4 mirrors 000249 1733164337.487210 write 0x00386d thru 0x0100b0 (0xc844 bytes) 000250 1733164337.519159 mapread 0x00d6a2 thru 0x01c1c2 (0xeb21 bytes) 000251 1733164337.526377 write 0x002c5e thru 0x00cae1 (0x9e84 bytes) 000252 1733164337.531058 trunc from 0x031c2b to 0x036053 (0x4429 bytes) 000253 1733164337.548909 mapread 0x00acc5 thru 0x010ba2 (0x5ede bytes) 000256 1733164337.550413 mapwrite 0x032b37 thru 0x037b9e (0x5068 bytes) 000257 1733164337.585885 mapread 0x02ee52 thru 0x033d54 (0x4f03 bytes) 000258 1733164337.586783 trunc from 0x037b9f to 0x0378b4 (0x2ea bytes) 000259 1733164337.592135 mapread 0x0065e9 thru 0x0084ae (0x1ec6 bytes) 000260 1733164337.592793 mapwrite 0x03dd43 thru 0x03ffff (0x22bd bytes) 000261 1733164337.629294 mapwrite 0x00cc7a thru 0x017069 (0xa3f0 bytes) 000264 1733164337.654596 MIRROR_RESYNC 4 mirrors 000267 1733164337.863597 mapread 0x0372b7 thru 0x03976d (0x24b7 bytes) 000269 1733164337.874567 mapread 0x02275a thru 0x0248d4 (0x217b bytes) 000272 1733164337.878805 mapwrite 0x00b191 thru 0x00b6ac (0x51c bytes) 000273 1733164337.957158 mapread 0x004218 thru 0x011596 (0xd37f bytes) 000278 1733164337.961905 MIRROR_EXTEND to 5 mirrors 000281 1733164338.183079 mapwrite 0x01a264 thru 0x01a267 (0x4 bytes) 000282 1733164338.281890 mapwrite 0x00b081 thru 0x0183d2 (0xd352 bytes) 000285 1733164338.315834 read 0x03dadb thru 0x03ffff (0x2525 bytes) 000286 1733164338.319995 mapread 0x029aa1 thru 0x035744 (0xbca4 bytes) 000287 1733164338.326818 mapread 0x02be3d thru 0x02ff5b (0x411f bytes) 000288 1733164338.327374 trunc from 0x040000 to 0x01a0ce (0x25f31 bytes) 000292 1733164338.338396 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000293 1733164338.424286 MIRROR_EXTEND to 5 mirrors 000294 1733164338.589894 mapread 0x00812a thru 0x011fd3 (0x9eaa bytes) 000295 1733164338.599555 mapread 0x01247f thru 0x01a0ce (0x7c50 bytes) 000297 1733164338.607405 read 0x00aead thru 0x0178a8 (0xc9fc bytes) 000300 1733164338.613382 read 0x0128e6 thru 0x018f8d (0x66a8 bytes) 000301 1733164338.618394 MIRROR_EXTEND to 6 mirrors 000302 1733164338.793700 mapwrite 0x027e84 thru 0x0280b8 (0x235 bytes) 000303 1733164338.861231 write 0x0237b0 thru 0x02dc68 (0xa4b9 bytes) 000304 1733164338.866499 read 0x02d7b7 thru 0x02dc68 (0x4b2 bytes) 000305 1733164338.868916 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000309 1733164338.955017 trunc from 0x02dc69 to 0x01fdf7 (0xde71 bytes) 000310 1733164338.962058 mapwrite 0x004620 thru 0x00ae84 (0x6865 bytes) 000311 1733164338.999805 mapread 0x00bd62 thru 0x017a13 (0xbcb2 bytes) 000312 1733164339.006535 read 0x0041bb thru 0x00b234 (0x707a bytes) 000313 1733164339.009801 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000316 1733164339.090696 trunc from 0x01fdf8 to 0x021d21 (0x1f2a bytes) 000317 1733164339.098574 read 0x00fee6 thru 0x01f655 (0xf770 bytes) 000318 1733164339.103047 mapread 0x017e0a thru 0x021aea (0x9ce1 bytes) 000319 1733164339.108281 write 0x00c8c0 thru 0x01a419 (0xdb5a bytes) 000320 1733164339.113428 trunc from 0x021d22 to 0x0337ba (0x11a99 bytes) 000321 1733164339.123673 MIRROR_RESYNC 4 mirrors 000322 1733164339.393885 write 0x026da6 thru 0x0330f6 (0xc351 bytes) 000328 1733164339.420304 trunc from 0x0337bb to 0x00615b (0x2d65f bytes) 000329 1733164339.435870 mapwrite 0x01a679 thru 0x01f1a5 (0x4b2d bytes) 000330 1733164339.473729 mapwrite 0x012a1b thru 0x012ce1 (0x2c7 bytes) 000331 1733164339.501688 MIRROR_EXTEND to 5 mirrors 000333 1733164339.649243 mapread 0x01f112 thru 0x01f1a5 (0x94 bytes) 000334 1733164339.654906 MIRROR_EXTEND to 6 mirrors 000335 1733164339.830314 trunc from 0x01f1a6 to 0x03d348 (0x1e1a3 bytes) 000337 1733164339.878790 mapwrite 0x0089bc thru 0x00d16f (0x47b4 bytes) 000338 1733164339.915177 mapread 0x039558 thru 0x03a65b (0x1104 bytes) 000340 1733164339.919810 read 0x035b5c thru 0x03ce95 (0x733a bytes) 000342 1733164339.920635 read 0x007a00 thru 0x01391f (0xbf20 bytes) 000343 1733164339.926617 mapwrite 0x021244 thru 0x026417 (0x51d4 bytes) 000344 1733164339.950122 mapread 0x00319e thru 0x008a02 (0x5865 bytes) 000345 1733164339.950813 write 0x0101b8 thru 0x01284f (0x2698 bytes) 000347 1733164339.951869 trunc from 0x03d349 to 0x00f5ae (0x2dd9a bytes) 000348 1733164339.962451 read 0x001e6e thru 0x00beb7 (0xa04a bytes) 000350 1733164339.963753 trunc from 0x00f5af to 0x037667 (0x280b9 bytes) 000351 1733164339.972799 write 0x0336de thru 0x0399f8 (0x631b bytes) 000352 1733164339.979180 trunc from 0x0399f9 to 0x013907 (0x260f1 bytes) 000353 1733164339.989269 write 0x014dee thru 0x01751f (0x2732 bytes) 000357 1733164339.991109 mapread 0x009ef2 thru 0x01751f (0xd62e bytes) 000359 1733164339.992514 mapread 0x010013 thru 0x0167f4 (0x67e2 bytes) 000361 1733164339.994045 mapwrite 0x028cce thru 0x035e11 (0xd144 bytes) 000364 1733164340.063839 trunc from 0x035e12 to 0x01c764 (0x196ad bytes) 000367 1733164340.074531 MIRROR_EXTEND to 7 mirrors 000368 1733164340.265654 mapwrite 0x02b491 thru 0x030053 (0x4bc3 bytes) 000370 1733164340.330350 read 0x00218b thru 0x00dcd2 (0xbb48 bytes) 000371 1733164340.334448 write 0x02be3b thru 0x02cd9d (0xf63 bytes) 000373 1733164340.335400 read 0x0289cf thru 0x02c539 (0x3b6b bytes) 000374 1733164340.335811 mapread 0x010194 thru 0x0118c1 (0x172e bytes) 000375 1733164340.342830 MIRROR_EXTEND to 8 mirrors 000377 1733164340.498527 MIRROR_EXTEND to 9 mirrors 000378 1733164340.656708 mapread 0x00c9b2 thru 0x00e310 (0x195f bytes) 000382 1733164340.676039 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000383 1733164340.769054 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000384 1733164340.875119 trunc from 0x030054 to 0x0182f1 (0x17d62 bytes) 000385 1733164340.916212 mapwrite 0x021444 thru 0x0291df (0x7d9c bytes) 000386 1733164340.962964 read 0x011a30 thru 0x013f3d (0x250e bytes) 000388 1733164340.966143 read 0x0044d3 thru 0x00d056 (0x8b84 bytes) 000390 1733164340.971418 write 0x01026b thru 0x01843c (0x81d2 bytes) 000391 1733164340.972381 read 0x002462 thru 0x003b5f (0x16fe bytes) 000392 1733164340.972717 read 0x01de68 thru 0x0291df (0xb378 bytes) 000393 1733164340.975507 MIRROR_RESYNC 7 mirrors 000394 1733164341.386968 read 0x001265 thru 0x0029cc (0x1768 bytes) 000395 1733164341.389947 trunc from 0x0291e0 to 0x024374 (0x4e6b bytes) 000396 1733164341.424492 trunc from 0x024375 to 0x017f22 (0xc452 bytes) 000397 1733164341.435602 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000400 1733164341.517390 trunc from 0x017f23 to 0x03108e (0x1916c bytes) 000401 1733164341.524839 trunc from 0x03108f to 0x021cd7 (0xf3b7 bytes) 000402 1733164341.530949 read 0x017be4 thru 0x01e725 (0x6b42 bytes) 000403 1733164341.535223 trunc from 0x021cd8 to 0x00670f (0x1b5c8 bytes) 000404 1733164341.545574 read 0x002f32 thru 0x003235 (0x304 bytes) 000406 1733164341.550020 mapwrite 0x03ee5a thru 0x03ffff (0x11a6 bytes) 000407 1733164341.604128 MIRROR_EXTEND to 7 mirrors 000408 1733164341.830819 mapwrite 0x03485a thru 0x03ffff (0xb7a6 bytes) 000410 1733164341.948087 mapwrite 0x00ea67 thru 0x019d2c (0xb2c6 bytes) 000413 1733164342.001412 mapread 0x021dfa thru 0x02c053 (0xa25a bytes) 000414 1733164342.014088 trunc from 0x040000 to 0x038127 (0x7ed8 bytes) 000415 1733164342.036846 read 0x02ccaf thru 0x038127 (0xb479 bytes) 000417 1733164342.040789 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000421 1733164342.150157 mapwrite 0x03e1b7 thru 0x03ffff (0x1e49 bytes) 000423 1733164342.203023 mapwrite 0x00eab7 thru 0x016c55 (0x819f bytes) 000424 1733164342.229139 write 0x016f14 thru 0x01a228 (0x3315 bytes) 000425 1733164342.232278 mapread 0x03a040 thru 0x03ffff (0x5fc0 bytes) 000426 1733164342.240955 write 0x021d3d thru 0x02cf92 (0xb256 bytes) 000428 1733164342.250537 trunc from 0x040000 to 0x00dd90 (0x3226f bytes) 000432 1733164342.263685 mapwrite 0x004db1 thru 0x0099ea (0x4c3a bytes) 000435 1733164342.318740 mapwrite 0x00da9f thru 0x017f28 (0xa48a bytes) 000436 1733164342.370871 mapread 0x016c4c thru 0x017f28 (0x12dd bytes) 000438 1733164342.371857 mapread 0x00577d thru 0x00eeec (0x9770 bytes) 000439 1733164342.372469 mapread 0x012da0 thru 0x017b73 (0x4dd4 bytes) 000440 1733164342.372923 write 0x02b57e thru 0x02f98a (0x440d bytes) 000441 1733164342.373511 trunc from 0x02f98b to 0x0011c9 (0x2e7c1 bytes) 000442 1733164342.381286 mapread 0x0005ff thru 0x0011c9 (0xbcb bytes) 000446 1733164342.385298 trunc from 0x0011ca to 0x010bdc (0xfa13 bytes) 000447 1733164342.396471 read 0x009d75 thru 0x010bdc (0x6e68 bytes) 000448 1733164342.399884 mapread 0x0070e1 thru 0x00c0fe (0x501e bytes) 000452 1733164342.413967 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000453 1733164342.514886 read 0x0026f2 thru 0x006c74 (0x4583 bytes) 000455 1733164342.519369 read 0x00e5eb thru 0x00f43a (0xe50 bytes) 000456 1733164342.524746 trunc from 0x010bdd to 0x030693 (0x1fab7 bytes) 000457 1733164342.536604 write 0x0165b7 thru 0x01cb98 (0x65e2 bytes) 000458 1733164342.545239 trunc from 0x030694 to 0x016295 (0x1a3fe bytes) 000459 1733164342.557405 mapwrite 0x039c35 thru 0x03ffff (0x63cb bytes) 000460 1733164342.612314 read 0x028b06 thru 0x030188 (0x7683 bytes) 000462 1733164342.619017 read 0x022e6d thru 0x032c68 (0xfdfc bytes) 000464 1733164342.619809 trunc from 0x040000 to 0x01c60c (0x239f3 bytes) 000466 1733164342.631852 mapread 0x00872a thru 0x011ba4 (0x947b bytes) 000467 1733164342.633073 trunc from 0x01c60d to 0x03969b (0x1d08f bytes) truncating to largest ever: 0x3e8ac 000468 1733164342.644316 trunc from 0x03969c to 0x03e8ab (0x5210 bytes) 000469 1733164342.657300 write 0x01fe0e thru 0x02f403 (0xf5f6 bytes) 000470 1733164342.670238 trunc from 0x03e8ac to 0x00a615 (0x34296 bytes) 000471 1733164342.677867 mapread 0x0040fc thru 0x006f71 (0x2e76 bytes) 000473 1733164342.678955 read 0x000fc8 thru 0x00a615 (0x964e bytes) 000474 1733164342.679801 write 0x023275 thru 0x02a4df (0x726b bytes) 000475 1733164342.682831 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000477 1733164342.776732 mapread 0x009037 thru 0x0149a5 (0xb96f bytes) 000479 1733164342.790733 read 0x021e78 thru 0x02a40c (0x8595 bytes) 000481 1733164342.795201 trunc from 0x02a4e0 to 0x02cabe (0x25df bytes) 000484 1733164342.803510 write 0x0198b2 thru 0x019f5f (0x6ae bytes) 000485 1733164342.804036 mapwrite 0x0370f5 thru 0x03ffff (0x8f0b bytes) 000487 1733164342.860854 mapwrite 0x02aae1 thru 0x037e2a (0xd34a bytes) 000488 1733164342.902733 trunc from 0x040000 to 0x03265b (0xd9a4 bytes) 000489 1733164342.911746 mapwrite 0x0153cf thru 0x018ffc (0x3c2e bytes) 000490 1733164342.939981 write 0x02f225 thru 0x033cb3 (0x4a8f bytes) 000491 1733164342.940801 write 0x0097ca thru 0x00cb51 (0x3388 bytes) 000492 1733164342.941476 mapwrite 0x01fe67 thru 0x02ba0e (0xbba8 bytes) 000493 1733164342.985455 mapwrite 0x031a34 thru 0x03ede7 (0xd3b4 bytes) 000494 1733164343.042103 mapread 0x00093f thru 0x00d41d (0xcadf bytes) 000495 1733164343.048576 read 0x03e6ba thru 0x03ede7 (0x72e bytes) 000496 1733164343.049017 mapread 0x017873 thru 0x023277 (0xba05 bytes) 000498 1733164343.050087 trunc from 0x03ede8 to 0x0028bf (0x3c528 bytes) 000500 1733164343.057665 mapwrite 0x01d548 thru 0x027fbc (0xaa75 bytes) 000501 1733164343.100822 write 0x005056 thru 0x0113f6 (0xc3a1 bytes) 000505 1733164343.102302 write 0x005244 thru 0x009e6e (0x4c2b bytes) 000507 1733164343.103311 mapwrite 0x0376f0 thru 0x03ffff (0x8910 bytes) 000511 1733164343.144052 read 0x01e818 thru 0x02c1f9 (0xd9e2 bytes) 000512 1733164343.144644 mapwrite 0x016548 thru 0x01eafb (0x85b4 bytes) 000515 1733164343.172575 mapwrite 0x019fba thru 0x025c56 (0xbc9d bytes) 000517 1733164343.196921 mapwrite 0x0395d1 thru 0x03ffff (0x6a2f bytes) 000518 1733164343.222123 write 0x01f163 thru 0x029011 (0x9eaf bytes) 000520 1733164343.223399 trunc from 0x040000 to 0x03370e (0xc8f1 bytes) 000522 1733164343.231527 write 0x01c3cb thru 0x025398 (0x8fce bytes) 000524 1733164343.233412 trunc from 0x03370f to 0x035ca1 (0x2593 bytes) 000525 1733164343.240747 mapread 0x02edec thru 0x035ca1 (0x6eb6 bytes) 000527 1733164343.245243 read 0x014467 thru 0x01e511 (0xa0ab bytes) 000528 1733164343.245779 read 0x035be8 thru 0x035ca1 (0xba bytes) 000529 1733164343.246170 read 0x0144e6 thru 0x01a2ad (0x5dc8 bytes) 000530 1733164343.249534 MIRROR_RESYNC 4 mirrors 000532 1733164343.498852 mapread 0x00ff30 thru 0x0155d8 (0x56a9 bytes) 000533 1733164343.511248 mapwrite 0x03e32d thru 0x03ffff (0x1cd3 bytes) 000534 1733164343.582902 mapwrite 0x036794 thru 0x03d9d3 (0x7240 bytes) 000535 1733164343.616531 write 0x02c019 thru 0x031747 (0x572f bytes) 000540 1733164343.625672 trunc from 0x040000 to 0x01f14c (0x20eb3 bytes) 000542 1733164343.637643 MIRROR_EXTEND to 5 mirrors 000543 1733164343.793594 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000544 1733164343.864614 trunc from 0x01f14d to 0x034540 (0x153f4 bytes) 000545 1733164343.890494 mapwrite 0x001ae6 thru 0x004c08 (0x3123 bytes) 000547 1733164343.919072 trunc from 0x034541 to 0x027fbf (0xc581 bytes) 000548 1733164343.926921 mapread 0x001565 thru 0x007efa (0x6996 bytes) 000549 1733164343.927714 mapwrite 0x037d6b thru 0x03ffff (0x8295 bytes) 000550 1733164343.959597 write 0x00dccd thru 0x017959 (0x9c8d bytes) 000552 1733164343.962832 read 0x0192d1 thru 0x0249a5 (0xb6d5 bytes) 000553 1733164343.967039 trunc from 0x040000 to 0x00e263 (0x31d9c bytes) 000555 1733164343.972386 mapread 0x00605d thru 0x00e263 (0x8207 bytes) 000556 1733164343.973143 write 0x01fdb6 thru 0x028ff8 (0x9243 bytes) 000557 1733164343.974268 mapwrite 0x02a8c2 thru 0x02d2a0 (0x29df bytes) 000558 1733164344.008801 write 0x00d4cd thru 0x01b687 (0xe1bb bytes) 000560 1733164344.014187 MIRROR_RESYNC 4 mirrors 000561 1733164344.265196 read 0x01bdd7 thru 0x01fef0 (0x411a bytes) 000562 1733164344.269000 write 0x006ac5 thru 0x00bc19 (0x5155 bytes) 000564 1733164344.315258 mapread 0x02069e thru 0x02d2a0 (0xcc03 bytes) 000565 1733164344.332030 MIRROR_EXTEND to 5 mirrors 000567 1733164344.504468 read 0x02009d thru 0x025823 (0x5787 bytes) 000568 1733164344.511293 MIRROR_RESYNC 5 mirrors 000569 1733164344.731814 write 0x02e075 thru 0x038ec0 (0xae4c bytes) 000571 1733164344.759105 MIRROR_EXTEND to 6 mirrors 000572 1733164344.960090 MIRROR_RESYNC 6 mirrors 000578 1733164345.251828 write 0x00df8c thru 0x0181b9 (0xa22e bytes) 000579 1733164345.284591 write 0x01e6e2 thru 0x0260c5 (0x79e4 bytes) 000585 1733164345.289950 trunc from 0x038ec1 to 0x00ee0a (0x2a0b6 bytes) 000586 1733164345.303405 read 0x00e367 thru 0x00ee0a (0xaa4 bytes) 000587 1733164345.303853 read 0x001e38 thru 0x005fe4 (0x41ad bytes) 000588 1733164345.309022 trunc from 0x00ee0b to 0x014073 (0x5269 bytes) 000589 1733164345.314442 write 0x0310f4 thru 0x0390cc (0x7fd9 bytes) 000590 1733164345.315454 mapwrite 0x03c7b7 thru 0x03ffff (0x3849 bytes) 000591 1733164345.356139 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000593 1733164345.444496 mapwrite 0x029930 thru 0x035d74 (0xc445 bytes) 000594 1733164345.496190 mapwrite 0x0356ce thru 0x03ffff (0xa932 bytes) 000595 1733164345.530295 write 0x01577c thru 0x0171e7 (0x1a6c bytes) 000597 1733164345.535425 mapread 0x0131a2 thru 0x017a87 (0x48e6 bytes) 000598 1733164345.535825 read 0x028a84 thru 0x02c3a3 (0x3920 bytes) 000600 1733164345.536242 read 0x03b166 thru 0x03ffff (0x4e9a bytes) 000602 1733164345.536750 write 0x00725a thru 0x016199 (0xef40 bytes) 000603 1733164345.541033 read 0x006352 thru 0x00b623 (0x52d2 bytes) 000604 1733164345.543672 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000605 1733164345.617126 trunc from 0x040000 to 0x03ba5d (0x45a2 bytes) 000606 1733164345.622975 read 0x0340e1 thru 0x03ba5d (0x797d bytes) 000609 1733164345.627632 trunc from 0x03ba5e to 0x0154df (0x2657e bytes) 000611 1733164345.636926 MIRROR_RESYNC 4 mirrors 000612 1733164345.837015 read 0x01354f thru 0x0154df (0x1f91 bytes) 000613 1733164345.840099 read 0x013187 thru 0x0154df (0x2359 bytes) 000614 1733164345.840414 write 0x019f1e thru 0x028546 (0xe629 bytes) 000616 1733164345.867296 MIRROR_RESYNC 4 mirrors 000617 1733164346.111920 read 0x00c7f6 thru 0x00ed74 (0x257f bytes) 000618 1733164346.113987 mapread 0x0120fa thru 0x01bed0 (0x9dd7 bytes) 000619 1733164346.121039 MIRROR_EXTEND to 5 mirrors 000621 1733164346.291770 MIRROR_RESYNC 5 mirrors 000623 1733164346.363688 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000625 1733164346.450201 write 0x024810 thru 0x03205f (0xd850 bytes) 000626 1733164346.475487 read 0x02dbe7 thru 0x03205f (0x4479 bytes) 000627 1733164346.475980 mapwrite 0x00602e thru 0x00827d (0x2250 bytes) 000628 1733164346.502394 read 0x020c0f thru 0x02c6d4 (0xbac6 bytes) 000630 1733164346.507009 mapread 0x028989 thru 0x03205f (0x96d7 bytes) 000631 1733164346.507668 trunc from 0x032060 to 0x0385ce (0x656f bytes) 000632 1733164346.516223 MIRROR_RESYNC 4 mirrors 000633 1733164346.722186 mapread 0x001ef0 thru 0x004ed7 (0x2fe8 bytes) 000634 1733164346.728663 trunc from 0x0385cf to 0x00e2a3 (0x2a32b bytes) 000635 1733164346.757235 mapread 0x0072a8 thru 0x009802 (0x255b bytes) 000640 1733164346.762393 write 0x00b44b thru 0x016faf (0xbb65 bytes) 000641 1733164346.763799 write 0x035dd7 thru 0x03619f (0x3c9 bytes) 000642 1733164346.764483 mapwrite 0x00ade2 thru 0x014094 (0x92b3 bytes) 000644 1733164346.791502 write 0x00cb48 thru 0x01731e (0xa7d7 bytes) 000645 1733164346.794254 read 0x03469b thru 0x03619f (0x1b05 bytes) 000646 1733164346.797230 mapwrite 0x006162 thru 0x0065cd (0x46c bytes) 000650 1733164346.821207 mapread 0x021e1d thru 0x02a67a (0x885e bytes) 000654 1733164346.830390 trunc from 0x0361a0 to 0x032689 (0x3b16 bytes) 000655 1733164346.838943 write 0x024271 thru 0x02ef7b (0xad0b bytes) 000658 1733164346.840637 read 0x02b603 thru 0x02d077 (0x1a75 bytes) 000659 1733164346.841070 mapwrite 0x0122aa thru 0x01b5bd (0x9314 bytes) 000660 1733164346.867010 trunc from 0x03268a to 0x0292dc (0x93ad bytes) 000661 1733164346.873286 mapwrite 0x00c60e thru 0x019cbb (0xd6ae bytes) 000662 1733164346.903660 MIRROR_RESYNC 4 mirrors 000664 1733164347.143384 mapwrite 0x014d45 thru 0x0173b7 (0x2673 bytes) 000667 1733164347.217075 MIRROR_EXTEND to 5 mirrors 000668 1733164347.389389 mapread 0x01c781 thru 0x029227 (0xcaa7 bytes) 000669 1733164347.408439 MIRROR_RESYNC 5 mirrors 000670 1733164347.645869 mapread 0x012d11 thru 0x01a940 (0x7c30 bytes) 000671 1733164347.655066 mapread 0x00c848 thru 0x0127c1 (0x5f7a bytes) 000673 1733164347.661589 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000674 1733164347.753155 read 0x010f2c thru 0x017727 (0x67fc bytes) 000675 1733164347.756230 mapwrite 0x02c796 thru 0x034838 (0x80a3 bytes) 000676 1733164347.813308 MIRROR_EXTEND to 5 mirrors 000677 1733164347.962993 mapwrite 0x03158d thru 0x0357c3 (0x4237 bytes) 000678 1733164348.033082 read 0x00497a thru 0x012979 (0xe000 bytes) 000680 1733164348.040001 mapread 0x0025dc thru 0x00c49e (0x9ec3 bytes) 000683 1733164348.045643 mapread 0x031fed thru 0x0357c3 (0x37d7 bytes) 000685 1733164348.047156 mapread 0x021529 thru 0x02f8c2 (0xe39a bytes) 000686 1733164348.055645 trunc from 0x0357c4 to 0x0013e2 (0x343e1 bytes) 000687 1733164348.064189 trunc from 0x0013e3 to 0x003dc5 (0x29e3 bytes) 000689 1733164348.069384 read 0x0028d2 thru 0x003dc5 (0x14f4 bytes) 000690 1733164348.073127 trunc from 0x003dc6 to 0x015d4d (0x11f88 bytes) 000691 1733164348.078155 mapwrite 0x02cade thru 0x037e44 (0xb367 bytes) 000692 1733164348.117020 MIRROR_EXTEND to 6 mirrors 000695 1733164348.279573 read 0x013a7b thru 0x0229a9 (0xef2f bytes) 000696 1733164348.285055 trunc from 0x037e45 to 0x019709 (0x1e73b bytes) 000697 1733164348.318739 trunc from 0x01970a to 0x008a23 (0x10ce6 bytes) 000700 1733164348.327415 mapwrite 0x00805d thru 0x012f09 (0xaead bytes) 000701 1733164348.370552 mapwrite 0x020f1c thru 0x03090a (0xf9ef bytes) 000703 1733164348.416584 trunc from 0x03090b to 0x01ada0 (0x15b6a bytes) 000705 1733164348.428923 MIRROR_RESYNC 6 mirrors 000706 1733164348.714765 MIRROR_EXTEND to 7 mirrors 000707 1733164348.882966 mapread 0x011e81 thru 0x01ada0 (0x8f20 bytes) 000708 1733164348.894512 trunc from 0x01ada1 to 0x039be7 (0x1ee47 bytes) 000709 1733164348.928314 MIRROR_RESYNC 7 mirrors 000710 1733164349.242882 MIRROR_RESYNC 7 mirrors 000711 1733164349.326783 read 0x004413 thru 0x013690 (0xf27e bytes) 000713 1733164349.329993 mapread 0x0130ee thru 0x0196e8 (0x65fb bytes) 000714 1733164349.335418 mapread 0x036657 thru 0x039be7 (0x3591 bytes) 000716 1733164349.337970 trunc from 0x039be8 to 0x01ca87 (0x1d160 bytes) 000717 1733164349.362227 trunc from 0x01ca88 to 0x01be99 (0xbee bytes) 000719 1733164349.369536 mapwrite 0x033fc6 thru 0x03ffff (0xc03a bytes) 000720 1733164349.408389 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000721 1733164349.468088 write 0x03ca49 thru 0x03e561 (0x1b19 bytes) 000722 1733164349.471951 write 0x039008 thru 0x03ffff (0x6ff8 bytes) 000724 1733164349.475284 mapread 0x034ec5 thru 0x03b0ae (0x61ea bytes) 000726 1733164349.476223 mapwrite 0x0181a1 thru 0x0186c5 (0x525 bytes) 000727 1733164349.498987 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000729 1733164349.567030 mapwrite 0x038eca thru 0x03ffff (0x7136 bytes) 000730 1733164349.587627 read 0x025e0f thru 0x027470 (0x1662 bytes) 000731 1733164349.590538 write 0x01a673 thru 0x029ef1 (0xf87f bytes) 000732 1733164349.596172 mapread 0x00d652 thru 0x0100f5 (0x2aa4 bytes) 000733 1733164349.599149 write 0x02f0a6 thru 0x031ac9 (0x2a24 bytes) 000735 1733164349.602082 write 0x024825 thru 0x028f9d (0x4779 bytes) 000736 1733164349.602971 mapread 0x034902 thru 0x03ffff (0xb6fe bytes) 000738 1733164349.603706 read 0x016fd8 thru 0x019011 (0x203a bytes) 000739 1733164349.605633 MIRROR_EXTEND to 6 mirrors 000741 1733164349.763489 mapwrite 0x0308ae thru 0x03404d (0x37a0 bytes) 000742 1733164349.818937 trunc from 0x040000 to 0x00581e (0x3a7e1 bytes) 000744 1733164349.828940 mapread 0x0051b9 thru 0x00581e (0x666 bytes) 000745 1733164349.832428 mapread 0x0022ce thru 0x00581e (0x3551 bytes) 000746 1733164349.833057 mapwrite 0x03d4c7 thru 0x03ffff (0x2b39 bytes) 000747 1733164349.859477 trunc from 0x040000 to 0x032d1d (0xd2e2 bytes) 000748 1733164349.865309 write 0x01d9dd thru 0x01f39c (0x19c0 bytes) 000752 1733164349.870499 mapread 0x017bda thru 0x02274b (0xab72 bytes) 000753 1733164349.871185 write 0x001021 thru 0x0023b6 (0x1396 bytes) 000754 1733164349.871878 trunc from 0x032d1e to 0x00817c (0x2aba1 bytes) 000755 1733164349.878730 trunc from 0x00817d to 0x0368d8 (0x2e75c bytes) 000756 1733164349.884190 write 0x00a0d5 thru 0x00dc92 (0x3bbe bytes) 000757 1733164349.888432 trunc from 0x0368d9 to 0x020c10 (0x15cc8 bytes) 000758 1733164349.895276 read 0x012cea thru 0x020c10 (0xdf27 bytes) 000759 1733164349.899885 mapwrite 0x03afb5 thru 0x03f520 (0x456c bytes) 000760 1733164349.926763 read 0x011e08 thru 0x011e45 (0x3e bytes) 000761 1733164349.929435 MIRROR_RESYNC 6 mirrors 000763 1733164350.219301 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000764 1733164350.308628 mapwrite 0x03d80b thru 0x03ffff (0x27f5 bytes) 000765 1733164350.366505 trunc from 0x040000 to 0x01a2a7 (0x25d58 bytes) 000767 1733164350.373515 trunc from 0x01a2a8 to 0x01093d (0x996a bytes) 000769 1733164350.378784 write 0x033825 thru 0x035664 (0x1e40 bytes) 000770 1733164350.379661 read 0x02c332 thru 0x02faed (0x37bc bytes) 000771 1733164350.382114 mapread 0x01cc49 thru 0x02c426 (0xf7de bytes) 000772 1733164350.387737 trunc from 0x035665 to 0x010827 (0x24e3d bytes) 000773 1733164350.393670 write 0x005347 thru 0x00bc9a (0x6954 bytes) 000774 1733164350.397604 mapread 0x00ed5f thru 0x010827 (0x1ac9 bytes) 000776 1733164350.398668 read 0x00fd47 thru 0x010827 (0xae1 bytes) 000778 1733164350.399605 mapread 0x002351 thru 0x00fa63 (0xd713 bytes) 000780 1733164350.402561 MIRROR_EXTEND to 6 mirrors 000784 1733164350.532664 read 0x00609b thru 0x010827 (0xa78d bytes) 000787 1733164350.541506 mapread 0x005259 thru 0x0071a5 (0x1f4d bytes) 000789 1733164350.547980 mapwrite 0x028d4f thru 0x0313d1 (0x8683 bytes) 000791 1733164350.603654 mapwrite 0x01d393 thru 0x025360 (0x7fce bytes) 000792 1733164350.626170 read 0x00b77e thru 0x01675e (0xafe1 bytes) 000794 1733164350.630625 write 0x015fc3 thru 0x018b9e (0x2bdc bytes) 000796 1733164350.631671 read 0x00b668 thru 0x01a3a2 (0xed3b bytes) 000797 1733164350.632115 mapwrite 0x03aa06 thru 0x03ffff (0x55fa bytes) 000798 1733164350.668165 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000800 1733164350.730969 MIRROR_RESYNC 5 mirrors 000801 1733164350.923141 trunc from 0x040000 to 0x039aaf (0x6550 bytes) 000803 1733164350.954768 read 0x02d3ad thru 0x03569f (0x82f3 bytes) 000804 1733164350.958216 mapwrite 0x01025e thru 0x018046 (0x7de9 bytes) 000806 1733164350.984821 read 0x02b34c thru 0x03058d (0x5242 bytes) 000807 1733164350.989199 mapwrite 0x02ec75 thru 0x02f9ed (0xd79 bytes) 000811 1733164351.001951 read 0x006f7b thru 0x00760d (0x693 bytes) 000812 1733164351.002423 mapwrite 0x034d54 thru 0x03d266 (0x8513 bytes) 000814 1733164351.030529 MIRROR_RESYNC 5 mirrors 000816 1733164351.243750 mapwrite 0x0284da thru 0x036d9c (0xe8c3 bytes) 000817 1733164351.311948 mapwrite 0x003e75 thru 0x0049e3 (0xb6f bytes) 000818 1733164351.329836 write 0x017e6c thru 0x01ae8d (0x3022 bytes) 000820 1733164351.334264 read 0x00b5d9 thru 0x00bb17 (0x53f bytes) 000823 1733164351.335467 mapread 0x02dfb1 thru 0x0386b1 (0xa701 bytes) 000824 1733164351.338223 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000825 1733164351.406522 MIRROR_EXTEND to 5 mirrors 000828 1733164351.561896 write 0x03c63b thru 0x03ffff (0x39c5 bytes) 000829 1733164351.587066 write 0x020e34 thru 0x022133 (0x1300 bytes) 000830 1733164351.593399 trunc from 0x040000 to 0x016a42 (0x295bd bytes) 000831 1733164351.609807 mapread 0x01262b thru 0x014e44 (0x281a bytes) 000832 1733164351.614266 mapwrite 0x02ff6d thru 0x035761 (0x57f5 bytes) 000834 1733164351.649034 write 0x0392a9 thru 0x03fc29 (0x6981 bytes) 000835 1733164351.649837 read 0x03e06b thru 0x03fc29 (0x1bbf bytes) 000838 1733164351.652891 MIRROR_EXTEND to 6 mirrors 000839 1733164351.792035 read 0x01977e thru 0x022b1f (0x93a2 bytes) 000841 1733164351.797199 trunc from 0x03fc2a to 0x022831 (0x1d3f8 bytes) 000842 1733164351.826824 trunc from 0x022832 to 0x01f80c (0x3025 bytes) 000843 1733164351.837427 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000845 1733164351.905596 trunc from 0x01f80d to 0x01e903 (0xf09 bytes) 000846 1733164351.912832 MIRROR_RESYNC 5 mirrors 000848 1733164352.165360 write 0x003e69 thru 0x00e8be (0xaa56 bytes) 000851 1733164352.189193 write 0x03ffec thru 0x03ffff (0x14 bytes) 000852 1733164352.190104 read 0x0395b3 thru 0x03f024 (0x5a72 bytes) 000853 1733164352.193493 write 0x03acab thru 0x03ffff (0x5355 bytes) 000854 1733164352.194137 mapwrite 0x0276af thru 0x033ec3 (0xc815 bytes) 000855 1733164352.230871 mapread 0x03221f thru 0x03f270 (0xd052 bytes) 000857 1733164352.231779 mapread 0x02383e thru 0x026a93 (0x3256 bytes) 000860 1733164352.234966 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000863 1733164352.300468 mapread 0x001b09 thru 0x005b38 (0x4030 bytes) 000864 1733164352.311431 read 0x03dcd8 thru 0x03ffff (0x2328 bytes) 000867 1733164352.314503 trunc from 0x040000 to 0x014170 (0x2be8f bytes) 000869 1733164352.319937 write 0x01c9be thru 0x022f92 (0x65d5 bytes) 000871 1733164352.321135 mapread 0x00eb85 thru 0x011622 (0x2a9e bytes) 000872 1733164352.324446 trunc from 0x022f93 to 0x001460 (0x21b32 bytes) 000874 1733164352.330739 read 0x0011d8 thru 0x001460 (0x289 bytes) 000879 1733164352.332408 trunc from 0x001461 to 0x016eea (0x15a8a bytes) 000882 1733164352.339359 trunc from 0x016eeb to 0x02d887 (0x1699d bytes) 000886 1733164352.345979 trunc from 0x02d888 to 0x00c3f5 (0x21492 bytes) 000887 1733164352.353534 read 0x008ae2 thru 0x00c3f5 (0x3914 bytes) 000888 1733164352.358690 write 0x032fa4 thru 0x03bec5 (0x8f22 bytes) 000889 1733164352.359617 mapwrite 0x0243b1 thru 0x0272d9 (0x2f29 bytes) 000890 1733164352.389243 mapread 0x00f8f9 thru 0x018be4 (0x92ec bytes) 000891 1733164352.393006 MIRROR_RESYNC 4 mirrors 000892 1733164352.596851 trunc from 0x03bec6 to 0x03774b (0x477a bytes) 000893 1733164352.620986 read 0x0247a6 thru 0x02db76 (0x93d1 bytes) 000894 1733164352.626454 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000896 1733164352.686125 read 0x010356 thru 0x019cbd (0x9968 bytes) 000900 1733164352.690669 read 0x012fcb thru 0x018739 (0x576f bytes) 000902 1733164352.691434 mapwrite 0x02f65c thru 0x03507b (0x5a20 bytes) 000903 1733164352.720532 write 0x012fa6 thru 0x014ba6 (0x1c01 bytes) 000904 1733164352.721186 write 0x021a47 thru 0x02ca7e (0xb038 bytes) 000905 1733164352.722122 read 0x017fc0 thru 0x027990 (0xf9d1 bytes) 000906 1733164352.722601 mapread 0x016c62 thru 0x01de90 (0x722f bytes) 000909 1733164352.723751 mapread 0x00c24e thru 0x0131fe (0x6fb1 bytes) 000910 1733164352.724244 read 0x035a76 thru 0x03774b (0x1cd6 bytes) 000911 1733164352.724625 trunc from 0x03774c to 0x02c532 (0xb219 bytes) 000912 1733164352.730941 trunc from 0x02c533 to 0x03552a (0x8ff8 bytes) 000913 1733164352.738519 MIRROR_EXTEND to 4 mirrors 000914 1733164352.887191 read 0x027a91 thru 0x02a026 (0x2596 bytes) 000915 1733164352.889766 write 0x028509 thru 0x037aca (0xf5c2 bytes) 000917 1733164352.912371 trunc from 0x037acb to 0x00679c (0x3132e bytes) 000918 1733164352.925314 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000925 1733164352.991781 mapwrite 0x03277c thru 0x03b1e3 (0x8a68 bytes) 000926 1733164353.028902 mapread 0x022778 thru 0x0246e6 (0x1f6f bytes) 000930 1733164353.035864 mapwrite 0x025cf3 thru 0x030a58 (0xad66 bytes) 000931 1733164353.066746 trunc from 0x03b1e4 to 0x016675 (0x24b6e bytes) 000933 1733164353.071812 mapread 0x00da4b thru 0x014b60 (0x7116 bytes) 000934 1733164353.075616 mapwrite 0x013013 thru 0x019a15 (0x6a03 bytes) 000935 1733164353.103151 mapwrite 0x02a674 thru 0x02ced5 (0x2862 bytes) 000936 1733164353.128414 mapwrite 0x03c412 thru 0x03cc78 (0x867 bytes) 000937 1733164353.155388 MIRROR_RESYNC 3 mirrors 000939 1733164353.317997 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000941 1733164353.401136 write 0x02b35a thru 0x03581a (0xa4c1 bytes) 000943 1733164353.425041 read 0x0178d4 thru 0x02106c (0x9799 bytes) 000944 1733164353.428706 write 0x005e63 thru 0x00dc1a (0x7db8 bytes) 000946 1733164353.432697 mapwrite 0x00ff0d thru 0x012f23 (0x3017 bytes) 000947 1733164353.459309 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e2:0x0]: Invalid argument (22) 000948 1733164353.498539 trunc from 0x03cc79 to 0x035813 (0x7465 bytes) 000950 1733164353.509637 mapread 0x00a2dd thru 0x00a684 (0x3a8 bytes) 000951 1733164353.512633 mapwrite 0x00af28 thru 0x014666 (0x973f bytes) 000954 1733164353.542090 write 0x032f1f thru 0x033c2d (0xd0f bytes) 000955 1733164353.546006 mapread 0x017a35 thru 0x01d059 (0x5625 bytes) 000956 1733164353.549788 read 0x009614 thru 0x00b225 (0x1c12 bytes) 000960 1733164353.551112 mapread 0x0103fa thru 0x01a3db (0x9fe2 bytes) 000961 1733164353.553623 read 0x01deaf thru 0x021ac2 (0x3c14 bytes) 000965 1733164353.555312 mapread 0x02c0f5 thru 0x035411 (0x931d bytes) 000967 1733164353.556318 write 0x036807 thru 0x03da38 (0x7232 bytes) 000968 1733164353.557218 mapwrite 0x037d08 thru 0x038d00 (0xff9 bytes) 000969 1733164353.586711 MIRROR_EXTEND to 2 mirrors 000971 1733164353.717198 trunc from 0x03da39 to 0x017613 (0x26425 bytes) 000972 1733164353.746701 MIRROR_EXTEND to 3 mirrors 000974 1733164353.876677 write 0x03d5db thru 0x03ffff (0x2a25 bytes) 000975 1733164353.903591 trunc from 0x040000 to 0x001340 (0x3ecbf bytes) 000976 1733164353.916204 read 0x001140 thru 0x001340 (0x201 bytes) 000978 1733164353.918751 read 0x001156 thru 0x001340 (0x1eb bytes) 000979 1733164353.919156 mapwrite 0x01c765 thru 0x02b574 (0xee10 bytes) 000980 1733164353.953528 MIRROR_EXTEND to 4 mirrors 000981 1733164354.099722 mapread 0x0135b2 thru 0x018631 (0x5080 bytes) 000984 1733164354.109180 MIRROR_EXTEND to 5 mirrors 000985 1733164354.274970 mapread 0x007648 thru 0x012d2e (0xb6e7 bytes) 000986 1733164354.292699 trunc from 0x02b575 to 0x037852 (0xc2de bytes) 000988 1733164354.315581 trunc from 0x037853 to 0x0012d0 (0x36582 bytes) 000990 1733164354.321688 write 0x02c50f thru 0x02d54d (0x103f bytes) 000991 1733164354.322427 read 0x0211cc thru 0x026c69 (0x5a9e bytes) 000993 1733164354.325455 mapwrite 0x02df04 thru 0x033570 (0x566d bytes) 000995 1733164354.350101 MIRROR_RESYNC 5 mirrors 000996 1733164354.551627 write 0x020b01 thru 0x0225a8 (0x1aa8 bytes) 000999 1733164354.570485 mapread 0x013639 thru 0x01e51d (0xaee5 bytes) 001000 1733164354.576095 trunc from 0x033571 to 0x008f89 (0x2a5e7 bytes) All operations completed A-OK! PASS 70a (24s) == sanity-flr test 200a: stress test ===================== 13:32:35 (1733164355) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6006816 .. Extending file size to 5370176 .. done Extending file size to 8546400 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8974624 .. Extending file size to 8971232 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8981920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9241984 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9072000 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 30641 30642 30644 30645 30648 done 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) 192.168.201.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:33:41 (1733164421) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4099968 .. Extending file size to 1785376 .. failed Extending file size to 3566528 .. Extending file size to 7465184 .. Extending file size to 4676256 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8279264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9152544 .. Extending file size to 7580704 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9048256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9283616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9235552 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 548 549 551 552 555 done 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) 192.168.201.159@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:34:47 (1733164487) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0321911 s, 65.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.111497 s, 18.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:34:52 (1733164492) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.026674 s, 78.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 13:34:54 (1733164494) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x730:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x8b3:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:34:56 (1733164496) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274631 s, 76.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:34:57 (1733164497) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0295379 s, 71.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b7:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:34:59 (1733164499) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0317783 s, 66.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8b9:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:35:23 (1733164523) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg159-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172099 s, 60.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d7:0x0] } PASS 204d (40s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:36:04 (1733164564) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg159-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0302264 s, 69.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:36:44 (1733164604) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg159-server mds-ost sync done. Creating new pool oleg159-server: Pool lustre.test_204f created Adding targets to pool oleg159-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg159-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0232519 s, 90.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg159-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg159-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg159-server: Pool lustre.test_204f destroyed PASS 204f (55s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:37:39 (1733164659) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:37:41 (1733164661) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:37:43 (1733164663) oleg159-server: Pool lustre.test_206 created Adding targets to pool oleg159-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg159-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg159-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg159-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg159-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:37:56 (1733164676) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180917 s, 58.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107224 s, 97.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x573c6290 Mirror 3: 0x573c6290 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:37:59 (1733164679) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:38:00 (1733164680) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:38:01 (1733164681) Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0257803 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1550000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12f0000 Read this (4 bytes): BBBB 192.168.201.159@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg159-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:38:15 (1733164695) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00231422 s, 3.5 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:38:16 (1733164696) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00979131 s, 107 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x755 lmm_fid: [0x200000404:0x755:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x755 lmm_fid: [0x200000404:0x755:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8c7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835797 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-02 13:38:16.000000000 -0500 Modify: 2024-12-02 13:38:16.000000000 -0500 Change: 2024-12-02 13:38:17.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:38:18 (1733164698) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 791 sec ============ 13:38:19 (1733164699) === sanity-flr: start cleanup 13:38:19 (1733164699) === === sanity-flr: finish cleanup 13:38:20 (1733164700) ===