-----============= acceptance-small: sanity-flr ============----- Fri Sep 20 13:05:54 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:06:02 (1726851962) === oleg248-client.virtnet: executing check_config_client /mnt/lustre oleg248-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg248-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6f89000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6f89000.idle_timeout=debug disable quota as required oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:06:15 (1726851975) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7721] == sanity-flr test 0a: lfs mirror create with -N option == 13:06:15 (1726851975) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:06:23 (1726851983) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:06:24 (1726851984) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:06:26 (1726851986) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:06:36 (1726851996) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:06:38 (1726851998) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 13:06:39 (1726851999) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6f89000.stats=clear osc.lustre-OST0001-osc-ffff8800b6f89000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:06:42 (1726852002) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:06:46 (1726852006) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:06:49 (1726852009) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:06:50 (1726852010) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 13:06:54 (1726852014) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:06:57 (1726852017) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:07:00 (1726852020) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:07:04 (1726852024) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:07:09 (1726852029) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:07:12 (1726852032) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 1.31846 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122684 s, 8.5 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:07:17 (1726852037) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.105895 s, 89.1 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:07:21 (1726852041) Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (28s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:07:49 (1726852069) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0903866 s, 46.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:08:04 (1726852084) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:08:31 (1726852111) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.0291 s, 50.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 20 13:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.405s user 0m0.002s sys 0m0.136s Stopping /mnt/lustre-ost1 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.423s user 0m0.000s sys 0m0.154s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg248-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg248-server oleg248-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.409s user 0m0.002s sys 0m0.129s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg248-server: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg248-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f89000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:08:59 (1726852139) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:09:01 (1726852141) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:09:02 (1726852142) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:09:03 (1726852143) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 13:09:07 (1726852147) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.282324 s, 44.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:09:12 (1726852152) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124457 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180896 s, 58.0 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:09:15 (1726852155) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:09:18 (1726852158) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00121095 s, 846 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000808936 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0012293 s, 833 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0011311 s, 905 kB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 13:09:22 (1726852162) create test files with size 15743196 .. d8166e0a03a1e4735a5c165a6d89fc18 - a05fab3a806a9a89b629d22a5062a819 - 20b65cfd8d84254b1d877926371c0999 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 13:09:33 (1726852173) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:09:39 (1726852179) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:09:42 (1726852182) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0644549 s, 16.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196884 s, 53.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 13:09:46 (1726852186) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0721893 s, 29.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.153631 s, 27.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0952701 s, 22.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 13:09:51 (1726852191) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 13:09:53 (1726852193) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:09:54 (1726852194) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294855 s, 35.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 13:09:58 (1726852198) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.195127 s, 16.1 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:10:03 (1726852203) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x49:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:10:07 (1726852207) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.129725 s, 80.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (5s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:10:12 (1726852212) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a7894000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.127195 s, 82.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg248-client.virtnet /mnt/lustre (opts:) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b6f8f000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:10:19 (1726852219) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (14s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:10:33 (1726852233) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:10:45 (1726852245) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:10:47 (1726852247) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0835452 s, 37.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0602809 s, 52.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 6313b944c0224311a2ccd65a45913554 - mirror 1 checksum: d55725ae203a093853ab0995134a5740 - mirror 2 checksum: 6313b944c0224311a2ccd65a45913554 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Sep 20 13:10:46 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Sep 20 13:10:46 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6313b944c0224311a2ccd65a45913554 - mirror 1 checksum: d55725ae203a093853ab0995134a5740 - mirror 2 checksum: d55725ae203a093853ab0995134a5740 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Sep 20 13:10:46 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Sep 20 13:10:46 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:10:51 (1726852251) Failing mds1 on oleg248-server Stopping /mnt/lustre-mds1 (opts:) on oleg248-server 13:10:53 (1726852253) shut down Failover mds1 to oleg248-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 13:11:06 (1726852266) targets are mounted 13:11:06 (1726852266) facet_failover done oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193033 s, 54.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:11:15 (1726852275) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:11:16 (1726852276) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:11:17 (1726852277) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00564126 s, 2.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:11:20 (1726852280) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:11:21 (1726852281) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:11:22 (1726852282) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xb:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:11:31 (1726852291) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:11:51 (1726852311) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x14:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:12:10 (1726852330) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:12:30 (1726852350) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 13:12:34 (1726852354) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 23730 mirror split pid 23732 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7472: 23730 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 23732 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 13:13:37 (1726852417) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2418 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1e9ef 000002 1726852418.134963 trunc from 00000000 to 0x01e9ee (0x1e9ef bytes) 000004 1726852418.144998 mapread 0x015279 thru 0x01e9ee (0x9776 bytes) 000005 1726852418.160610 MIRROR_EXTEND to 2 mirrors 000006 1726852418.394481 MIRROR_RESYNC 2 mirrors 000007 1726852418.466409 write 0x01c709 thru 0x02623c (0x9b34 bytes) 000008 1726852418.498742 mapread 0x002c6a thru 0x00b824 (0x8bbb bytes) 000011 1726852418.507885 trunc from 0x02623d to 0x01d1a7 (0x9095 bytes) truncating to largest ever: 0x32346 000012 1726852418.524847 trunc from 0x01d1a8 to 0x032345 (0x1519e bytes) 000015 1726852418.534295 write 0x0065a6 thru 0x0082af (0x1d0a bytes) 000019 1726852448.096877 read 0x023198 thru 0x025b02 (0x296b bytes) 000020 1726852448.113007 MIRROR_EXTEND to 3 mirrors 000027 1726852448.417481 mapwrite 0x00cd1e thru 0x010b07 (0x3dea bytes) 000028 1726852448.539307 write 0x03a48d thru 0x03c5c0 (0x2134 bytes) 000029 1726852448.543459 MIRROR_RESYNC 3 mirrors 000030 1726852449.020329 mapwrite 0x02aa46 thru 0x0337e8 (0x8da3 bytes) 000031 1726852449.117818 mapwrite 0x01fc58 thru 0x028b64 (0x8f0d bytes) 000032 1726852449.168826 mapread 0x02277f thru 0x023eb6 (0x1738 bytes) 000033 1726852449.170448 write 0x0223bb thru 0x02eaa2 (0xc6e8 bytes) 000034 1726852449.173779 write 0x02080f thru 0x02f20d (0xe9ff bytes) 000035 1726852449.181367 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000037 1726852449.383929 trunc from 0x03c5c1 to 0x02cfa5 (0xf61b bytes) 000038 1726852449.420422 trunc from 0x02cfa6 to 0x010992 (0x1c613 bytes) 000039 1726852449.441772 mapread 0x002970 thru 0x010992 (0xe023 bytes) 000044 1726852449.482381 read 0x005635 thru 0x00e78f (0x915b bytes) 000045 1726852449.491000 MIRROR_RESYNC 2 mirrors 000046 1726852449.863703 trunc from 0x010993 to 0x01f6fd (0xed6b bytes) 000047 1726852449.932775 mapwrite 0x00fa17 thru 0x012f12 (0x34fc bytes) 000051 1726852450.015278 write 0x035fb7 thru 0x038ee6 (0x2f30 bytes) 000052 1726852450.020560 MIRROR_EXTEND to 3 mirrors 000053 1726852450.371907 write 0x0271b3 thru 0x02fa23 (0x8871 bytes) 000054 1726852450.419658 mapwrite 0x037c7f thru 0x03ffff (0x8381 bytes) 000055 1726852450.487762 mapwrite 0x0205e4 thru 0x02c98d (0xc3aa bytes) 000056 1726852450.540618 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000062 1726852450.677738 MIRROR_RESYNC 2 mirrors 000063 1726852450.975190 write 0x01dae9 thru 0x01e90e (0xe26 bytes) 000064 1726852451.009697 trunc from 0x040000 to 0x01b664 (0x2499b bytes) 000067 1726852451.027754 mapwrite 0x035454 thru 0x03ffff (0xabac bytes) 000068 1726852451.096450 mapwrite 0x03368f thru 0x03ffff (0xc971 bytes) 000069 1726852451.140619 MIRROR_EXTEND to 3 mirrors 000070 1726852451.419408 MIRROR_RESYNC 3 mirrors 000073 1726852451.648055 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000076 1726852451.918929 mapread 0x03a9e1 thru 0x03ffff (0x561f bytes) 000079 1726852451.992859 MIRROR_RESYNC 2 mirrors 000080 1726852452.168446 read 0x01256f thru 0x01340b (0xe9d bytes) 000081 1726852452.177619 read 0x02e467 thru 0x0350ae (0x6c48 bytes) 000083 1726852452.205751 MIRROR_EXTEND to 3 mirrors 000084 1726852452.562939 trunc from 0x040000 to 0x01f169 (0x20e96 bytes) 000087 1726852452.644057 mapwrite 0x0183eb thru 0x01b347 (0x2f5d bytes) 000088 1726852452.725966 read 0x003387 thru 0x0132c6 (0xff40 bytes) 000089 1726852452.743630 write 0x02efe5 thru 0x03b2a2 (0xc2be bytes) 000091 1726852452.745239 trunc from 0x03b2a3 to 0x00b65e (0x2fc44 bytes) 000092 1726852452.758561 mapread 0x000140 thru 0x002532 (0x23f3 bytes) 000094 1726852452.767261 read 0x00930e thru 0x00b65e (0x2351 bytes) 000095 1726852452.768258 write 0x03e442 thru 0x03ffff (0x1bbe bytes) 000096 1726852452.775830 MIRROR_EXTEND to 4 mirrors truncating to largest ever: 0x3b88c 000099 1726852453.066423 trunc from 0x040000 to 0x03b88b (0x4774 bytes) 000100 1726852453.123522 mapread 0x004a9f thru 0x012d8d (0xe2ef bytes) 000101 1726852453.145508 mapwrite 0x02d6b7 thru 0x03900c (0xb956 bytes) 000102 1726852453.224021 mapwrite 0x002a85 thru 0x00ccae (0xa22a bytes) 000103 1726852453.275371 write 0x03353c thru 0x03b25b (0x7d20 bytes) 000108 1726852453.280556 write 0x027c1c thru 0x02c031 (0x4416 bytes) 000109 1726852453.281410 trunc from 0x03b88c to 0x0117b7 (0x2a0d4 bytes) 000112 1726852453.305211 MIRROR_RESYNC 4 mirrors 000113 1726852453.731731 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000116 1726852453.886954 write 0x030440 thru 0x033228 (0x2de9 bytes) 000117 1726852453.962094 MIRROR_EXTEND to 4 mirrors 000119 1726852454.309663 mapwrite 0x01db84 thru 0x028a74 (0xaef1 bytes) 000120 1726852454.436384 mapwrite 0x021096 thru 0x02f78a (0xe6f5 bytes) 000121 1726852454.478723 mapread 0x00528a thru 0x01275e (0xd4d5 bytes) 000123 1726852454.486974 mapwrite 0x01148c thru 0x01d952 (0xc4c7 bytes) 000124 1726852454.527536 read 0x0068b6 thru 0x0086c8 (0x1e13 bytes) 000125 1726852454.527987 mapwrite 0x03ef1a thru 0x03ffff (0x10e6 bytes) 000129 1726852454.597288 mapwrite 0x031527 thru 0x031ff8 (0xad2 bytes) 000130 1726852454.646845 MIRROR_EXTEND to 5 mirrors 000132 1726852454.894766 trunc from 0x040000 to 0x00f766 (0x30899 bytes) 000134 1726852454.963465 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000136 1726852455.087510 MIRROR_RESYNC 4 mirrors 000140 1726852455.472217 write 0x00a2e3 thru 0x0150d8 (0xadf6 bytes) 000141 1726852455.525273 mapwrite 0x02e0bf thru 0x031be2 (0x3b24 bytes) 000144 1726852455.609728 mapwrite 0x01e396 thru 0x027e50 (0x9abb bytes) 000145 1726852455.662172 MIRROR_RESYNC 4 mirrors 000148 1726852456.021160 write 0x039753 thru 0x03a0e7 (0x995 bytes) 000153 1726852456.068595 mapwrite 0x00f0d9 thru 0x018341 (0x9269 bytes) 000155 1726852456.146641 MIRROR_EXTEND to 5 mirrors 000156 1726852456.391948 mapwrite 0x02de12 thru 0x038b07 (0xacf6 bytes) 000159 1726852456.511466 write 0x01851c thru 0x01aff1 (0x2ad6 bytes) 000160 1726852456.518561 write 0x02598c thru 0x031508 (0xbb7d bytes) 000161 1726852456.523607 write 0x00b164 thru 0x011767 (0x6604 bytes) 000163 1726852456.530221 trunc from 0x03a0e8 to 0x026e1c (0x132cb bytes) 000164 1726852456.552203 MIRROR_RESYNC 5 mirrors 000165 1726852457.024082 mapread 0x00ee8e thru 0x0122e1 (0x3454 bytes) 000168 1726852457.035921 read 0x0129b1 thru 0x012e71 (0x4c1 bytes) 000170 1726852457.039375 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000172 1726852457.189732 trunc from 0x026e1d to 0x02cf20 (0x6104 bytes) 000173 1726852457.245279 mapwrite 0x032db1 thru 0x0369af (0x3bff bytes) 000174 1726852457.324129 write 0x03fee3 thru 0x03ffff (0x11d bytes) 000176 1726852457.331391 MIRROR_EXTEND to 5 mirrors 000177 1726852457.621508 mapread 0x01a0fe thru 0x026b88 (0xca8b bytes) 000181 1726852457.649337 mapwrite 0x02b4e7 thru 0x03b20f (0xfd29 bytes) 000182 1726852457.773605 mapread 0x02edbb thru 0x039267 (0xa4ad bytes) 000183 1726852457.774484 read 0x00637d thru 0x007f36 (0x1bba bytes) 000184 1726852457.778530 trunc from 0x040000 to 0x00540e (0x3abf1 bytes) 000185 1726852457.811431 MIRROR_RESYNC 5 mirrors 000189 1726852458.321102 trunc from 0x00540f to 0x033175 (0x2dd67 bytes) 000191 1726852458.386058 mapwrite 0x034a3f thru 0x03ffff (0xb5c1 bytes) 000194 1726852458.472475 trunc from 0x040000 to 0x0188d1 (0x2772e bytes) 000195 1726852458.485796 write 0x004292 thru 0x005c4c (0x19bb bytes) 000196 1726852458.494163 write 0x02746d thru 0x0367b2 (0xf346 bytes) 000200 1726852458.496763 write 0x00dd38 thru 0x012c14 (0x4edd bytes) 000201 1726852458.502420 mapwrite 0x018477 thru 0x021d70 (0x98fa bytes) 000202 1726852458.563374 write 0x03c12d thru 0x03ed51 (0x2c25 bytes) 000203 1726852458.564178 trunc from 0x03ed52 to 0x007686 (0x376cb bytes) 000204 1726852458.580669 mapread 0x000360 thru 0x0037e3 (0x3484 bytes) 000205 1726852458.594992 MIRROR_EXTEND to 6 mirrors 000209 1726852458.939439 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000211 1726852459.096294 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000212 1726852459.260516 write 0x00fd3f thru 0x01ec86 (0xef48 bytes) 000215 1726852459.312789 mapread 0x01c494 thru 0x01ec86 (0x27f3 bytes) 000217 1726852459.320509 mapread 0x0077e0 thru 0x015db0 (0xe5d1 bytes) 000218 1726852459.333745 read 0x014676 thru 0x01d81f (0x91aa bytes) 000219 1726852459.334204 write 0x02ce2a thru 0x02ee39 (0x2010 bytes) 000220 1726852459.334857 trunc from 0x02ee3a to 0x00c4a2 (0x22997 bytes) 000221 1726852459.364185 mapread 0x0067dd thru 0x00a4a7 (0x3ccb bytes) 000222 1726852459.369297 MIRROR_RESYNC 4 mirrors 000224 1726852459.924611 read 0x004fd3 thru 0x00c4a2 (0x74d0 bytes) 000225 1726852459.932777 mapwrite 0x0211d0 thru 0x0276f2 (0x6523 bytes) 000226 1726852460.097180 read 0x0022fe thru 0x005131 (0x2e34 bytes) 000227 1726852460.105992 trunc from 0x0276f3 to 0x0026dc (0x25016 bytes) 000232 1726852460.121380 mapread 0x000624 thru 0x0026dc (0x20b9 bytes) 000233 1726852460.123303 mapread 0x0010f1 thru 0x001890 (0x7a0 bytes) 000234 1726852460.126427 write 0x019695 thru 0x01e7da (0x5146 bytes) truncating to largest ever: 0x3e0fc 000235 1726852460.127874 trunc from 0x01e7db to 0x03e0fb (0x1f921 bytes) 000237 1726852460.140565 write 0x006d5a thru 0x00bed2 (0x5179 bytes) 000241 1726852460.150967 mapwrite 0x01271d thru 0x01797f (0x5263 bytes) 000243 1726852460.220116 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000244 1726852460.366899 MIRROR_EXTEND to 4 mirrors 000246 1726852460.719671 mapwrite 0x00a5cd thru 0x014d05 (0xa739 bytes) 000247 1726852460.893616 trunc from 0x03e0fc to 0x0329c9 (0xb732 bytes) 000250 1726852460.930764 MIRROR_RESYNC 4 mirrors 000251 1726852461.454672 write 0x017119 thru 0x018660 (0x1548 bytes) 000253 1726852461.515532 write 0x039842 thru 0x03ffff (0x67be bytes) 000254 1726852461.516809 mapwrite 0x010cc7 thru 0x01836d (0x76a7 bytes) 000255 1726852461.581576 read 0x0392ca thru 0x03efb8 (0x5cef bytes) 000256 1726852461.582051 read 0x023238 thru 0x02a827 (0x75f0 bytes) 000257 1726852461.590093 mapread 0x03f361 thru 0x03ffff (0xc9f bytes) 000259 1726852461.590980 mapwrite 0x016e54 thru 0x019e13 (0x2fc0 bytes) 000260 1726852461.637535 trunc from 0x040000 to 0x027a59 (0x185a6 bytes) 000261 1726852461.667330 MIRROR_EXTEND to 5 mirrors 000262 1726852462.007033 write 0x02e69d thru 0x038eac (0xa810 bytes) 000264 1726852462.059585 trunc from 0x038ead to 0x025eea (0x12fc2 bytes) 000265 1726852462.086473 trunc from 0x025eeb to 0x02a695 (0x47ab bytes) 000266 1726852462.096053 write 0x03f03c thru 0x03ffff (0xfc4 bytes) 000268 1726852462.097117 write 0x024d75 thru 0x034b2b (0xfdb7 bytes) 000269 1726852462.105421 write 0x0387d4 thru 0x03ffff (0x782c bytes) 000270 1726852462.112304 write 0x01dd4b thru 0x02416e (0x6424 bytes) 000271 1726852462.126409 MIRROR_EXTEND to 6 mirrors 000272 1726852462.525202 read 0x0236b4 thru 0x028b98 (0x54e5 bytes) 000275 1726852462.553630 trunc from 0x040000 to 0x031a3f (0xe5c0 bytes) 000276 1726852462.618131 trunc from 0x031a40 to 0x013993 (0x1e0ac bytes) 000277 1726852462.632280 mapwrite 0x03ba90 thru 0x03ffff (0x4570 bytes) 000281 1726852462.696833 trunc from 0x040000 to 0x022b79 (0x1d486 bytes) 000282 1726852462.711048 read 0x00e586 thru 0x01dcc7 (0xf742 bytes) 000283 1726852462.717132 trunc from 0x022b7a to 0x00a444 (0x18735 bytes) 000284 1726852462.734534 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000285 1726852462.888692 write 0x003f51 thru 0x00d559 (0x9609 bytes) 000286 1726852462.894783 mapread 0x0004aa thru 0x00d559 (0xd0b0 bytes) 000291 1726852462.908769 write 0x03ecbf thru 0x03ffff (0x1341 bytes) 000292 1726852462.911906 write 0x03d2a9 thru 0x03ffff (0x2d57 bytes) 000293 1726852462.918745 mapwrite 0x03f726 thru 0x03ffff (0x8da bytes) 000296 1726852462.988861 read 0x03f68d thru 0x03ffff (0x973 bytes) 000297 1726852462.989356 mapread 0x02daf2 thru 0x02f208 (0x1717 bytes) 000300 1726852463.005297 write 0x021839 thru 0x02ab73 (0x933b bytes) 000302 1726852463.013590 trunc from 0x040000 to 0x01673d (0x298c2 bytes) 000303 1726852463.021704 mapwrite 0x03f1af thru 0x03ffff (0xe51 bytes) 000307 1726852463.079500 MIRROR_EXTEND to 6 mirrors 000309 1726852463.406946 MIRROR_RESYNC 6 mirrors 000311 1726852463.913746 mapwrite 0x017cee thru 0x0249a7 (0xccba bytes) 000315 1726852464.080897 MIRROR_EXTEND to 7 mirrors 000316 1726852464.432312 write 0x012fab thru 0x019797 (0x67ed bytes) 000319 1726852464.481961 mapwrite 0x00718e thru 0x013625 (0xc498 bytes) 000321 1726852464.536725 mapread 0x01abd8 thru 0x029c3b (0xf064 bytes) 000322 1726852464.552191 mapwrite 0x0384e2 thru 0x03ffff (0x7b1e bytes) 000323 1726852464.592226 mapwrite 0x028a24 thru 0x0364f9 (0xdad6 bytes) 000324 1726852464.626326 read 0x00c739 thru 0x011293 (0x4b5b bytes) 000327 1726852464.630811 MIRROR_EXTEND to 8 mirrors 000330 1726852464.913563 MIRROR_RESYNC 8 mirrors 000331 1726852465.388521 read 0x0261ae thru 0x02867f (0x24d2 bytes) 000334 1726852465.393802 read 0x034dfd thru 0x03ffff (0xb203 bytes) 000336 1726852465.401783 read 0x03a339 thru 0x03ffff (0x5cc7 bytes) 000338 1726852465.402221 read 0x0015b4 thru 0x00b206 (0x9c53 bytes) 000339 1726852465.402433 read 0x01edc7 thru 0x021b4d (0x2d87 bytes) 000340 1726852465.402648 read 0x00beb5 thru 0x01214a (0x6296 bytes) 000341 1726852465.405028 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000342 1726852465.615776 trunc from 0x040000 to 0x02dfc6 (0x12039 bytes) 000346 1726852465.683371 write 0x016276 thru 0x01ee43 (0x8bce bytes) 000347 1726852465.688219 mapwrite 0x0203af thru 0x0213f4 (0x1046 bytes) 000349 1726852465.724732 trunc from 0x02dfc7 to 0x007992 (0x26634 bytes) 000350 1726852465.738936 read 0x005eea thru 0x006a0e (0xb25 bytes) 000351 1726852465.744076 read 0x00446f thru 0x007992 (0x3524 bytes) 000352 1726852465.749018 trunc from 0x007993 to 0x001368 (0x662a bytes) 000354 1726852465.762723 write 0x036ec7 thru 0x03ffff (0x9139 bytes) 000355 1726852465.766084 mapwrite 0x01c267 thru 0x01c541 (0x2db bytes) 000357 1726852465.819956 write 0x0272e7 thru 0x02c4db (0x51f5 bytes) 000358 1726852465.820661 mapread 0x010c14 thru 0x02057c (0xf969 bytes) 000359 1726852465.821571 mapread 0x03dc18 thru 0x03ffff (0x23e8 bytes) 000362 1726852465.827870 MIRROR_EXTEND to 8 mirrors 000363 1726852466.207807 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000365 1726852466.461090 read 0x0373ad thru 0x03ec9f (0x78f3 bytes) 000370 1726852466.509986 read 0x02466a thru 0x02fe57 (0xb7ee bytes) 000373 1726852466.544059 mapread 0x011e2d thru 0x0187e9 (0x69bd bytes) 000374 1726852466.557084 trunc from 0x040000 to 0x01cb37 (0x234c8 bytes) 000375 1726852466.610729 mapread 0x012688 thru 0x013bf5 (0x156e bytes) 000378 1726852466.622073 trunc from 0x01cb38 to 0x036b77 (0x1a040 bytes) 000379 1726852466.646832 read 0x00a828 thru 0x010fdf (0x67b8 bytes) 000383 1726852466.657002 write 0x018db0 thru 0x01e13b (0x538c bytes) 000384 1726852466.657843 mapwrite 0x032b24 thru 0x03ffff (0xd4dc bytes) 000385 1726852466.749169 trunc from 0x040000 to 0x00fa8b (0x30574 bytes) 000386 1726852466.765960 mapwrite 0x00550b thru 0x0101ca (0xacc0 bytes) 000387 1726852466.830991 mapwrite 0x007535 thru 0x00bac5 (0x4591 bytes) 000390 1726852466.874120 mapread 0x00ab62 thru 0x00f2a5 (0x4744 bytes) 000392 1726852466.876063 write 0x02da45 thru 0x035277 (0x7833 bytes) 000393 1726852466.878698 write 0x00c8b8 thru 0x00ef57 (0x26a0 bytes) 000394 1726852466.879543 mapread 0x034f2f thru 0x035277 (0x349 bytes) 000399 1726852466.882228 trunc from 0x035278 to 0x037538 (0x22c1 bytes) 000400 1726852466.896802 write 0x02234a thru 0x03098d (0xe644 bytes) 000401 1726852466.903718 trunc from 0x037539 to 0x00ef46 (0x285f2 bytes) 000402 1726852466.915583 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000404 1726852467.084830 MIRROR_EXTEND to 7 mirrors 000405 1726852467.473637 trunc from 0x00ef47 to 0x01a9d5 (0xba8f bytes) 000406 1726852467.535732 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000407 1726852467.702360 MIRROR_RESYNC 6 mirrors 000408 1726852468.393365 write 0x00ac27 thru 0x010158 (0x5532 bytes) 000409 1726852468.450329 read 0x007832 thru 0x016f40 (0xf70f bytes) 000410 1726852468.458721 mapread 0x009303 thru 0x00e8a6 (0x55a4 bytes) 000413 1726852468.460312 read 0x00a653 thru 0x0118e4 (0x7292 bytes) 000414 1726852468.461439 write 0x00d13a thru 0x01612b (0x8ff2 bytes) 000415 1726852468.462724 write 0x018545 thru 0x022035 (0x9af1 bytes) 000416 1726852468.469845 mapread 0x008e1c thru 0x00da42 (0x4c27 bytes) 000417 1726852468.471299 write 0x030eb6 thru 0x03cf3d (0xc088 bytes) 000420 1726852468.473349 write 0x035f06 thru 0x03ffff (0xa0fa bytes) 000421 1726852468.474842 mapwrite 0x0363ee thru 0x03f7ce (0x93e1 bytes) 000422 1726852468.537821 MIRROR_RESYNC 6 mirrors 000423 1726852469.098556 MIRROR_RESYNC 6 mirrors 000425 1726852469.226581 trunc from 0x040000 to 0x012975 (0x2d68a bytes) 000429 1726852469.300658 write 0x024197 thru 0x028b69 (0x49d3 bytes) 000430 1726852469.301493 mapwrite 0x039e73 thru 0x03bf5d (0x20eb bytes) 000431 1726852469.379197 mapwrite 0x01dd3a thru 0x02d751 (0xfa18 bytes) 000435 1726852469.454218 mapread 0x029a89 thru 0x030eb8 (0x7430 bytes) 000436 1726852469.455711 write 0x02acb7 thru 0x03918a (0xe4d4 bytes) 000437 1726852469.456652 write 0x032e71 thru 0x03ffff (0xd18f bytes) 000439 1726852469.463986 MIRROR_EXTEND to 7 mirrors 000440 1726852469.781095 read 0x02c65e thru 0x02cbb9 (0x55c bytes) 000441 1726852469.789596 mapwrite 0x03baae thru 0x03ffff (0x4552 bytes) 000444 1726852469.943421 mapread 0x0238e4 thru 0x032999 (0xf0b6 bytes) 000445 1726852469.957731 read 0x03c81e thru 0x03ffff (0x37e2 bytes) 000446 1726852469.967915 MIRROR_EXTEND to 8 mirrors 000447 1726852470.364402 mapwrite 0x02d55f thru 0x02f8cf (0x2371 bytes) 000449 1726852470.525453 write 0x01a726 thru 0x028dbb (0xe696 bytes) 000453 1726852470.535538 write 0x03bc2d thru 0x03f6d0 (0x3aa4 bytes) 000454 1726852470.540073 mapwrite 0x0316da thru 0x03d1a2 (0xbac9 bytes) 000455 1726852470.602210 trunc from 0x040000 to 0x004564 (0x3ba9b bytes) 000458 1726852470.617217 trunc from 0x004565 to 0x0034ee (0x1076 bytes) 000460 1726852470.634487 MIRROR_EXTEND to 9 mirrors 000463 1726852470.998592 mapwrite 0x023612 thru 0x02f5a9 (0xbf98 bytes) 000466 1726852471.131125 mapread 0x00e268 thru 0x00efb2 (0xd4b bytes) 000468 1726852471.140976 mapwrite 0x00b298 thru 0x018023 (0xcd8c bytes) 000469 1726852471.213549 mapread 0x0089ae thru 0x00e631 (0x5c84 bytes) 000470 1726852471.214764 mapwrite 0x0172d3 thru 0x020782 (0x94b0 bytes) 000475 1726852471.264661 MIRROR_EXTEND to 10 mirrors 000477 1726852471.607868 read 0x02592d thru 0x02f5a9 (0x9c7d bytes) 000479 1726852471.624075 write 0x00f920 thru 0x01571e (0x5dff bytes) 000480 1726852471.682087 MIRROR_EXTEND to 11 mirrors 000481 1726852471.954200 mapread 0x002602 thru 0x00e91d (0xc31c bytes) 000482 1726852471.975555 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000483 1726852472.126777 write 0x03edcc thru 0x03ffff (0x1234 bytes) 000485 1726852472.183331 write 0x02fba5 thru 0x034497 (0x48f3 bytes) 000486 1726852472.187677 read 0x029c24 thru 0x0373a0 (0xd77d bytes) 000487 1726852472.193442 write 0x0176e4 thru 0x0188ac (0x11c9 bytes) 000488 1726852472.202092 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000489 1726852472.325015 write 0x00c3d2 thru 0x01003f (0x3c6e bytes) 000491 1726852472.335199 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000492 1726852472.459813 trunc from 0x040000 to 0x02678f (0x19870 bytes) 000495 1726852472.480578 mapread 0x0017ac thru 0x00842d (0x6c82 bytes) 000496 1726852472.488751 write 0x039b50 thru 0x03ffff (0x64b0 bytes) 000499 1726852472.490453 trunc from 0x040000 to 0x03ac8b (0x5374 bytes) 000503 1726852472.500868 write 0x03cf56 thru 0x03ffff (0x30aa bytes) 000504 1726852472.501987 mapwrite 0x0005da thru 0x001a23 (0x144a bytes) 000506 1726852472.546390 write 0x038c67 thru 0x03fa4f (0x6de9 bytes) 000509 1726852472.554026 trunc from 0x040000 to 0x036ef3 (0x910c bytes) 000510 1726852472.563045 mapwrite 0x029c22 thru 0x030314 (0x66f3 bytes) 000513 1726852472.615290 trunc from 0x036ef4 to 0x02fd4c (0x71a7 bytes) 000514 1726852472.630310 write 0x028c02 thru 0x02ba27 (0x2e26 bytes) 000519 1726852472.635635 MIRROR_RESYNC 8 mirrors 000523 1726852473.284043 write 0x00e0d2 thru 0x01daf8 (0xfa27 bytes) 000526 1726852473.341579 write 0x02408b thru 0x024443 (0x3b9 bytes) 000528 1726852473.353338 trunc from 0x02fd4d to 0x0183f7 (0x17955 bytes) 000529 1726852473.369089 MIRROR_EXTEND to 9 mirrors 000530 1726852473.709963 read 0x0160bf thru 0x0183f7 (0x2339 bytes) 000531 1726852473.713128 mapwrite 0x00a1ef thru 0x00fb82 (0x5994 bytes) 000533 1726852473.820690 MIRROR_EXTEND to 10 mirrors 000534 1726852474.239620 MIRROR_RESYNC 10 mirrors 000535 1726852474.913473 trunc from 0x0183f8 to 0x03ae0e (0x22a17 bytes) 000537 1726852474.988556 trunc from 0x03ae0f to 0x00cefa (0x2df14 bytes) 000538 1726852475.002279 read 0x001512 thru 0x0061eb (0x4cda bytes) 000539 1726852475.007251 read 0x001460 thru 0x00cefa (0xba9b bytes) 000540 1726852475.011140 trunc from 0x00cefb to 0x00a4c4 (0x2a36 bytes) 000541 1726852475.024852 mapwrite 0x0139fb thru 0x0155de (0x1be4 bytes) 000542 1726852475.085027 read 0x00f2b7 thru 0x011b7c (0x28c6 bytes) 000545 1726852475.089794 write 0x039a4f thru 0x03f884 (0x5e36 bytes) 000546 1726852475.091652 mapread 0x0273a5 thru 0x02e17a (0x6dd6 bytes) 000547 1726852475.102792 MIRROR_EXTEND to 11 mirrors 000549 1726852475.384488 mapread 0x0231ef thru 0x0284b0 (0x52c2 bytes) 000550 1726852475.396884 read 0x019d4d thru 0x01caf6 (0x2daa bytes) 000551 1726852475.403337 read 0x0019ca thru 0x009727 (0x7d5e bytes) 000552 1726852475.413874 write 0x0073e1 thru 0x00f809 (0x8429 bytes) 000553 1726852475.462140 write 0x03b84a thru 0x03ffff (0x47b6 bytes) 000555 1726852475.466319 read 0x036da5 thru 0x03ffff (0x925b bytes) 000558 1726852475.473117 read 0x0010b6 thru 0x00b440 (0xa38b bytes) 000559 1726852475.473645 trunc from 0x040000 to 0x01ec50 (0x213af bytes) 000561 1726852475.491641 mapread 0x003d56 thru 0x006557 (0x2802 bytes) 000564 1726852475.493412 write 0x0025e6 thru 0x006d94 (0x47af bytes) 000565 1726852475.494177 write 0x039ac9 thru 0x03c249 (0x2781 bytes) 000566 1726852475.494876 trunc from 0x03c24a to 0x02e193 (0xe0b6 bytes) 000567 1726852475.503392 trunc from 0x02e194 to 0x010ccd (0x1d4c6 bytes) 000568 1726852475.514556 mapread 0x000322 thru 0x003247 (0x2f26 bytes) 000569 1726852475.518543 MIRROR_RESYNC 11 mirrors 000572 1726852476.309011 read 0x0091ba thru 0x0099ff (0x846 bytes) 000574 1726852476.312460 write 0x01525a thru 0x015771 (0x518 bytes) 000575 1726852476.351251 mapwrite 0x02c65e thru 0x033b58 (0x74fb bytes) 000576 1726852476.402045 trunc from 0x033b59 to 0x01af22 (0x18c36 bytes) 000578 1726852476.411210 write 0x03ba0f thru 0x03bbbc (0x1ae bytes) 000579 1726852476.412849 write 0x030508 thru 0x03dc27 (0xd720 bytes) 000580 1726852476.421662 trunc from 0x03dc28 to 0x0302bf (0xd968 bytes) 000582 1726852476.430980 mapread 0x020358 thru 0x02a763 (0xa40c bytes) 000583 1726852476.440834 read 0x01eac5 thru 0x02434f (0x588b bytes) 000584 1726852476.441407 read 0x00046c thru 0x00fa4a (0xf5df bytes) 000586 1726852476.447467 mapwrite 0x0141b9 thru 0x01b433 (0x727b bytes) 000587 1726852476.491218 trunc from 0x0302c0 to 0x009764 (0x26b5b bytes) 000588 1726852476.502042 MIRROR_EXTEND to 12 mirrors 000589 1726852476.738811 write 0x011b5c thru 0x01c661 (0xab06 bytes) 000592 1726852476.800840 mapread 0x014be7 thru 0x01c661 (0x7a7b bytes) 000594 1726852476.801877 write 0x03eb07 thru 0x03ffff (0x14f9 bytes) 000597 1726852476.803616 trunc from 0x040000 to 0x0247c6 (0x1b839 bytes) 000600 1726852476.837602 write 0x03d836 thru 0x03ffff (0x27ca bytes) 000603 1726852476.838980 mapread 0x03002b thru 0x03a306 (0xa2dc bytes) 000606 1726852476.868054 mapread 0x02a736 thru 0x02c9d3 (0x229e bytes) 000607 1726852476.868671 trunc from 0x040000 to 0x0309c7 (0xf638 bytes) 000608 1726852476.877064 read 0x020334 thru 0x027aac (0x7779 bytes) 000609 1726852476.883941 write 0x03a113 thru 0x03ffff (0x5eed bytes) 000610 1726852476.885229 mapread 0x039100 thru 0x03ffff (0x6f00 bytes) 000612 1726852476.891885 mapread 0x036360 thru 0x03c484 (0x6125 bytes) 000613 1726852476.893394 mapwrite 0x02ad42 thru 0x02bf9a (0x1259 bytes) 000614 1726852476.937376 mapwrite 0x017469 thru 0x01b8db (0x4473 bytes) 000615 1726852476.981543 MIRROR_EXTEND to 13 mirrors 000616 1726852477.263586 trunc from 0x040000 to 0x02cc98 (0x13367 bytes) 000619 1726852477.317278 mapread 0x029021 thru 0x02cc98 (0x3c78 bytes) 000620 1726852477.330619 mapwrite 0x02b08f thru 0x0339d2 (0x8944 bytes) 000622 1726852477.411250 trunc from 0x0339d3 to 0x016cd3 (0x1ccff bytes) 000623 1726852477.425929 mapread 0x00a39f thru 0x00c511 (0x2173 bytes) 000625 1726852477.436112 write 0x02f508 thru 0x03f2b1 (0xfdaa bytes) 000627 1726852477.440175 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000628 1726852477.571923 read 0x01110b thru 0x018d4d (0x7c43 bytes) 000630 1726852477.579490 read 0x02ca41 thru 0x034087 (0x7647 bytes) 000632 1726852477.587683 read 0x0120e3 thru 0x01d8e5 (0xb803 bytes) 000635 1726852477.588782 read 0x02ae41 thru 0x02f15b (0x431b bytes) 000636 1726852477.589196 mapwrite 0x01e524 thru 0x022cb0 (0x478d bytes) 000637 1726852477.650263 mapwrite 0x004358 thru 0x00f88e (0xb537 bytes) 000638 1726852477.706918 mapread 0x02bc81 thru 0x0383fd (0xc77d bytes) 000639 1726852477.707730 mapread 0x00dd35 thru 0x01041c (0x26e8 bytes) 000640 1726852477.708464 write 0x0304ae thru 0x035691 (0x51e4 bytes) 000645 1726852477.710299 read 0x0082e9 thru 0x00e5b5 (0x62cd bytes) 000646 1726852477.712699 read 0x019699 thru 0x027545 (0xdead bytes) 000648 1726852477.717285 MIRROR_EXTEND to 13 mirrors 000649 1726852478.030304 mapwrite 0x0201fd thru 0x025185 (0x4f89 bytes) 000650 1726852478.232793 read 0x00d907 thru 0x016f05 (0x95ff bytes) 000652 1726852478.244083 MIRROR_RESYNC 13 mirrors 000653 1726852479.591561 mapwrite 0x02f5fc thru 0x039bf8 (0xa5fd bytes) 000654 1726852479.735881 write 0x032030 thru 0x03d633 (0xb604 bytes) 000655 1726852479.736922 mapwrite 0x0135b5 thru 0x017f37 (0x4983 bytes) 000657 1726852479.802051 mapread 0x011b5f thru 0x01e0bb (0xc55d bytes) 000658 1726852479.814941 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000659 1726852479.968492 trunc from 0x03f2b2 to 0x009e8c (0x35425 bytes) 000660 1726852479.988273 MIRROR_RESYNC 12 mirrors 000661 1726852480.946129 mapread 0x002e61 thru 0x006227 (0x33c7 bytes) 000662 1726852480.957772 trunc from 0x009e8d to 0x037222 (0x2d396 bytes) 000664 1726852481.005313 write 0x0288cb thru 0x028b50 (0x286 bytes) 000665 1726852481.017993 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000666 1726852481.189207 trunc from 0x037223 to 0x002242 (0x34fe0 bytes) 000667 1726852481.203180 mapread 0x0004f2 thru 0x002242 (0x1d51 bytes) 000668 1726852481.222299 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000669 1726852481.360115 mapread 0x0007f3 thru 0x002242 (0x1a50 bytes) 000670 1726852481.371270 trunc from 0x002243 to 0x021148 (0x1ef06 bytes) 000671 1726852481.381735 mapwrite 0x01539c thru 0x017b4f (0x27b4 bytes) 000672 1726852481.439437 read 0x01edef thru 0x0207b5 (0x19c7 bytes) 000673 1726852481.443849 mapwrite 0x03f4f7 thru 0x03ffff (0xb09 bytes) 000675 1726852481.510492 trunc from 0x040000 to 0x00ca20 (0x335df bytes) 000676 1726852481.525461 mapread 0x0000b3 thru 0x000a77 (0x9c5 bytes) 000677 1726852481.526261 read 0x002114 thru 0x00ca20 (0xa90d bytes) 000680 1726852481.541618 MIRROR_EXTEND to 11 mirrors 000681 1726852481.881196 mapwrite 0x00aebf thru 0x014bdf (0x9d21 bytes) 000682 1726852482.024375 trunc from 0x014be0 to 0x006479 (0xe766 bytes) 000684 1726852482.048395 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000686 1726852482.195488 mapwrite 0x03ce24 thru 0x03ea56 (0x1c33 bytes) 000687 1726852482.300132 MIRROR_EXTEND to 11 mirrors 000691 1726852482.715446 MIRROR_EXTEND to 12 mirrors 000692 1726852483.115479 read 0x035926 thru 0x037f01 (0x25dc bytes) 000695 1726852483.129088 trunc from 0x03ea57 to 0x0146b6 (0x2a3a0 bytes) 000697 1726852483.223718 mapread 0x0127d9 thru 0x0146b6 (0x1ede bytes) 000699 1726852483.236937 read 0x002320 thru 0x00b3d5 (0x90b6 bytes) 000701 1726852483.244585 trunc from 0x0146b7 to 0x02bb88 (0x174d2 bytes) 000702 1726852483.258841 write 0x00ec7b thru 0x00f3fb (0x781 bytes) 000703 1726852483.259974 read 0x0267aa thru 0x02bb88 (0x53df bytes) 000705 1726852483.272771 mapwrite 0x00abbd thru 0x01182b (0x6c6f bytes) 000706 1726852483.320842 trunc from 0x02bb89 to 0x0377e4 (0xbc5c bytes) 000709 1726852483.336968 mapread 0x034b42 thru 0x0377e4 (0x2ca3 bytes) 000710 1726852483.346611 write 0x003404 thru 0x012de1 (0xf9de bytes) 000711 1726852483.352513 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000712 1726852483.567744 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000716 1726852483.725035 read 0x009aea thru 0x015953 (0xbe6a bytes) 000719 1726852483.735560 trunc from 0x0377e5 to 0x008128 (0x2f6bc bytes) 000721 1726852483.752572 mapwrite 0x01fbd6 thru 0x020be7 (0x1012 bytes) 000722 1726852483.828806 read 0x011929 thru 0x015f2c (0x4604 bytes) 000723 1726852483.838223 mapread 0x0209ae thru 0x020be7 (0x23a bytes) 000725 1726852483.839432 trunc from 0x020be8 to 0x013f88 (0xcc5f bytes) 000726 1726852483.849478 trunc from 0x013f89 to 0x032f08 (0x1ef80 bytes) 000727 1726852483.861210 mapwrite 0x036c94 thru 0x03801a (0x1387 bytes) 000728 1726852483.925216 mapwrite 0x0388bc thru 0x03ffff (0x7744 bytes) 000731 1726852483.992349 trunc from 0x040000 to 0x02e9f3 (0x1160c bytes) 000732 1726852484.004063 write 0x02ab5e thru 0x0305d1 (0x5a74 bytes) 000733 1726852484.007786 MIRROR_RESYNC 10 mirrors 000734 1726852484.960087 mapread 0x018a42 thru 0x022556 (0x9b15 bytes) 000736 1726852484.991029 trunc from 0x0305d2 to 0x0361dd (0x5c0c bytes) 000737 1726852485.057556 MIRROR_EXTEND to 11 mirrors 000738 1726852485.460938 write 0x019ea9 thru 0x026fe1 (0xd139 bytes) 000740 1726852485.556010 MIRROR_EXTEND to 12 mirrors 000742 1726852486.023346 mapwrite 0x0000c9 thru 0x001d85 (0x1cbd bytes) 000743 1726852486.171478 mapwrite 0x01881c thru 0x01a0b2 (0x1897 bytes) 000745 1726852486.251927 mapread 0x030811 thru 0x0361dd (0x59cd bytes) 000746 1726852486.268119 trunc from 0x0361de to 0x039637 (0x345a bytes) 000748 1726852486.298874 trunc from 0x039638 to 0x000c62 (0x389d5 bytes) 000753 1726852486.331991 MIRROR_RESYNC 12 mirrors 000755 1726852487.303276 write 0x02933c thru 0x02c904 (0x35c9 bytes) 000757 1726852487.353931 mapread 0x0132c3 thru 0x021c12 (0xe950 bytes) 000758 1726852487.380054 mapread 0x000562 thru 0x008c46 (0x86e5 bytes) 000759 1726852487.397657 mapwrite 0x009c07 thru 0x00ec59 (0x5053 bytes) 000760 1726852487.449542 MIRROR_RESYNC 12 mirrors 000763 1726852488.255254 trunc from 0x02c905 to 0x0379d1 (0xb0cd bytes) 000765 1726852488.300460 mapwrite 0x00803f thru 0x0081ea (0x1ac bytes) 000766 1726852488.335058 mapwrite 0x00f7db thru 0x01d808 (0xe02e bytes) 000767 1726852488.392978 MIRROR_RESYNC 12 mirrors 000768 1726852489.101104 mapread 0x0101c8 thru 0x0111df (0x1018 bytes) 000769 1726852489.105761 mapwrite 0x020bd5 thru 0x02d5c7 (0xc9f3 bytes) 000770 1726852489.208569 MIRROR_RESYNC 12 mirrors 000771 1726852490.127676 trunc from 0x0379d2 to 0x02849c (0xf535 bytes) 000772 1726852490.188921 trunc from 0x02849d to 0x01e41b (0xa081 bytes) 000773 1726852490.207977 trunc from 0x01e41c to 0x03bde0 (0x1d9c5 bytes) 000775 1726852490.220057 mapwrite 0x03d218 thru 0x03ffff (0x2de8 bytes) 000777 1726852490.297460 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000778 1726852490.456765 trunc from 0x040000 to 0x026d06 (0x192f9 bytes) 000779 1726852490.468035 write 0x03f10e thru 0x03ffff (0xef2 bytes) 000780 1726852490.468800 mapread 0x03575a thru 0x03f676 (0x9f1d bytes) 000781 1726852490.482996 mapread 0x02cf13 thru 0x03c607 (0xf6f5 bytes) 000782 1726852490.492545 write 0x027c13 thru 0x02e71b (0x6b09 bytes) 000783 1726852490.494034 mapread 0x02f11f thru 0x032e26 (0x3d08 bytes) 000784 1726852490.494679 trunc from 0x040000 to 0x000d21 (0x3f2de bytes) 000785 1726852490.510640 MIRROR_RESYNC 11 mirrors 000786 1726852491.368368 MIRROR_RESYNC 11 mirrors 000787 1726852491.491899 mapread 0x00097d thru 0x000d21 (0x3a5 bytes) 000788 1726852491.493446 read 0x0005e3 thru 0x000d21 (0x73f bytes) 000789 1726852491.493814 read 0x000bf4 thru 0x000d21 (0x12e bytes) 000792 1726852491.494326 mapread 0x000360 thru 0x000d21 (0x9c2 bytes) 000793 1726852491.499858 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000795 1726852491.675888 trunc from 0x000d22 to 0x00419a (0x3479 bytes) 000800 1726852491.746494 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000805 1726852491.884709 mapread 0x0035e9 thru 0x00419a (0xbb2 bytes) 000807 1726852491.897282 mapwrite 0x033ec9 thru 0x03b62f (0x7767 bytes) 000809 1726852491.967649 mapwrite 0x0227eb thru 0x02695c (0x4172 bytes) 000810 1726852492.024522 mapread 0x03221e thru 0x0379d8 (0x57bb bytes) 000811 1726852492.025224 trunc from 0x03b630 to 0x009637 (0x31ff8 bytes) 000812 1726852492.043057 write 0x0273f4 thru 0x035587 (0xe194 bytes) 000814 1726852492.045075 read 0x02572e thru 0x02c5e0 (0x6eb3 bytes) 000815 1726852492.051055 read 0x011d55 thru 0x01f285 (0xd531 bytes) 000817 1726852492.063698 MIRROR_RESYNC 9 mirrors 000818 1726852492.856945 MIRROR_EXTEND to 10 mirrors 000819 1726852493.227191 read 0x021cba thru 0x023012 (0x1359 bytes) 000823 1726852493.230943 mapread 0x00e347 thru 0x01ca30 (0xe6ea bytes) 000827 1726852493.250631 read 0x019b59 thru 0x01bbb2 (0x205a bytes) 000828 1726852493.250920 write 0x00d2f9 thru 0x01653a (0x9242 bytes) 000830 1726852493.299636 write 0x021b2e thru 0x02ae50 (0x9323 bytes) 000831 1726852493.300368 trunc from 0x035588 to 0x016b7c (0x1ea0b bytes) 000833 1726852493.321553 trunc from 0x016b7d to 0x013c18 (0x2f64 bytes) 000835 1726852493.337924 mapread 0x00309a thru 0x00a5c3 (0x752a bytes) 000837 1726852493.340863 read 0x011ffb thru 0x012574 (0x57a bytes) 000839 1726852493.341637 write 0x01fa79 thru 0x022da8 (0x3330 bytes) 000840 1726852493.343181 mapread 0x00a148 thru 0x00ca52 (0x290b bytes) 000841 1726852493.345570 mapread 0x0210c1 thru 0x022da8 (0x1ce8 bytes) 000843 1726852493.347593 mapread 0x005cc5 thru 0x0141da (0xe516 bytes) 000848 1726852493.358676 mapread 0x00be47 thru 0x00f96a (0x3b24 bytes) 000849 1726852493.363447 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000850 1726852493.506125 MIRROR_RESYNC 9 mirrors 000851 1726852494.266830 trunc from 0x022da9 to 0x01c4b7 (0x68f1 bytes) 000854 1726852494.310398 write 0x01da32 thru 0x0210aa (0x3679 bytes) 000856 1726852494.312088 read 0x01ecde thru 0x0210aa (0x23cd bytes) 000857 1726852494.315291 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000861 1726852494.461748 write 0x017fe9 thru 0x01956a (0x1582 bytes) 000862 1726852494.470745 mapread 0x01cef4 thru 0x0210aa (0x41b7 bytes) 000863 1726852494.476421 trunc from 0x0210ab to 0x038782 (0x176d8 bytes) 000864 1726852494.485843 read 0x02370d thru 0x03285a (0xf14e bytes) 000865 1726852494.491980 write 0x000a8a thru 0x00c4a9 (0xba20 bytes) 000868 1726852494.502388 mapwrite 0x00bcb0 thru 0x015675 (0x99c6 bytes) 000869 1726852494.569243 mapwrite 0x028e93 thru 0x035030 (0xc19e bytes) 000870 1726852494.643000 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000871 1726852494.823243 MIRROR_EXTEND to 8 mirrors 000872 1726852495.192570 trunc from 0x038783 to 0x025141 (0x13641 bytes) 000873 1726852495.249712 read 0x010bdf thru 0x019a71 (0x8e93 bytes) 000874 1726852495.255936 mapread 0x008d4e thru 0x00cdcf (0x4082 bytes) 000879 1726852495.276556 MIRROR_EXTEND to 9 mirrors 000880 1726852495.561615 MIRROR_RESYNC 9 mirrors 000882 1726852496.233008 mapread 0x002aef thru 0x002b38 (0x4a bytes) 000884 1726852496.239507 write 0x03e4f5 thru 0x03ffff (0x1b0b bytes) 000885 1726852496.292510 trunc from 0x040000 to 0x0090ef (0x36f10 bytes) 000886 1726852496.304711 mapwrite 0x012d60 thru 0x02037c (0xd61d bytes) 000889 1726852496.366671 read 0x01465e thru 0x01de59 (0x97fc bytes) 000891 1726852496.367467 write 0x0219e5 thru 0x0278e4 (0x5f00 bytes) 000892 1726852496.368535 trunc from 0x0278e5 to 0x026a96 (0xe4e bytes) 000893 1726852496.380349 MIRROR_EXTEND to 10 mirrors 000894 1726852496.783814 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000895 1726852496.928899 trunc from 0x026a97 to 0x021d1e (0x4d78 bytes) 000896 1726852496.981059 trunc from 0x021d1f to 0x0222d8 (0x5ba bytes) 000897 1726852496.993924 trunc from 0x0222d9 to 0x00a35c (0x17f7c bytes) 000898 1726852497.001638 trunc from 0x00a35d to 0x00f483 (0x5127 bytes) 000899 1726852497.011562 write 0x010b0c thru 0x018bd4 (0x80c9 bytes) 000900 1726852497.012897 mapwrite 0x03eb6a thru 0x03ffff (0x1496 bytes) 000902 1726852497.083078 mapread 0x00d9cb thru 0x01920e (0xb844 bytes) 000903 1726852497.099069 read 0x015c40 thru 0x01f071 (0x9432 bytes) 000904 1726852497.099815 trunc from 0x040000 to 0x01e229 (0x21dd6 bytes) 000905 1726852497.114120 read 0x0047a0 thru 0x00f862 (0xb0c3 bytes) 000906 1726852497.122676 mapread 0x00c32e thru 0x012f95 (0x6c68 bytes) 000908 1726852497.123882 mapread 0x01c9a1 thru 0x01cf2a (0x58a bytes) 000909 1726852497.129267 MIRROR_RESYNC 9 mirrors 000911 1726852497.840446 read 0x019009 thru 0x01e229 (0x5221 bytes) 000912 1726852497.847912 trunc from 0x01e22a to 0x00c52a (0x11cff bytes) 000915 1726852497.934225 write 0x0299a0 thru 0x0300da (0x673b bytes) 000916 1726852497.939507 mapread 0x01aac9 thru 0x01e164 (0x369c bytes) 000919 1726852497.953277 trunc from 0x0300db to 0x036447 (0x636d bytes) 000920 1726852497.964357 trunc from 0x036448 to 0x0384ee (0x20a7 bytes) 000921 1726852497.974234 trunc from 0x0384ef to 0x006030 (0x324be bytes) 000922 1726852497.987349 write 0x01e2b7 thru 0x027a45 (0x978f bytes) 000925 1726852497.989003 trunc from 0x027a46 to 0x012ba1 (0x14ea4 bytes) 000927 1726852498.004971 trunc from 0x012ba2 to 0x03b47b (0x288da bytes) 000928 1726852498.015713 mapread 0x004199 thru 0x009032 (0x4e9a bytes) 000929 1726852498.026954 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000930 1726852498.137051 write 0x004239 thru 0x0047cd (0x595 bytes) 000933 1726852498.141025 write 0x03b726 thru 0x03ffff (0x48da bytes) 000936 1726852498.144979 mapread 0x0201ae thru 0x02ace7 (0xab3a bytes) 000937 1726852498.156841 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000939 1726852498.271039 MIRROR_RESYNC 7 mirrors 000940 1726852498.810849 mapread 0x011125 thru 0x0146f3 (0x35cf bytes) 000941 1726852498.823476 MIRROR_RESYNC 7 mirrors 000942 1726852498.935167 trunc from 0x040000 to 0x016fc6 (0x29039 bytes) 000947 1726852498.986337 write 0x03555a thru 0x03c803 (0x72aa bytes) 000949 1726852498.987687 write 0x014727 thru 0x015b11 (0x13eb bytes) 000952 1726852498.994940 read 0x0340bb thru 0x03a0b3 (0x5ff9 bytes) 000953 1726852498.998625 read 0x00ac6e thru 0x018819 (0xdbac bytes) 000954 1726852499.009755 MIRROR_RESYNC 7 mirrors 000955 1726852499.651467 mapwrite 0x01a3d4 thru 0x01bb73 (0x17a0 bytes) 000956 1726852499.748784 mapwrite 0x02b661 thru 0x031037 (0x59d7 bytes) 000957 1726852499.787889 mapwrite 0x0236cd thru 0x024ea4 (0x17d8 bytes) 000959 1726852499.819014 read 0x00cde6 thru 0x00e427 (0x1642 bytes) 000962 1726852499.824867 mapwrite 0x02daff thru 0x03c7f4 (0xecf6 bytes) 000963 1726852499.875716 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000965 1726852499.996294 read 0x034b85 thru 0x037b05 (0x2f81 bytes) 000967 1726852500.002016 mapread 0x003c08 thru 0x00e801 (0xabfa bytes) 000971 1726852500.025278 read 0x0005a3 thru 0x00bb63 (0xb5c1 bytes) 000972 1726852500.033482 mapwrite 0x037d33 thru 0x03ffff (0x82cd bytes) 000973 1726852500.088216 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000974 1726852500.239722 write 0x02fb3f thru 0x031df8 (0x22ba bytes) 000975 1726852500.247973 read 0x02bbd6 thru 0x03a3e8 (0xe813 bytes) 000979 1726852500.253903 mapread 0x02c4d3 thru 0x02e7c4 (0x22f2 bytes) 000981 1726852500.255023 read 0x016710 thru 0x02297c (0xc26d bytes) 000982 1726852500.268399 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000985 1726852500.411536 mapwrite 0x032e6d thru 0x03ffff (0xd193 bytes) 000987 1726852500.464687 trunc from 0x040000 to 0x022a2f (0x1d5d0 bytes) 000988 1726852500.475086 mapwrite 0x0108dc thru 0x01e40e (0xdb33 bytes) 000990 1726852500.531993 write 0x03e59f thru 0x03ffff (0x1a61 bytes) 000992 1726852500.533462 trunc from 0x040000 to 0x00c4f7 (0x33b08 bytes) 000993 1726852500.548784 mapread 0x009452 thru 0x00c4f7 (0x30a6 bytes) 000994 1726852500.553598 MIRROR_RESYNC 4 mirrors 000995 1726852500.848395 read 0x003eb6 thru 0x00c4f7 (0x8642 bytes) 000996 1726852500.854585 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) 000997 1726852501.001812 read 0x003fd6 thru 0x00c4f7 (0x8522 bytes) All operations completed A-OK! PASS 70a (86s) == sanity-flr test 200a: stress test ===================== 13:15:03 (1726852503) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4317024 .. Extending file size to 7190720 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7575456 .. Extending file size to 7407872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7580544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8107456 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8315968 .. Extending file size to 8294528 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8404928 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8935360 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9050592 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 28227 28228 28230 28232 28234 done 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:16:13 (1726852573) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4490400 .. Extending file size to 2655776 .. done Extending file size to 7419136 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7694048 .. failed Extending file size to 7968416 .. Extending file size to 5491552 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6467744 .. Extending file size to 7413024 .. Extending file size to 9015200 .. failed Extending file size to 7680352 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8388800 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8447392 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8546016 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8883680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8906464 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 30080 30081 30084 30085 30087 failed 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) 192.168.202.148@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:17:23 (1726852643) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4480 2204032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 3840 2204800 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4480 2204032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 3840 2204800 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0518681 s, 40.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.239612 s, 8.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:17:31 (1726852651) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0666467 s, 31.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 13:17:34 (1726852654) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x78:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:17:37 (1726852657) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0717286 s, 29.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:17:40 (1726852660) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0713354 s, 29.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:17:44 (1726852664) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0634195 s, 33.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37d:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:18:10 (1726852690) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0769986 s, 13.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37d:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:18:53 (1726852733) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0540373 s, 38.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:19:39 (1726852779) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg248-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg248-server: Pool lustre.test_204f created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.071364 s, 29.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg248-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:20:42 (1726852842) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:20:46 (1726852846) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:20:49 (1726852849) oleg248-server: Pool lustre.test_206 created Adding targets to pool oleg248-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg248-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg248-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg248-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:21:07 (1726852867) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026729 s, 39.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191264 s, 54.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xbb435623 Mirror 3: 0xbb435623 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:21:12 (1726852872) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:21:13 (1726852873) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:21:15 (1726852875) Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.111495 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2270000 Read this (4 bytes): BBBB 192.168.202.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg248-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:21:37 (1726852897) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0135382 s, 605 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:21:41 (1726852901) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0231307 s, 45.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x99 lmm_fid: [0x200000405:0x99:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x99 lmm_fid: [0x200000405:0x99:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611289 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-09-20 13:21:41.000000000 -0400 Modify: 2024-09-20 13:21:41.000000000 -0400 Change: 2024-09-20 13:21:42.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:21:45 (1726852905) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 951 sec ============ 13:21:46 (1726852906) === sanity-flr: start cleanup 13:21:47 (1726852907) === === sanity-flr: finish cleanup 13:21:50 (1726852910) ===