-----============= acceptance-small: sanity-flr ============----- Fri May 9 07:48:09 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg446-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:48:14 (1746791294) === oleg446-client.virtnet: executing check_config_client /mnt/lustre oleg446-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg446-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6cd4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6cd4000.idle_timeout=debug disable quota as required oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:48:22 (1746791302) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7018] == sanity-flr test 0a: lfs mirror create with -N option == 07:48:23 (1746791303) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:48:26 (1746791306) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:48:26 (1746791306) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:48:27 (1746791307) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:48:31 (1746791311) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:48:32 (1746791312) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 07:48:32 (1746791312) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6cd4000.stats=clear osc.lustre-OST0001-osc-ffff8800b6cd4000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:48:34 (1746791314) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:48:36 (1746791316) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:48:38 (1746791318) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:48:39 (1746791319) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 07:48:40 (1746791320) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:48:42 (1746791322) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:48:43 (1746791323) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:48:44 (1746791324) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:48:46 (1746791326) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:48:48 (1746791328) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.148508 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333081 s, 31.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:48:50 (1746791330) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0670305 s, 156 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:48:51 (1746791331) Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 4 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:49:17 (1746791357) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.137402 s, 68.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:49:27 (1746791367) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:49:43 (1746791383) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.703106 s, 74.6 MB/s -rw-r--r-- 1 root root 52428800 May 9 07:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.214s user 0m0.000s sys 0m0.100s Stopping /mnt/lustre-ost1 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.209s user 0m0.001s sys 0m0.100s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg446-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg446-server oleg446-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.216s user 0m0.001s sys 0m0.095s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg446-server: oleg446-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg446-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6cd4000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:50:03 (1746791403) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:50:04 (1746791404) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:50:05 (1746791405) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 07:50:06 (1746791406) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 07:50:07 (1746791407) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0937056 s, 134 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:50:10 (1746791410) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00719124 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00712873 s, 147 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:50:11 (1746791411) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:50:14 (1746791414) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000529943 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00067526 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000556008 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00045942 s, 2.2 MB/s flv=10 olv=10 PASS 36d (1s) == sanity-flr test 37: mirror I/O API verification ======= 07:50:15 (1746791415) create test files with size 15740044 .. 8dcb92dac5b448da57ba436d394df758 - b1cec6c30d09c007f527105035416eda - baa6ba21ebf6b055100701cd6e3784f3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 07:50:20 (1746791420) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:50:23 (1746791423) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:50:24 (1746791424) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207983 s, 50.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106841 s, 98.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 07:50:26 (1746791426) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0216802 s, 96.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0756461 s, 55.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0360611 s, 58.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 07:50:28 (1746791428) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 07:50:29 (1746791429) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:50:30 (1746791430) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00876087 s, 120 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 07:50:31 (1746791431) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:50:32 (1746791432) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:50:34 (1746791434) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0499953 s, 210 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:50:58 (1746791458) llite.lustre-ffff88012bdbd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0603057 s, 174 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg446-client.virtnet /mnt/lustre (opts:) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800aaf57800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:51:01 (1746791461) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:51:07 (1746791467) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:51:14 (1746791474) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:51:14 (1746791474) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0563627 s, 55.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0480374 s, 65.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: e76e76663a850ab6106077cdfc3da762 - mirror 1 checksum: 609aaf2fc048b7855946a43db709d711 - mirror 2 checksum: e76e76663a850ab6106077cdfc3da762 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 9 07:51:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 9 07:51:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e76e76663a850ab6106077cdfc3da762 - mirror 1 checksum: 609aaf2fc048b7855946a43db709d711 - mirror 2 checksum: 609aaf2fc048b7855946a43db709d711 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 9 07:51:15 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 9 07:51:15 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:51:17 (1746791477) Failing mds1 on oleg446-server Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 07:51:23 (1746791483) shut down facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server Failover mds1 to oleg446-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 07:51:36 (1746791496) targets are mounted 07:51:36 (1746791496) facet_failover done oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812014 s, 129 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:51:42 (1746791502) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:51:43 (1746791503) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:51:44 (1746791504) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00281309 s, 5.8 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:51:45 (1746791505) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:51:46 (1746791506) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:51:47 (1746791507) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:51:50 (1746791510) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:52:08 (1746791528) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:52:25 (1746791545) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:52:41 (1746791561) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 07:52:43 (1746791563) mirror create pid 22499 mirror split pid 22500 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22499 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22500 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 07:53:44 (1746791624) Seed set to 1625 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1746791625.096776 mapwrite 0x03f620 thru 0x03ffff (0x9e0 bytes) 000002 1746791625.125592 mapread 0x01fcc7 thru 0x024761 (0x4a9b bytes) 000003 1746791625.129453 read 0x00f81d thru 0x018d92 (0x9576 bytes) 000006 1746791625.132267 mapwrite 0x0321fe thru 0x033fc9 (0x1dcc bytes) truncating to largest ever: 0x28226 000007 1746791625.146767 trunc from 0x040000 to 0x028225 (0x17dda bytes) 000008 1746791625.152310 mapwrite 0x03ab22 thru 0x03ffff (0x54de bytes) 000014 1746791625.171515 mapread 0x01f37b thru 0x021704 (0x238a bytes) 000015 1746791625.172215 mapwrite 0x016edd thru 0x023aec (0xcc10 bytes) 000016 1746791625.185196 read 0x01feb2 thru 0x021366 (0x14b5 bytes) 000018 1746791625.185715 write 0x031b18 thru 0x035a64 (0x3f4d bytes) 000019 1746791625.188681 read 0x02b874 thru 0x0344a5 (0x8c32 bytes) 000020 1746791625.189034 mapwrite 0x033c21 thru 0x03b616 (0x79f6 bytes) 000022 1746791625.200560 trunc from 0x040000 to 0x00faee (0x30511 bytes) 000024 1746791625.205081 read 0x005c14 thru 0x00db19 (0x7f06 bytes) truncating to largest ever: 0x2ef9f 000032 1746791625.207413 trunc from 0x00faef to 0x02ef9e (0x1f4b0 bytes) 000033 1746791625.211161 read 0x02c6e4 thru 0x02e955 (0x2272 bytes) 000036 1746791625.214814 write 0x00ab1d thru 0x0183d4 (0xd8b8 bytes) 000040 1746791625.216600 trunc from 0x02ef9f to 0x0031c1 (0x2bddd bytes) 000043 1746791625.221449 write 0x014458 thru 0x01c1cd (0x7d76 bytes) 000044 1746791625.222294 trunc from 0x01c1ce to 0x028072 (0xbea5 bytes) truncating to largest ever: 0x30caa 000046 1746791625.226335 trunc from 0x028073 to 0x030ca9 (0x8c37 bytes) 000047 1746791625.229854 read 0x0306a3 thru 0x030ca9 (0x607 bytes) 000048 1746791625.232880 write 0x0013a8 thru 0x003c34 (0x288d bytes) 000051 1746791625.234135 mapwrite 0x0096bf thru 0x013c05 (0xa547 bytes) 000052 1746791625.248765 trunc from 0x030caa to 0x017ace (0x191db bytes) 000054 1746791625.252697 write 0x014133 thru 0x019b20 (0x59ee bytes) 000057 1746791625.253946 mapread 0x013ca3 thru 0x019b20 (0x5e7e bytes) 000058 1746791625.254620 write 0x0007e1 thru 0x00d790 (0xcfb0 bytes) 000062 1746791625.257811 MIRROR_EXTEND to 2 mirrors 000063 1746791625.351030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000064 1746791625.379340 read 0x011324 thru 0x013618 (0x22f5 bytes) 000065 1746791625.381033 write 0x01b32b thru 0x02344f (0x8125 bytes) 000068 1746791625.383680 trunc from 0x023450 to 0x01d47b (0x5fd4 bytes) 000071 1746791625.390026 write 0x02a7a6 thru 0x02c17a (0x19d5 bytes) 000073 1746791625.390643 mapread 0x026043 thru 0x02c17a (0x6138 bytes) 000075 1746791625.395409 trunc from 0x02c17b to 0x01360a (0x18b70 bytes) 000078 1746791625.399218 mapwrite 0x00d435 thru 0x012f13 (0x5adf bytes) 000079 1746791625.413409 read 0x00bac6 thru 0x013298 (0x77d3 bytes) 000080 1746791625.413712 read 0x00e20b thru 0x01360a (0x5400 bytes) 000084 1746791625.414620 read 0x00daf4 thru 0x00e2fa (0x807 bytes) 000086 1746791625.415108 read 0x00892f thru 0x00e38f (0x5a61 bytes) 000088 1746791625.415535 read 0x011ae7 thru 0x01360a (0x1b24 bytes) 000090 1746791625.415981 mapread 0x00f645 thru 0x01360a (0x3fc6 bytes) 000093 1746791625.416968 read 0x005393 thru 0x00c453 (0x70c1 bytes) 000098 1746791625.422186 mapread 0x0132fa thru 0x01360a (0x311 bytes) 000103 1746791625.423566 mapread 0x00f09a thru 0x01134f (0x22b6 bytes) 000108 1746791625.425127 trunc from 0x01360b to 0x00c170 (0x749a bytes) 000112 1746791625.429843 read 0x001a54 thru 0x00a438 (0x89e5 bytes) 000114 1746791625.430383 read 0x00a098 thru 0x00c170 (0x20d9 bytes) truncating to largest ever: 0x31b34 000115 1746791625.430660 trunc from 0x00c171 to 0x031b33 (0x259c3 bytes) 000117 1746791625.434088 read 0x012228 thru 0x020fc1 (0xed9a bytes) 000119 1746791625.439258 MIRROR_EXTEND to 2 mirrors 000120 1746791625.540733 mapwrite 0x019f67 thru 0x01d952 (0x39ec bytes) 000121 1746791625.584485 mapread 0x003c07 thru 0x0111fe (0xd5f8 bytes) 000122 1746791625.588482 mapwrite 0x03fa26 thru 0x03ffff (0x5da bytes) 000125 1746791625.611668 mapread 0x00217d thru 0x00a0a6 (0x7f2a bytes) 000126 1746791625.612176 read 0x014fc8 thru 0x01b34e (0x6387 bytes) 000127 1746791625.613667 MIRROR_RESYNC 2 mirrors 000129 1746791625.704857 write 0x0017e8 thru 0x007ae0 (0x62f9 bytes) 000130 1746791625.720916 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000131 1746791625.753108 read 0x030117 thru 0x03442b (0x4315 bytes) 000132 1746791625.755491 read 0x026c56 thru 0x028f6c (0x2317 bytes) 000134 1746791625.759436 write 0x02f77f thru 0x03ed0d (0xf58f bytes) 000135 1746791625.760238 read 0x024322 thru 0x02f659 (0xb338 bytes) 000136 1746791625.760581 trunc from 0x040000 to 0x0103f5 (0x2fc0a bytes) 000137 1746791625.767847 mapwrite 0x00b8fb thru 0x0110df (0x57e5 bytes) 000139 1746791625.786252 read 0x007c00 thru 0x00b339 (0x373a bytes) 000141 1746791625.786779 mapwrite 0x02b5d8 thru 0x03740a (0xbe33 bytes) 000145 1746791625.807178 read 0x0341ec thru 0x03740a (0x321f bytes) truncating to largest ever: 0x3d820 000147 1746791625.807984 trunc from 0x03740b to 0x03d81f (0x6415 bytes) 000148 1746791625.811386 trunc from 0x03d820 to 0x01ae19 (0x22a06 bytes) 000153 1746791625.815465 trunc from 0x01ae1a to 0x00573e (0x156db bytes) 000156 1746791625.819361 mapwrite 0x01ddf4 thru 0x0285c1 (0xa7ce bytes) 000157 1746791625.841040 read 0x00d775 thru 0x011001 (0x388d bytes) 000159 1746791625.841553 read 0x01cadd thru 0x025404 (0x8928 bytes) 000161 1746791625.842015 read 0x011f9e thru 0x01eefe (0xcf61 bytes) 000163 1746791625.842498 trunc from 0x0285c2 to 0x003078 (0x25549 bytes) 000167 1746791625.846727 trunc from 0x003079 to 0x011360 (0xe2e8 bytes) 000168 1746791625.849990 write 0x0057ad thru 0x00b74a (0x5f9e bytes) 000170 1746791625.852855 mapread 0x008b7f thru 0x0094f5 (0x977 bytes) 000177 1746791625.854560 read 0x00ffd1 thru 0x011360 (0x1390 bytes) 000181 1746791625.855789 trunc from 0x011361 to 0x000899 (0x10ac7 bytes) 000183 1746791625.859910 read 0x00023e thru 0x000899 (0x65c bytes) 000184 1746791625.860250 read 0x00017b thru 0x000899 (0x71f bytes) 000185 1746791625.860543 mapwrite 0x02b654 thru 0x03a841 (0xf1ee bytes) 000187 1746791625.881241 read 0x02e287 thru 0x02fe3f (0x1bb9 bytes) 000189 1746791625.881717 mapread 0x029fda thru 0x039875 (0xf89c bytes) 000191 1746791625.882351 trunc from 0x03a842 to 0x01cf60 (0x1d8e1 bytes) 000193 1746791625.887266 MIRROR_EXTEND to 2 mirrors 000194 1746791625.978244 trunc from 0x01cf61 to 0x020802 (0x38a2 bytes) 000195 1746791625.997224 read 0x01d7ae thru 0x020802 (0x3055 bytes) 000197 1746791625.999093 mapwrite 0x034ce6 thru 0x03ffff (0xb31a bytes) 000198 1746791626.022880 trunc from 0x040000 to 0x0078d2 (0x3872d bytes) 000202 1746791626.027313 mapread 0x0041bf thru 0x004ba0 (0x9e2 bytes) 000204 1746791626.029844 write 0x011545 thru 0x014051 (0x2b0d bytes) 000205 1746791626.030366 trunc from 0x014052 to 0x0366b7 (0x22666 bytes) 000206 1746791626.033935 mapread 0x01fa77 thru 0x02f26e (0xf7f8 bytes) 000207 1746791626.038653 read 0x0149bd thru 0x019ce2 (0x5326 bytes) 000210 1746791626.041790 read 0x026378 thru 0x0357ce (0xf457 bytes) 000211 1746791626.042298 trunc from 0x0366b8 to 0x020c90 (0x15a27 bytes) 000212 1746791626.046221 read 0x006897 thru 0x0089db (0x2145 bytes) 000214 1746791626.046833 mapread 0x0052e6 thru 0x00c74c (0x7467 bytes) 000215 1746791626.047420 mapwrite 0x03046d thru 0x03bce5 (0xb879 bytes) 000216 1746791626.069520 mapwrite 0x018b83 thru 0x024a6c (0xbeea bytes) 000217 1746791626.083587 MIRROR_RESYNC 2 mirrors 000219 1746791626.169461 mapwrite 0x00cbee thru 0x01b7ab (0xebbe bytes) 000220 1746791626.212281 mapwrite 0x0063c5 thru 0x012a85 (0xc6c1 bytes) 000221 1746791626.225375 mapwrite 0x00864f thru 0x014554 (0xbf06 bytes) 000222 1746791626.237348 trunc from 0x03bce6 to 0x0016f6 (0x3a5ef bytes) 000223 1746791626.245470 mapread 0x000002 thru 0x0016f6 (0x16f5 bytes) 000224 1746791626.249566 MIRROR_RESYNC 2 mirrors 000226 1746791626.328886 write 0x03122b thru 0x0392f2 (0x80c8 bytes) 000227 1746791626.340722 trunc from 0x0392f3 to 0x0354af (0x3e43 bytes) 000229 1746791626.349604 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000232 1746791626.379424 trunc from 0x0354b0 to 0x005116 (0x30399 bytes) 000235 1746791626.383833 read 0x00289f thru 0x005116 (0x2878 bytes) 000237 1746791626.385560 mapwrite 0x03ba10 thru 0x03ffff (0x45f0 bytes) 000238 1746791626.405119 trunc from 0x040000 to 0x01e9ad (0x21652 bytes) 000239 1746791626.408571 write 0x003fad thru 0x00469a (0x6ee bytes) 000240 1746791626.409102 trunc from 0x01e9ae to 0x0094ce (0x154df bytes) 000241 1746791626.411942 mapread 0x002c04 thru 0x0085da (0x59d7 bytes) 000242 1746791626.413972 trunc from 0x0094cf to 0x00bcf3 (0x2825 bytes) 000243 1746791626.416686 trunc from 0x00bcf4 to 0x032268 (0x26575 bytes) 000244 1746791626.419625 write 0x02905c thru 0x02ea79 (0x5a1e bytes) 000246 1746791626.422416 write 0x03d33b thru 0x03f0d2 (0x1d98 bytes) 000248 1746791626.423450 trunc from 0x03f0d3 to 0x025696 (0x19a3c bytes) 000251 1746791626.427013 read 0x01300a thru 0x018687 (0x567e bytes) 000253 1746791626.429482 mapwrite 0x00f224 thru 0x0192bc (0xa099 bytes) truncating to largest ever: 0x3fd1a 000254 1746791626.441839 trunc from 0x025697 to 0x03fd19 (0x1a683 bytes) 000255 1746791626.445302 read 0x00f1dd thru 0x016236 (0x705a bytes) 000257 1746791626.447161 MIRROR_EXTEND to 2 mirrors 000258 1746791626.539607 mapread 0x01ee13 thru 0x0218ef (0x2add bytes) 000259 1746791626.544882 trunc from 0x03fd1a to 0x02b063 (0x14cb6 bytes) 000260 1746791626.560879 mapread 0x0194cb thru 0x028927 (0xf45d bytes) 000261 1746791626.564946 write 0x009357 thru 0x011021 (0x7ccb bytes) 000262 1746791626.567167 trunc from 0x02b064 to 0x007471 (0x23bf2 bytes) 000264 1746791626.572669 mapwrite 0x01c0cb thru 0x025a45 (0x997b bytes) 000265 1746791626.592782 MIRROR_RESYNC 2 mirrors 000266 1746791626.676355 MIRROR_EXTEND to 3 mirrors 000267 1746791626.764252 read 0x001831 thru 0x00f826 (0xdff6 bytes) 000268 1746791626.766485 read 0x0033c5 thru 0x012afc (0xf738 bytes) 000269 1746791626.768982 trunc from 0x025a46 to 0x0293c5 (0x3980 bytes) 000270 1746791626.787688 write 0x0122fd thru 0x020a32 (0xe736 bytes) 000271 1746791626.791070 read 0x01a833 thru 0x02597e (0xb14c bytes) 000272 1746791626.793422 mapread 0x00dd8e thru 0x01a6fa (0xc96d bytes) 000273 1746791626.793885 trunc from 0x0293c6 to 0x002bea (0x267db bytes) 000275 1746791626.798943 trunc from 0x002beb to 0x03bd88 (0x3919e bytes) 000277 1746791626.802931 mapwrite 0x037d76 thru 0x03dba9 (0x5e34 bytes) 000279 1746791626.822765 trunc from 0x03dbaa to 0x017b64 (0x26045 bytes) 000280 1746791626.826693 trunc from 0x017b65 to 0x016437 (0x172d bytes) 000281 1746791626.830023 mapread 0x001084 thru 0x00deab (0xce28 bytes) 000286 1746791626.841598 trunc from 0x016438 to 0x0035cd (0x12e6a bytes) 000288 1746791626.846263 trunc from 0x0035ce to 0x0204a7 (0x1ceda bytes) 000293 1746791626.853460 MIRROR_EXTEND to 4 mirrors 000295 1746791626.939994 trunc from 0x0204a8 to 0x03c786 (0x1c2df bytes) 000296 1746791626.960712 mapwrite 0x036207 thru 0x0378ed (0x16e7 bytes) 000302 1746791626.977144 mapread 0x032a90 thru 0x037ea6 (0x5417 bytes) 000303 1746791626.979768 trunc from 0x03c787 to 0x016de8 (0x2599e bytes) 000306 1746791626.984659 mapread 0x00023b thru 0x00c122 (0xbee8 bytes) 000308 1746791626.989996 read 0x005fa1 thru 0x00c9b2 (0x6a12 bytes) 000309 1746791626.990290 mapread 0x007876 thru 0x00e15d (0x68e8 bytes) 000312 1746791626.991070 mapwrite 0x016136 thru 0x0177de (0x16a9 bytes) 000314 1746791627.006229 mapread 0x00376d thru 0x00fc45 (0xc4d9 bytes) 000318 1746791627.007254 write 0x0242a7 thru 0x028451 (0x41ab bytes) 000321 1746791627.008162 mapread 0x01045c thru 0x013ff4 (0x3b99 bytes) 000322 1746791627.008521 write 0x02d2d9 thru 0x0368cd (0x95f5 bytes) 000323 1746791627.009091 trunc from 0x0368ce to 0x0170cd (0x1f800 bytes) 000325 1746791627.012645 mapwrite 0x02012a thru 0x026220 (0x60f7 bytes) 000326 1746791627.029013 trunc from 0x026221 to 0x02f63c (0x941c bytes) 000327 1746791627.032315 mapread 0x026d3e thru 0x02f63c (0x88ff bytes) 000328 1746791627.035826 write 0x0071ed thru 0x007c10 (0xa24 bytes) 000329 1746791627.036421 write 0x011cce thru 0x01e874 (0xcba7 bytes) 000330 1746791627.037192 read 0x01a784 thru 0x020034 (0x58b1 bytes) 000333 1746791627.037883 read 0x011e72 thru 0x012068 (0x1f7 bytes) 000334 1746791627.038190 trunc from 0x02f63d to 0x029e34 (0x5808 bytes) 000335 1746791627.043198 MIRROR_RESYNC 4 mirrors 000337 1746791627.179229 trunc from 0x029e35 to 0x01fdd6 (0xa05e bytes) 000339 1746791627.194712 trunc from 0x01fdd7 to 0x02e228 (0xe452 bytes) 000342 1746791627.199623 mapread 0x00e6fc thru 0x01113e (0x2a43 bytes) 000344 1746791627.205548 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000345 1746791627.250692 mapread 0x020620 thru 0x02306e (0x2a4f bytes) 000347 1746791627.256566 write 0x0090d8 thru 0x013133 (0xa05c bytes) 000349 1746791627.262328 MIRROR_RESYNC 3 mirrors 000350 1746791627.366574 mapwrite 0x013b3d thru 0x014620 (0xae4 bytes) 000352 1746791627.407249 mapwrite 0x018d90 thru 0x01ea8d (0x5cfe bytes) 000353 1746791627.423140 mapread 0x01aa24 thru 0x021a65 (0x7042 bytes) 000355 1746791627.424208 trunc from 0x02e229 to 0x035078 (0x6e50 bytes) 000356 1746791627.432169 write 0x037bbe thru 0x03ffff (0x8442 bytes) 000357 1746791627.432931 mapread 0x00e995 thru 0x014727 (0x5d93 bytes) 000359 1746791627.433580 mapread 0x022b94 thru 0x02c28d (0x96fa bytes) 000361 1746791627.436667 read 0x0307c0 thru 0x0385c3 (0x7e04 bytes) 000362 1746791627.439482 MIRROR_RESYNC 3 mirrors 000363 1746791627.544711 write 0x03e63c thru 0x03f13a (0xaff bytes) 000364 1746791627.557824 write 0x0093cd thru 0x0109eb (0x761f bytes) 000365 1746791627.561365 MIRROR_RESYNC 3 mirrors 000366 1746791627.664625 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000367 1746791627.722582 mapwrite 0x02598a thru 0x030649 (0xacc0 bytes) 000368 1746791627.770126 mapwrite 0x002a02 thru 0x00a815 (0x7e14 bytes) 000369 1746791627.785269 read 0x03d8d9 thru 0x03ffff (0x2727 bytes) 000376 1746791627.789888 MIRROR_EXTEND to 3 mirrors 000381 1746791627.884755 mapwrite 0x02da10 thru 0x033a27 (0x6018 bytes) 000382 1746791627.923834 write 0x019b30 thru 0x0210a3 (0x7574 bytes) 000387 1746791627.927376 read 0x013db9 thru 0x01e682 (0xa8ca bytes) 000388 1746791627.927768 mapread 0x0231df thru 0x031575 (0xe397 bytes) 000390 1746791627.930327 write 0x01256a thru 0x0189a0 (0x6437 bytes) 000392 1746791627.931106 read 0x01613f thru 0x017517 (0x13d9 bytes) 000393 1746791627.932822 MIRROR_EXTEND to 4 mirrors 000394 1746791628.049661 write 0x03f196 thru 0x03ffff (0xe6a bytes) 000396 1746791628.064536 read 0x0197b9 thru 0x028928 (0xf170 bytes) 000399 1746791628.067547 read 0x0339c0 thru 0x0386bb (0x4cfc bytes) 000402 1746791628.071285 trunc from 0x040000 to 0x029d9c (0x16263 bytes) 000403 1746791628.076471 mapwrite 0x0041c2 thru 0x005866 (0x16a5 bytes) 000404 1746791628.087969 write 0x003fdc thru 0x0138e0 (0xf905 bytes) 000406 1746791628.089344 mapread 0x001fa5 thru 0x00b2f1 (0x934d bytes) 000407 1746791628.089941 mapwrite 0x0345bf thru 0x0388e2 (0x4324 bytes) 000410 1746791628.110604 trunc from 0x0388e3 to 0x01b551 (0x1d391 bytes) 000411 1746791628.114041 trunc from 0x01b552 to 0x00c024 (0xf52d bytes) 000414 1746791628.117841 read 0x008059 thru 0x00807d (0x25 bytes) 000415 1746791628.118146 mapwrite 0x01a7f4 thru 0x024876 (0xa083 bytes) 000417 1746791628.137573 trunc from 0x024877 to 0x000f64 (0x23912 bytes) 000420 1746791628.142628 mapread 0x000092 thru 0x000f64 (0xed3 bytes) 000425 1746791628.144793 trunc from 0x000f65 to 0x024057 (0x230f3 bytes) 000426 1746791628.148939 mapwrite 0x008834 thru 0x009a90 (0x125d bytes) 000429 1746791628.168266 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000430 1746791628.214829 MIRROR_EXTEND to 4 mirrors 000431 1746791628.301859 MIRROR_EXTEND to 5 mirrors 000434 1746791628.399114 write 0x039d6a thru 0x03ffff (0x6296 bytes) 000435 1746791628.415085 MIRROR_EXTEND to 6 mirrors 000437 1746791628.518273 mapread 0x00693f thru 0x01349c (0xcb5e bytes) 000438 1746791628.525212 trunc from 0x040000 to 0x015bfe (0x2a401 bytes) 000441 1746791628.544510 write 0x0096b0 thru 0x00c75d (0x30ae bytes) 000442 1746791628.547234 trunc from 0x015bff to 0x026ce1 (0x110e3 bytes) 000444 1746791628.551581 mapread 0x001388 thru 0x001790 (0x409 bytes) 000446 1746791628.553172 read 0x00a402 thru 0x01534b (0xaf4a bytes) 000447 1746791628.555773 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000448 1746791628.601135 mapread 0x010e91 thru 0x01649e (0x560e bytes) 000449 1746791628.606540 read 0x00cdc1 thru 0x018e58 (0xc098 bytes) 000450 1746791628.608250 write 0x03acb1 thru 0x03ffff (0x534f bytes) 000453 1746791628.609967 trunc from 0x040000 to 0x00ef3f (0x310c0 bytes) 000454 1746791628.614718 mapwrite 0x0037f7 thru 0x0118dc (0xe0e6 bytes) 000457 1746791628.636762 read 0x00bc84 thru 0x0105a0 (0x491d bytes) 000459 1746791628.637464 trunc from 0x0118dd to 0x02d5da (0x1bcfe bytes) 000460 1746791628.641131 mapread 0x012c9d thru 0x014e03 (0x2167 bytes) 000461 1746791628.644474 mapwrite 0x031c62 thru 0x03847f (0x681e bytes) 000462 1746791628.665615 trunc from 0x038480 to 0x011a87 (0x269f8 bytes) 000463 1746791628.669451 trunc from 0x011a88 to 0x01298a (0xf03 bytes) 000464 1746791628.672728 read 0x006cdd thru 0x010b49 (0x9e6d bytes) 000465 1746791628.673077 trunc from 0x01298b to 0x029331 (0x169a7 bytes) 000467 1746791628.677073 trunc from 0x029332 to 0x00ac3d (0x1e6f4 bytes) 000469 1746791628.680731 mapread 0x001ec5 thru 0x009148 (0x7284 bytes) 000471 1746791628.681591 write 0x032a09 thru 0x03d7da (0xadd2 bytes) 000472 1746791628.683645 MIRROR_RESYNC 5 mirrors 000473 1746791628.839521 trunc from 0x03d7db to 0x00f87c (0x2df5e bytes) 000474 1746791628.856866 mapread 0x00bd53 thru 0x00cec8 (0x1176 bytes) 000475 1746791628.859311 trunc from 0x00f87d to 0x004952 (0xaf2a bytes) 000476 1746791628.863491 write 0x02a2ef thru 0x033bde (0x98f0 bytes) 000477 1746791628.865260 MIRROR_RESYNC 5 mirrors 000481 1746791629.012421 trunc from 0x033bdf to 0x0329d7 (0x1207 bytes) 000482 1746791629.030569 trunc from 0x0329d8 to 0x021394 (0x11643 bytes) 000484 1746791629.035040 trunc from 0x021395 to 0x03b490 (0x1a0fc bytes) 000485 1746791629.038654 write 0x0047ef thru 0x006cf4 (0x2506 bytes) 000488 1746791629.042563 mapread 0x00a5df thru 0x014cdc (0xa6fe bytes) 000489 1746791629.046935 mapwrite 0x00ba98 thru 0x01b9fe (0xff67 bytes) 000490 1746791629.062702 read 0x015a91 thru 0x01600b (0x57b bytes) 000491 1746791629.063049 mapread 0x01afc0 thru 0x02666c (0xb6ad bytes) 000492 1746791629.066947 mapwrite 0x01fa85 thru 0x026051 (0x65cd bytes) 000493 1746791629.080570 mapread 0x003395 thru 0x00d080 (0x9cec bytes) 000494 1746791629.082618 read 0x029cd9 thru 0x02b072 (0x139a bytes) 000497 1746791629.083645 mapread 0x00c599 thru 0x012f97 (0x69ff bytes) 000498 1746791629.084265 read 0x0305a9 thru 0x035d39 (0x5791 bytes) 000500 1746791629.089038 MIRROR_EXTEND to 6 mirrors 000501 1746791629.189978 read 0x000949 thru 0x00f5fc (0xecb4 bytes) 000502 1746791629.193397 read 0x036715 thru 0x03b490 (0x4d7c bytes) 000504 1746791629.197659 trunc from 0x03b491 to 0x03a88a (0xc06 bytes) 000506 1746791629.218012 trunc from 0x03a88b to 0x03108e (0x97fc bytes) 000508 1746791629.222102 mapread 0x00af99 thru 0x010185 (0x51ed bytes) 000509 1746791629.231964 MIRROR_RESYNC 6 mirrors 000512 1746791629.404188 trunc from 0x03108f to 0x013d4a (0x1d344 bytes) 000513 1746791629.422569 trunc from 0x013d4b to 0x00a54b (0x97ff bytes) 000514 1746791629.426945 trunc from 0x00a54c to 0x00f381 (0x4e36 bytes) 000518 1746791629.431027 read 0x006ae9 thru 0x00f381 (0x8899 bytes) 000519 1746791629.433397 mapwrite 0x007be1 thru 0x0133b6 (0xb7d6 bytes) 000520 1746791629.454388 mapread 0x0099ee thru 0x0132ab (0x98be bytes) 000521 1746791629.454849 mapwrite 0x038acd thru 0x03ffff (0x7533 bytes) 000523 1746791629.474391 write 0x0317c5 thru 0x033562 (0x1d9e bytes) 000525 1746791629.476955 MIRROR_RESYNC 6 mirrors 000526 1746791629.646264 read 0x028d18 thru 0x0308ab (0x7b94 bytes) 000528 1746791629.648515 read 0x00455c thru 0x011332 (0xcdd7 bytes) 000530 1746791629.651529 read 0x0325c7 thru 0x03b7ee (0x9228 bytes) 000534 1746791629.652060 mapread 0x019c83 thru 0x01aa7f (0xdfd bytes) 000536 1746791629.652543 mapwrite 0x02c02c thru 0x0316a0 (0x5675 bytes) 000537 1746791629.693127 mapwrite 0x00a01a thru 0x00e2d6 (0x42bd bytes) 000539 1746791629.704827 mapwrite 0x027248 thru 0x0303f3 (0x91ac bytes) 000541 1746791629.717757 MIRROR_RESYNC 6 mirrors 000542 1746791629.894406 read 0x0061f7 thru 0x015257 (0xf061 bytes) 000545 1746791629.896852 mapwrite 0x024bc4 thru 0x02baf0 (0x6f2d bytes) 000546 1746791629.940410 trunc from 0x040000 to 0x02575f (0x1a8a0 bytes) 000547 1746791629.950628 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000548 1746791629.998157 trunc from 0x025760 to 0x0021c8 (0x23597 bytes) 000549 1746791630.003065 MIRROR_EXTEND to 6 mirrors 000550 1746791630.106200 mapwrite 0x032e3d thru 0x037632 (0x47f6 bytes) 000551 1746791630.144512 mapwrite 0x008964 thru 0x012372 (0x9a0f bytes) 000552 1746791630.162797 write 0x030f20 thru 0x03469a (0x377b bytes) 000554 1746791630.163511 mapread 0x035d2c thru 0x037632 (0x1907 bytes) 000556 1746791630.164181 mapread 0x024467 thru 0x026966 (0x2500 bytes) 000557 1746791630.166732 write 0x01a983 thru 0x02a531 (0xfbaf bytes) 000558 1746791630.170442 MIRROR_RESYNC 6 mirrors 000560 1746791630.349439 mapwrite 0x004962 thru 0x0064ae (0x1b4d bytes) 000561 1746791630.390086 trunc from 0x037633 to 0x02c11a (0xb518 bytes) 000563 1746791630.397357 trunc from 0x02c11b to 0x005657 (0x26ac3 bytes) 000564 1746791630.400871 write 0x003f47 thru 0x00d2e4 (0x939e bytes) 000565 1746791630.401576 trunc from 0x00d2e5 to 0x015b51 (0x886d bytes) 000566 1746791630.404744 trunc from 0x015b52 to 0x028b58 (0x13007 bytes) 000567 1746791630.408700 write 0x032b55 thru 0x03ffff (0xd4ab bytes) 000568 1746791630.410705 MIRROR_EXTEND to 7 mirrors 000569 1746791630.524148 trunc from 0x040000 to 0x0171e1 (0x28e1e bytes) 000571 1746791630.548569 write 0x0235a9 thru 0x02fb94 (0xc5ec bytes) 000574 1746791630.550498 read 0x006f7b thru 0x0083f6 (0x147c bytes) 000575 1746791630.552756 mapwrite 0x000f92 thru 0x005aed (0x4b5c bytes) 000578 1746791630.573785 write 0x006d9c thru 0x01399e (0xcc03 bytes) 000579 1746791630.576166 read 0x024373 thru 0x027388 (0x3016 bytes) 000580 1746791630.576468 read 0x01cc86 thru 0x01f474 (0x27ef bytes) 000581 1746791630.578988 trunc from 0x02fb95 to 0x0337c4 (0x3c30 bytes) 000582 1746791630.583886 read 0x013a5a thru 0x01f94a (0xbef1 bytes) 000583 1746791630.584414 write 0x036e48 thru 0x03f2c7 (0x8480 bytes) 000585 1746791630.586619 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000586 1746791630.639018 mapwrite 0x00096d thru 0x00f158 (0xe7ec bytes) 000587 1746791630.661198 mapwrite 0x007b6e thru 0x011853 (0x9ce6 bytes) 000588 1746791630.676100 mapwrite 0x016338 thru 0x01f9db (0x96a4 bytes) 000592 1746791630.691071 write 0x003bab thru 0x005381 (0x17d7 bytes) 000595 1746791630.692226 trunc from 0x03f2c8 to 0x028b57 (0x16770 bytes) 000597 1746791630.696542 write 0x01eda8 thru 0x025a66 (0x6cbf bytes) 000598 1746791630.698341 mapwrite 0x00db0e thru 0x017c5e (0xa151 bytes) 000599 1746791630.713066 mapread 0x0211fa thru 0x028b57 (0x795e bytes) 000600 1746791630.715194 write 0x002d32 thru 0x003712 (0x9e1 bytes) 000602 1746791630.716042 trunc from 0x028b58 to 0x02687b (0x22dc bytes) 000603 1746791630.720052 mapread 0x00d2d4 thru 0x015ce8 (0x8a15 bytes) 000604 1746791630.720619 mapread 0x0253c9 thru 0x0256d1 (0x309 bytes) 000605 1746791630.721072 write 0x009f79 thru 0x0102a3 (0x632b bytes) 000606 1746791630.721735 mapread 0x0011ea thru 0x006129 (0x4f40 bytes) 000609 1746791630.722949 trunc from 0x02687c to 0x003985 (0x22ef6 bytes) 000610 1746791630.726770 write 0x00a2e2 thru 0x0120a5 (0x7dc4 bytes) 000611 1746791630.727702 trunc from 0x0120a6 to 0x012c98 (0xbf3 bytes) 000612 1746791630.731205 read 0x003798 thru 0x00e0a5 (0xa90e bytes) 000614 1746791630.733551 mapwrite 0x009434 thru 0x013f82 (0xab4f bytes) 000615 1746791630.754551 mapread 0x004ba1 thru 0x006bf9 (0x2059 bytes) 000616 1746791630.755184 trunc from 0x013f83 to 0x025d2d (0x11dab bytes) 000617 1746791630.758923 trunc from 0x025d2e to 0x027c1e (0x1ef1 bytes) 000618 1746791630.763051 write 0x013b23 thru 0x016f97 (0x3475 bytes) 000621 1746791630.765661 mapread 0x0011b1 thru 0x006175 (0x4fc5 bytes) 000622 1746791630.766398 mapread 0x01de78 thru 0x027c1e (0x9da7 bytes) 000623 1746791630.770670 write 0x033c44 thru 0x03840d (0x47ca bytes) 000625 1746791630.772890 MIRROR_RESYNC 6 mirrors 000626 1746791630.949381 MIRROR_EXTEND to 7 mirrors 000627 1746791631.068931 read 0x00ada4 thru 0x014309 (0x9566 bytes) 000633 1746791631.072560 read 0x029b6e thru 0x031a34 (0x7ec7 bytes) 000634 1746791631.075628 mapwrite 0x036f1b thru 0x038b89 (0x1c6f bytes) 000636 1746791631.108185 write 0x01bdb6 thru 0x0297ba (0xda05 bytes) 000640 1746791631.111943 read 0x038890 thru 0x038b89 (0x2fa bytes) 000641 1746791631.112269 read 0x01d224 thru 0x0208f0 (0x36cd bytes) 000643 1746791631.112769 mapread 0x0063dd thru 0x00642a (0x4e bytes) 000644 1746791631.118053 MIRROR_EXTEND to 8 mirrors 000645 1746791631.229142 mapread 0x0255e3 thru 0x0256ff (0x11d bytes) 000647 1746791631.233267 mapread 0x02d114 thru 0x038b89 (0xba76 bytes) 000648 1746791631.240507 write 0x00b729 thru 0x00f198 (0x3a70 bytes) 000649 1746791631.260740 read 0x03378e thru 0x038b89 (0x53fc bytes) 000653 1746791631.264273 mapread 0x00a754 thru 0x00f4e0 (0x4d8d bytes) 000654 1746791631.264969 trunc from 0x038b8a to 0x01ccea (0x1be9f bytes) 000656 1746791631.273657 read 0x01b3ee thru 0x01ccea (0x18fd bytes) 000657 1746791631.276229 mapwrite 0x016083 thru 0x01dcec (0x7c6a bytes) 000658 1746791631.297681 read 0x00231f thru 0x011931 (0xf613 bytes) 000659 1746791631.298096 write 0x00239f thru 0x002742 (0x3a4 bytes) 000661 1746791631.298692 write 0x00c2d5 thru 0x00d6b6 (0x13e2 bytes) 000663 1746791631.300572 MIRROR_RESYNC 8 mirrors 000666 1746791631.494569 read 0x017e3a thru 0x019a5c (0x1c23 bytes) 000667 1746791631.496147 mapwrite 0x037d05 thru 0x03e63d (0x6939 bytes) 000668 1746791631.534663 mapread 0x00c907 thru 0x00e23e (0x1938 bytes) 000670 1746791631.539028 MIRROR_EXTEND to 9 mirrors 000672 1746791631.630499 trunc from 0x03e63e to 0x02f944 (0xecf9 bytes) 000673 1746791631.653583 mapwrite 0x01ba22 thru 0x02af66 (0xf545 bytes) 000675 1746791631.676429 trunc from 0x02f945 to 0x02f965 (0x21 bytes) 000677 1746791631.681268 trunc from 0x02f966 to 0x03b4e7 (0xbb82 bytes) 000679 1746791631.685523 mapwrite 0x035882 thru 0x039ed4 (0x4653 bytes) 000680 1746791631.702762 read 0x002595 thru 0x010bd5 (0xe641 bytes) 000681 1746791631.705475 read 0x01afea thru 0x01c5fb (0x1612 bytes) 000682 1746791631.705747 read 0x011f6b thru 0x0150c6 (0x315c bytes) 000685 1746791631.708530 write 0x01ebbc thru 0x02e6eb (0xfb30 bytes) 000686 1746791631.709365 trunc from 0x03b4e8 to 0x03e8c1 (0x33da bytes) 000687 1746791631.714327 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000690 1746791631.764125 read 0x002c55 thru 0x011aab (0xee57 bytes) 000692 1746791631.767657 write 0x032564 thru 0x03c5d5 (0xa072 bytes) 000693 1746791631.771003 write 0x0026cd thru 0x00bc1e (0x9552 bytes) 000695 1746791631.772315 read 0x02a088 thru 0x031415 (0x738e bytes) 000696 1746791631.778002 MIRROR_RESYNC 8 mirrors 000697 1746791631.994687 trunc from 0x03e8c2 to 0x032529 (0xc398 bytes) 000698 1746791632.014763 mapread 0x00b4e7 thru 0x016fb0 (0xbaca bytes) 000699 1746791632.021674 mapread 0x01bc64 thru 0x02551a (0x98b7 bytes) 000700 1746791632.025886 write 0x011a95 thru 0x01587e (0x3dea bytes) 000701 1746791632.027636 MIRROR_EXTEND to 9 mirrors 000702 1746791632.133056 trunc from 0x03252a to 0x00b8fb (0x26c2e bytes) 000704 1746791632.154081 write 0x028277 thru 0x0377e6 (0xf570 bytes) 000706 1746791632.155483 write 0x003c48 thru 0x00b55e (0x7917 bytes) 000707 1746791632.158171 read 0x02285f thru 0x031a97 (0xf239 bytes) 000708 1746791632.160213 mapread 0x0124cf thru 0x01caf7 (0xa629 bytes) 000709 1746791632.164604 mapwrite 0x031f96 thru 0x03deca (0xbf35 bytes) 000711 1746791632.185613 mapread 0x016e25 thru 0x0219cd (0xaba9 bytes) 000712 1746791632.186119 mapwrite 0x01089c thru 0x020635 (0xfd9a bytes) 000715 1746791632.200435 read 0x02035d thru 0x02c02b (0xbccf bytes) 000716 1746791632.200959 trunc from 0x03decb to 0x0104c9 (0x2da01 bytes) 000717 1746791632.205201 trunc from 0x0104ca to 0x0360e2 (0x25c19 bytes) 000719 1746791632.208803 write 0x03f6a5 thru 0x03ffff (0x95b bytes) 000720 1746791632.209295 trunc from 0x040000 to 0x004ad5 (0x3b52a bytes) 000721 1746791632.212479 mapwrite 0x0149b5 thru 0x015429 (0xa75 bytes) 000722 1746791632.229450 mapread 0x005f47 thru 0x014862 (0xe91c bytes) 000725 1746791632.230471 mapread 0x0129e4 thru 0x015429 (0x2a46 bytes) 000728 1746791632.231295 mapwrite 0x0070fc thru 0x011cd1 (0xabd6 bytes) 000729 1746791632.242953 read 0x00e09d thru 0x01220e (0x4172 bytes) 000730 1746791632.243420 mapread 0x013514 thru 0x015429 (0x1f16 bytes) 000733 1746791632.244763 mapwrite 0x0150ff thru 0x020f41 (0xbe43 bytes) 000734 1746791632.264661 write 0x00d2e8 thru 0x018b08 (0xb821 bytes) 000736 1746791632.265987 trunc from 0x020f42 to 0x012269 (0xecd8 bytes) 000737 1746791632.269632 write 0x034d04 thru 0x03edca (0xa0c7 bytes) 000738 1746791632.270371 trunc from 0x03edcb to 0x01758a (0x27840 bytes) 000742 1746791632.274486 read 0x007769 thru 0x01192e (0xa1c6 bytes) 000745 1746791632.275298 mapwrite 0x02eee5 thru 0x03d36e (0xe48a bytes) 000746 1746791632.295523 mapwrite 0x03b55d thru 0x03ee59 (0x38fd bytes) 000747 1746791632.309979 trunc from 0x03ee5a to 0x000cc6 (0x3e193 bytes) 000753 1746791632.314567 mapwrite 0x01572c thru 0x01ce78 (0x774d bytes) 000754 1746791632.339359 mapwrite 0x02e44a thru 0x0325a5 (0x415c bytes) 000756 1746791632.359823 mapwrite 0x00c3a3 thru 0x016325 (0x9f83 bytes) 000759 1746791632.382691 write 0x038d56 thru 0x039f2d (0x11d8 bytes) 000760 1746791632.385016 MIRROR_RESYNC 9 mirrors 000761 1746791632.625197 mapread 0x00d83c thru 0x010bd9 (0x339e bytes) 000763 1746791632.629643 write 0x003a5a thru 0x003acb (0x72 bytes) 000767 1746791632.645535 read 0x039243 thru 0x039f2d (0xceb bytes) 000770 1746791632.647417 write 0x02bd84 thru 0x031177 (0x53f4 bytes) 000772 1746791632.650150 read 0x01359d thru 0x01c259 (0x8cbd bytes) 000773 1746791632.652775 mapread 0x033095 thru 0x03772c (0x4698 bytes) 000774 1746791632.653398 write 0x03a70f thru 0x03ffff (0x58f1 bytes) 000775 1746791632.655293 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000779 1746791632.710398 MIRROR_EXTEND to 9 mirrors 000780 1746791632.806986 write 0x019ce1 thru 0x020aee (0x6e0e bytes) 000781 1746791632.823360 MIRROR_RESYNC 9 mirrors 000784 1746791633.048341 mapread 0x0162fe thru 0x017e2f (0x1b32 bytes) 000786 1746791633.051242 trunc from 0x040000 to 0x009490 (0x36b6f bytes) 000789 1746791633.071777 mapwrite 0x031cdd thru 0x03cb18 (0xae3c bytes) 000791 1746791633.096073 trunc from 0x03cb19 to 0x00f5fe (0x2d51a bytes) 000792 1746791633.101210 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000793 1746791633.149958 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000795 1746791633.196393 write 0x000095 thru 0x0004c7 (0x433 bytes) 000798 1746791633.198830 trunc from 0x00f5ff to 0x02e77e (0x1f180 bytes) 000799 1746791633.203153 mapwrite 0x006964 thru 0x011189 (0xa826 bytes) 000801 1746791633.224610 write 0x03147c thru 0x038b4e (0x76d3 bytes) 000803 1746791633.225639 mapwrite 0x03a8ce thru 0x03ffff (0x5732 bytes) 000804 1746791633.245301 trunc from 0x040000 to 0x009bc9 (0x36436 bytes) 000807 1746791633.249888 mapread 0x003a4d thru 0x0091f8 (0x57ac bytes) 000808 1746791633.250583 trunc from 0x009bca to 0x00af55 (0x138c bytes) 000812 1746791633.254756 read 0x001daf thru 0x00af55 (0x91a7 bytes) 000814 1746791633.256725 write 0x0389b8 thru 0x03d678 (0x4cc1 bytes) 000816 1746791633.257544 trunc from 0x03d679 to 0x01d1c9 (0x204af bytes) 000819 1746791633.261044 trunc from 0x01d1ca to 0x033a31 (0x16868 bytes) 000821 1746791633.264281 mapread 0x01017d thru 0x01c799 (0xc61d bytes) 000822 1746791633.268504 mapwrite 0x01283c thru 0x0214bc (0xec81 bytes) 000824 1746791633.285023 MIRROR_RESYNC 7 mirrors 000825 1746791633.468520 write 0x00428c thru 0x006273 (0x1fe8 bytes) 000827 1746791633.483530 trunc from 0x033a32 to 0x004370 (0x2f6c1 bytes) 000828 1746791633.491242 write 0x009cac thru 0x00c842 (0x2b97 bytes) 000830 1746791633.492254 mapwrite 0x029c06 thru 0x02be15 (0x2210 bytes) 000831 1746791633.512342 mapwrite 0x0201af thru 0x0270b4 (0x6f06 bytes) 000832 1746791633.524779 mapread 0x01dc0b thru 0x0235bd (0x59b3 bytes) 000833 1746791633.525483 mapwrite 0x01390a thru 0x0168d8 (0x2fcf bytes) 000834 1746791633.539557 MIRROR_RESYNC 7 mirrors 000835 1746791633.716877 mapwrite 0x02e5a5 thru 0x03e40e (0xfe6a bytes) 000836 1746791633.756292 MIRROR_EXTEND to 8 mirrors 000838 1746791633.855183 write 0x00edc7 thru 0x010eb2 (0x20ec bytes) 000839 1746791633.870541 write 0x03f915 thru 0x03ffff (0x6eb bytes) 000846 1746791633.872191 trunc from 0x040000 to 0x01d7ff (0x22800 bytes) 000847 1746791633.879715 read 0x017c71 thru 0x0186a5 (0xa35 bytes) 000848 1746791633.881157 read 0x0179e4 thru 0x01946c (0x1a89 bytes) 000849 1746791633.883572 trunc from 0x01d800 to 0x020b99 (0x339a bytes) 000851 1746791633.887806 MIRROR_EXTEND to 9 mirrors 000852 1746791633.982724 read 0x003c2b thru 0x00ef68 (0xb33e bytes) 000856 1746791633.985503 mapread 0x00315a thru 0x00e981 (0xb828 bytes) 000857 1746791633.985938 trunc from 0x020b9a to 0x01f205 (0x1994 bytes) 000858 1746791634.005535 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000860 1746791634.053179 mapwrite 0x02c625 thru 0x03456e (0x7f4a bytes) 000863 1746791634.076095 trunc from 0x03456f to 0x03dceb (0x977d bytes) 000864 1746791634.079364 read 0x034c9c thru 0x03622c (0x1591 bytes) 000870 1746791634.082022 mapread 0x018d85 thru 0x027997 (0xec13 bytes) 000873 1746791634.086725 mapread 0x01a7dc thru 0x01c3c0 (0x1be5 bytes) 000875 1746791634.087378 write 0x02216d thru 0x02cabe (0xa952 bytes) 000877 1746791634.089366 MIRROR_EXTEND to 9 mirrors 000878 1746791634.198813 write 0x025ee1 thru 0x0357c3 (0xf8e3 bytes) 000884 1746791634.219366 MIRROR_RESYNC 9 mirrors 000885 1746791634.455366 write 0x0200af thru 0x029e4a (0x9d9c bytes) 000887 1746791634.473588 trunc from 0x03dcec to 0x01eebf (0x1ee2c bytes) 000888 1746791634.483567 MIRROR_RESYNC 9 mirrors 000889 1746791634.693786 read 0x01bc51 thru 0x01eebf (0x326f bytes) 000891 1746791634.695588 mapwrite 0x025c77 thru 0x0307d2 (0xab5c bytes) 000892 1746791634.734692 mapread 0x01b6b6 thru 0x01ef98 (0x38e3 bytes) 000893 1746791634.737061 read 0x01fab2 thru 0x02dfa6 (0xe4f5 bytes) 000895 1746791634.737601 mapread 0x002731 thru 0x0044bc (0x1d8c bytes) 000898 1746791634.740606 trunc from 0x0307d3 to 0x02b43d (0x5395 bytes) 000900 1746791634.745287 write 0x000102 thru 0x002b17 (0x2a16 bytes) 000901 1746791634.745959 read 0x026139 thru 0x029974 (0x383c bytes) 000902 1746791634.746274 read 0x02872d thru 0x02b43d (0x2d11 bytes) 000903 1746791634.746635 mapwrite 0x033dbc thru 0x03d957 (0x9b9c bytes) 000904 1746791634.768867 mapread 0x02299f thru 0x02959a (0x6bfc bytes) 000905 1746791634.770669 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000907 1746791634.826176 read 0x0220ac thru 0x025dd0 (0x3d25 bytes) 000908 1746791634.828208 trunc from 0x03d958 to 0x02d610 (0x10347 bytes) 000910 1746791634.832135 read 0x01e16e thru 0x02cd8e (0xec21 bytes) 000911 1746791634.835276 mapwrite 0x017012 thru 0x021181 (0xa170 bytes) 000912 1746791634.848355 read 0x01d817 thru 0x024d8c (0x7576 bytes) 000913 1746791634.848716 mapwrite 0x00f91c thru 0x01d420 (0xdb05 bytes) 000915 1746791634.862920 mapread 0x009c92 thru 0x018b7a (0xeee9 bytes) 000916 1746791634.865231 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa80:0x0]: Invalid argument (22) 000918 1746791634.917028 read 0x0167e3 thru 0x016e94 (0x6b2 bytes) 000919 1746791634.918669 mapwrite 0x01ef0e thru 0x02b101 (0xc1f4 bytes) 000920 1746791634.937361 trunc from 0x02d611 to 0x008c98 (0x24978 bytes) 000921 1746791634.941108 write 0x0368c4 thru 0x0373c9 (0xb06 bytes) 000922 1746791634.942645 MIRROR_RESYNC 7 mirrors 000923 1746791635.129962 write 0x00ee11 thru 0x0159ab (0x6b9b bytes) 000924 1746791635.146532 MIRROR_RESYNC 7 mirrors 000925 1746791635.323125 mapread 0x002622 thru 0x007622 (0x5001 bytes) 000926 1746791635.326952 read 0x031718 thru 0x031b5e (0x447 bytes) 000930 1746791635.329509 MIRROR_RESYNC 7 mirrors 000933 1746791635.378590 write 0x03d881 thru 0x03ffff (0x277f bytes) 000935 1746791635.392773 trunc from 0x040000 to 0x0009af (0x3f650 bytes) 000936 1746791635.397114 read 0x000725 thru 0x0009af (0x28b bytes) 000937 1746791635.398531 trunc from 0x0009b0 to 0x01d89a (0x1ceeb bytes) 000940 1746791635.403124 MIRROR_EXTEND to 8 mirrors 000941 1746791635.488052 mapread 0x00ce0d thru 0x00ef5b (0x214f bytes) 000943 1746791635.495515 mapwrite 0x034e3e thru 0x03ffff (0xb1c2 bytes) 000944 1746791635.531012 write 0x00fe15 thru 0x01dc90 (0xde7c bytes) 000945 1746791635.534987 mapwrite 0x006379 thru 0x00b7cd (0x5455 bytes) 000948 1746791635.549072 mapread 0x02b4f8 thru 0x039da8 (0xe8b1 bytes) 000949 1746791635.552961 MIRROR_EXTEND to 9 mirrors 000950 1746791635.650546 mapread 0x039b5c thru 0x03e5d6 (0x4a7b bytes) 000951 1746791635.655419 trunc from 0x040000 to 0x01f23c (0x20dc3 bytes) 000952 1746791635.674716 trunc from 0x01f23d to 0x0100e8 (0xf154 bytes) 000953 1746791635.679196 mapread 0x00ead0 thru 0x0100e8 (0x1619 bytes) 000954 1746791635.681508 trunc from 0x0100e9 to 0x01834c (0x8264 bytes) 000955 1746791635.685644 write 0x007147 thru 0x00b28e (0x4148 bytes) 000956 1746791635.687667 mapwrite 0x03f073 thru 0x03ffff (0xf8d bytes) 000957 1746791635.704566 mapread 0x00ae37 thru 0x00be42 (0x100c bytes) 000961 1746791635.706042 write 0x035509 thru 0x039274 (0x3d6c bytes) 000965 1746791635.709017 read 0x03f54d thru 0x03ffff (0xab3 bytes) 000966 1746791635.709286 read 0x023056 thru 0x0286de (0x5689 bytes) 000969 1746791635.712482 mapwrite 0x03d4c9 thru 0x03ffff (0x2b37 bytes) 000970 1746791635.723405 write 0x0385e7 thru 0x03ffff (0x7a19 bytes) 000971 1746791635.723979 read 0x010946 thru 0x0169eb (0x60a6 bytes) 000974 1746791635.724639 mapread 0x01f99e thru 0x02cf53 (0xd5b6 bytes) 000976 1746791635.725272 mapread 0x03db6f thru 0x03ec3d (0x10cf bytes) 000977 1746791635.725605 write 0x03f273 thru 0x03ffff (0xd8d bytes) 000979 1746791635.726065 read 0x03739a thru 0x03ffff (0x8c66 bytes) 000984 1746791635.727092 mapwrite 0x008cb1 thru 0x00f09e (0x63ee bytes) 000986 1746791635.739470 write 0x00a6cb thru 0x013731 (0x9067 bytes) 000987 1746791635.740417 mapread 0x03d663 thru 0x03ffff (0x299d bytes) 000990 1746791635.741759 trunc from 0x040000 to 0x00dbb6 (0x32449 bytes) 000991 1746791635.745921 trunc from 0x00dbb7 to 0x02cce3 (0x1f12d bytes) 000992 1746791635.749207 trunc from 0x02cce4 to 0x006d79 (0x25f6a bytes) 000997 1746791635.753612 trunc from 0x006d7a to 0x022b37 (0x1bdbe bytes) 000998 1746791635.756965 mapread 0x01934d thru 0x022b37 (0x97eb bytes) 000999 1746791635.760424 mapwrite 0x00cbc1 thru 0x017347 (0xa787 bytes) 001000 1746791635.773213 trunc from 0x022b38 to 0x0093ea (0x1974d bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 71: check mirror extend parent fid ==== 07:53:56 (1746791636) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 07:53:57 (1746791637) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3307520 .. Extending file size to 7288288 .. failed Extending file size to 4850208 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8528256 .. Extending file size to 7186368 .. Extending file size to 9292096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8356032 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8800736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9310048 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9354048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 29767 29768 29769 29771 29774 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) 192.168.204.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:55:00 (1746791700) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6224896 .. Extending file size to 6537312 .. done Extending file size to 8051840 .. Extending file size to 6950336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7278112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7617952 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8099168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8530336 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8938304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8616960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8773888 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9282560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9026912 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9107232 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 1849 1850 1852 1854 1855 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) 192.168.204.146@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:56:03 (1746791763) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0297713 s, 70.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0712162 s, 29.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:56:08 (1746791768) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0203855 s, 103 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 07:56:09 (1746791769) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xab6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd48:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd48:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd49:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xd4b:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:56:11 (1746791771) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0239906 s, 87.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:56:12 (1746791772) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0267582 s, 78.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:56:14 (1746791774) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0220465 s, 95.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd50:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd50:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd51:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:56:37 (1746791797) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168809 s, 62.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd4f:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:57:21 (1746791841) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0253373 s, 82.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd51:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd51:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:58:06 (1746791886) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg446-server mds-ost sync done. Creating new pool test_204f oleg446-server: Pool lustre.test_204f created Adding targets to pool oleg446-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg446-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0214634 s, 97.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg446-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg446-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg446-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:59:02 (1746791942) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:59:04 (1746791944) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:59:05 (1746791945) oleg446-server: Pool lustre.test_206 created Adding targets to pool oleg446-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg446-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg446-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg446-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg446-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:59:19 (1746791959) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179615 s, 58.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00858992 s, 122 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x33520368 Mirror 3: 0x33520368 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:59:21 (1746791961) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:59:21 (1746791961) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:59:22 (1746791962) Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0151088 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1230000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1630000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1330000 Read this (4 bytes): BBBB 192.168.204.146@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg446-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:59:35 (1746791975) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00234937 s, 3.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:59:37 (1746791977) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00820814 s, 128 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xadb lmm_fid: [0x200000404:0xadb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xadb lmm_fid: [0x200000404:0xadb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd60:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836699 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-09 07:59:37.000000000 -0400 Modify: 2025-05-09 07:59:37.000000000 -0400 Change: 2025-05-09 07:59:38.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:59:39 (1746791979) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:59:39 (1746791979) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0831283 s, 493 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0118539 s, 346 kB/s size after second write -rw-r--r-- 1 root root 4096 May 9 07:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xadd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 9 07:59 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 691 sec ============ 07:59:41 (1746791981) === sanity-flr: start cleanup 07:59:41 (1746791981) === === sanity-flr: finish cleanup 07:59:43 (1746791983) ===