-----============= acceptance-small: sanity-flr ============----- Thu Oct 23 12:42:45 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg332-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:43:02 (1761237782) === oleg332-client.virtnet: executing check_config_client /mnt/lustre oleg332-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg332-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92340b083800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92340b083800.idle_timeout=debug disable quota as required oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:43:37 (1761237817) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 12:43:39 (1761237819) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:43:51 (1761237831) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:43:55 (1761237835) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:43:58 (1761237838) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:44:22 (1761237862) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:44:25 (1761237865) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 12:44:27 (1761237867) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92340b083800.stats=clear osc.lustre-OST0001-osc-ffff92340b083800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:44:32 (1761237872) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:44:40 (1761237880) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:44:46 (1761237886) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:44:49 (1761237889) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 12:44:56 (1761237896) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:45:01 (1761237901) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:45:07 (1761237907) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:45:15 (1761237915) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:45:25 (1761237925) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:45:31 (1761237931) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.873905 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565029 s, 18.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:45:40 (1761237940) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.168954 s, 37.2 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:45:46 (1761237946) Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (55s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:46:42 (1761238002) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.341257 s, 18.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:47:17 (1761238037) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (63s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:48:20 (1761238100) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.18681 s, 24.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 23 12:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.762s user 0m0.000s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.054s user 0m0.003s sys 0m0.184s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg332-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg332-server oleg332-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.082s user 0m0.001s sys 0m0.158s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg332-server: oleg332-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg332-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg332-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92340b083800.ost_server_uuid in FULL state after 0 sec PASS 33b (78s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:49:38 (1761238178) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:49:42 (1761238182) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:49:45 (1761238185) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:49:48 (1761238188) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:49:55 (1761238195) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.594008 s, 21.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:50:06 (1761238206) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225656 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227725 s, 46.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:50:13 (1761238213) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:50:20 (1761238220) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128782 s, 795 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000957673 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000837706 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000909169 s, 1.1 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:50:28 (1761238228) create test files with size 15730889 .. 45cf9ee6ee29cba9bfdc4ec3ace6ae73 - 16b17571a2b1dd54f943ede10b3b3608 - 215993c0f8d5176de19924c7faef23d7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre PASS 37 (16s) == sanity-flr test 38: resync ============================ 12:50:45 (1761238245) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:50:56 (1761238256) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:51:01 (1761238261) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500031 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180142 s, 58.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 12:51:09 (1761238269) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0763129 s, 27.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.220481 s, 19.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140838 s, 14.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 12:51:19 (1761238279) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:51:21 (1761238281) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:51:24 (1761238284) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203006 s, 51.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 12:51:31 (1761238291) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223989 s, 14.0 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:51:39 (1761238299) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x41:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:51:46 (1761238306) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.24399 s, 43.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:51:55 (1761238315) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9234087de800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.188436 s, 55.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg332-client.virtnet /mnt/lustre (opts:) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff92340abdc800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:52:07 (1761238327) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:52:41 (1761238361) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:53:11 (1761238391) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:53:14 (1761238394) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252363 s, 12.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.156744 s, 20.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 4226b4c337cb29d4d058291f5ec3736a - mirror 1 checksum: 4157cddca65551eda7443fa5a0a58d3e - mirror 2 checksum: 4226b4c337cb29d4d058291f5ec3736a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:53:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Oct 23 12:53:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4226b4c337cb29d4d058291f5ec3736a - mirror 1 checksum: 4157cddca65551eda7443fa5a0a58d3e - mirror 2 checksum: 4157cddca65551eda7443fa5a0a58d3e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:53:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Oct 23 12:53:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:53:23 (1761238403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402204 s, 26.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 308224.. 309247: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:53:30 (1761238410) Failing mds1 on oleg332-server Stopping /mnt/lustre-mds1 (opts:) on oleg332-server 12:53:41 (1761238421) shut down facet: mds1 facet_host: oleg332-server facet_failover_host: oleg332-server Failover mds1 to oleg332-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 12:54:03 (1761238443) targets are mounted 12:54:03 (1761238443) facet_failover done oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182532 s, 57.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:54:16 (1761238456) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0287501 s, 712 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0107811 s, 1.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:54:24 (1761238464) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0346505 s, 591 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00995687 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:54:30 (1761238470) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0168623 s, 972 kB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:54:37 (1761238477) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0563392 s, 364 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:54:45 (1761238485) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0170562 s, 240 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0646547 s, 63.4 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:54:52 (1761238492) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:55:14 (1761238514) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:55:40 (1761238540) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:56:02 (1761238562) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:56:26 (1761238586) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 12:56:35 (1761238595) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55912 mirror split pid 55914 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7639: 55914 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 55912 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:57:41 (1761238661) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8663 skipping zero size read 000003 1761238663.982689 mapwrite 0x0124d7 thru 0x021764 (0xf28e bytes) 000005 1761238664.070730 punch from 0x00adc4 to 0x00bd62 (0xf9f bytes) 000006 1761238664.076216 mapread 0x014396 thru 0x01b128 (0x6d93 bytes) fallocating to largest ever: 0x2d2f9 000007 1761238664.077333 fallocate 0x021ce1 thru 0x02d2f8 (0xb618 bytes) 000008 1761238664.083565 read 0x00620e thru 0x00a8d0 (0x46c3 bytes) 000009 1761238664.095559 mapwrite 0x030198 thru 0x030a1b (0x884 bytes) truncating to largest ever: 0x3bb36 000010 1761238664.130852 trunc from 0x030a1c to 0x03bb35 (0xb11a bytes) 000011 1761238664.146679 read 0x037192 thru 0x03ac8c (0x3afb bytes) 000012 1761238664.157050 fallocate 0x026417 thru 0x026a32 (0x61c bytes) 000013 1761238664.161651 mapwrite 0x03c433 thru 0x03ffff (0x3bcd bytes) 000014 1761238664.213045 trunc from 0x00d063 to 0x01211f (0x50bd bytes) 000016 1761238664.224850 write 0x035eae thru 0x03999c (0x3aef bytes) 000017 1761238664.226833 read 0x02d344 thru 0x030614 (0x32d1 bytes) 000018 1761238664.228444 mapwrite 0x01ed1c thru 0x02cfb2 (0xe297 bytes) 000020 1761238664.258359 trunc from 0x040000 to 0x00f1b6 (0x30e49 bytes) 000021 1761238664.280158 read 0x00187a thru 0x00dc70 (0xc3f7 bytes) 000022 1761238664.288981 mapwrite 0x03b68e thru 0x03ffff (0x4972 bytes) 000023 1761238664.334477 punch from 0x0343d2 to 0x037953 (0x3582 bytes) 000024 1761238664.339746 mapwrite 0x0309e1 thru 0x030fae (0x5ce bytes) fallocating to largest ever: 0x40000 000025 1761238664.369804 fallocate 0x03f9e6 thru 0x03ffff (0x61a bytes) 000026 1761238664.375428 mapwrite 0x03280b thru 0x03ffff (0xd7f5 bytes) 000027 1761238664.413374 write 0x00ed1f thru 0x01d1d7 (0xe4b9 bytes) 000028 1761238664.414925 trunc from 0x01993e to 0x0249c6 (0xb089 bytes) 000029 1761238664.427320 mapread 0x0239e2 thru 0x024cf8 (0x1317 bytes) 000030 1761238664.449160 write 0x030b44 thru 0x03f9a2 (0xee5f bytes) 000031 1761238664.451806 read 0x00e2d3 thru 0x01805a (0x9d88 bytes) 000032 1761238664.452506 write 0x020da9 thru 0x029301 (0x8559 bytes) 000033 1761238664.465398 read 0x00b40f thru 0x00c91d (0x150f bytes) 000034 1761238664.466343 mapread 0x03e0f9 thru 0x03ffff (0x1f07 bytes) 000036 1761238664.467955 write 0x006f05 thru 0x0145eb (0xd6e7 bytes) 000037 1761238664.470570 read 0x025133 thru 0x027ac6 (0x2994 bytes) 000038 1761238664.472305 punch from 0x03e919 to 0x03ffff (0x16e7 bytes) 000039 1761238664.498687 MIRROR_EXTEND to 2 mirrors 000040 1761238665.166029 MIRROR_RESYNC 2 mirrors 000041 1761238665.377792 write 0x03a7c5 thru 0x03c254 (0x1a90 bytes) 000042 1761238665.442332 trunc from 0x01d33d to 0x0282a7 (0xaf6b bytes) 000043 1761238665.448334 mapwrite 0x001812 thru 0x010783 (0xef72 bytes) 000044 1761238665.492568 fallocate 0x00e950 thru 0x018af2 (0xa1a3 bytes) 000045 1761238665.498736 fallocate 0x024310 thru 0x025660 (0x1351 bytes) 000046 1761238665.505035 mapread 0x01e289 thru 0x02b651 (0xd3c9 bytes) 000048 1761238665.525032 trunc from 0x040000 to 0x03ca5e (0x35a1 bytes) 000049 1761238665.569618 fallocate 0x000052 thru 0x007ae3 (0x7a92 bytes) 000050 1761238665.592280 MIRROR_EXTEND to 3 mirrors 000051 1761238666.327919 mapread 0x011b0f thru 0x01eb72 (0xd064 bytes) 000052 1761238666.359804 write 0x020c3a thru 0x02bdf1 (0xb1b8 bytes) 000053 1761238666.410928 trunc from 0x02efc9 to 0x039efd (0xaf35 bytes) 000054 1761238666.420910 trunc from 0x03ca5f to 0x0258de (0x17180 bytes) 000055 1761238666.453560 write 0x00e863 thru 0x018e43 (0xa5e1 bytes) 000056 1761238666.470603 trunc from 0x0258df to 0x02ed07 (0x9429 bytes) 000057 1761238666.487560 write 0x0333e2 thru 0x037509 (0x4128 bytes) 000058 1761238666.489565 mapread 0x01f59a thru 0x02c1a5 (0xcc0c bytes) 000059 1761238666.501511 fallocate 0x00456a thru 0x00c7fc (0x8293 bytes) 000060 1761238666.508935 fallocate 0x012ffe thru 0x01641f (0x3422 bytes) 000061 1761238666.514746 mapwrite 0x032623 thru 0x03c8ac (0xa28a bytes) 000062 1761238666.571212 MIRROR_RESYNC 3 mirrors 000063 1761238667.410993 trunc from 0x001877 to 0x00f263 (0xd9ed bytes) 000064 1761238667.484969 punch from 0x0078b4 to 0x00b156 (0x38a3 bytes) 000065 1761238667.493995 trunc from 0x02de9c to 0x03c6c0 (0xe825 bytes) 000066 1761238667.507984 read 0x01ec65 thru 0x022d8d (0x4129 bytes) 000067 1761238667.519903 trunc from 0x026bb1 to 0x02a2a9 (0x36f9 bytes) 000069 1761238667.525942 fallocate 0x02375c thru 0x025e49 (0x26ee bytes) 000070 1761238667.531235 mapread 0x01752b thru 0x021fb5 (0xaa8b bytes) 000071 1761238667.550389 read 0x01253b thru 0x018f90 (0x6a56 bytes) 000072 1761238667.551045 trunc from 0x002e97 to 0x007c97 (0x4e01 bytes) 000073 1761238667.556831 punch from 0x01eee4 to 0x027ced (0x8e0a bytes) 000074 1761238667.561550 trunc from 0x03c8ad to 0x038ba4 (0x3d08 bytes) 000075 1761238667.581158 punch from 0x016ad4 to 0x025758 (0xec85 bytes) 000076 1761238667.586525 read 0x02d6ab thru 0x038ba4 (0xb4fa bytes) 000077 1761238667.599415 mapread 0x00ddc2 thru 0x014fc8 (0x7207 bytes) 000078 1761238667.600428 mapread 0x0055de thru 0x014a50 (0xf473 bytes) 000079 1761238667.601372 read 0x005493 thru 0x010dfb (0xb969 bytes) 000080 1761238667.601848 trunc from 0x038ba5 to 0x0329da (0x61ca bytes) 000081 1761238667.623955 write 0x01abdd thru 0x02286e (0x7c92 bytes) 000082 1761238667.625677 mapwrite 0x038f28 thru 0x03ffff (0x70d8 bytes) 000083 1761238667.667851 mapwrite 0x01a1d5 thru 0x01a86e (0x69a bytes) 000084 1761238667.690640 mapwrite 0x02787e thru 0x031c66 (0xa3e9 bytes) 000085 1761238667.723251 write 0x023f65 thru 0x030044 (0xc0e0 bytes) 000086 1761238667.724873 mapread 0x028ad1 thru 0x02cb30 (0x4060 bytes) 000087 1761238667.727680 fallocate 0x0071cd thru 0x00e340 (0x7174 bytes) 000088 1761238667.734502 read 0x0261e9 thru 0x02be49 (0x5c61 bytes) 000089 1761238667.735808 mapread 0x0165c4 thru 0x025378 (0xedb5 bytes) 000090 1761238667.738049 trunc from 0x037365 to 0x03d714 (0x63b0 bytes) 000091 1761238667.745462 trunc from 0x040000 to 0x02be44 (0x141bb bytes) 000092 1761238667.763602 mapread 0x0126c6 thru 0x01d057 (0xa992 bytes) 000093 1761238667.765774 write 0x038ffd thru 0x03ecad (0x5cb1 bytes) 000094 1761238667.771950 MIRROR_RESYNC 3 mirrors 000095 1761238668.814159 trunc from 0x037417 to 0x03ecad (0x7897 bytes) 000096 1761238668.870425 read 0x020c95 thru 0x025f15 (0x5281 bytes) 000097 1761238668.883839 trunc from 0x03ecae to 0x0258e9 (0x193c4 bytes) 000098 1761238668.914805 write 0x02d5db thru 0x038bb8 (0xb5de bytes) 000099 1761238668.916482 trunc from 0x038bb9 to 0x03c9b2 (0x3dfa bytes) 000100 1761238668.943645 punch from 0x02001b to 0x02663b (0x6621 bytes) 000101 1761238668.948915 write 0x01a50e thru 0x027f34 (0xda27 bytes) 000102 1761238668.959364 punch from 0x004a11 to 0x00d90f (0x8eff bytes) 000103 1761238668.977628 trunc from 0x029cd2 to 0x039078 (0xf3a7 bytes) 000104 1761238669.023366 MIRROR_RESYNC 3 mirrors 000105 1761238670.104537 mapread 0x0372be thru 0x03c9b2 (0x56f5 bytes) 000106 1761238670.155491 mapread 0x02c683 thru 0x03597b (0x92f9 bytes) 000107 1761238670.174320 punch from 0x014e90 to 0x01ff82 (0xb0f3 bytes) 000108 1761238670.263513 read 0x024c4f thru 0x02b327 (0x66d9 bytes) 000109 1761238670.273622 fallocate 0x03bbc4 thru 0x03eb0b (0x2f48 bytes) 000110 1761238670.282380 trunc from 0x03c9b3 to 0x0325b1 (0xa401 bytes) 000111 1761238670.328084 MIRROR_RESYNC 3 mirrors 000112 1761238671.480804 write 0x029382 thru 0x02ba16 (0x2695 bytes) 000113 1761238671.525822 mapread 0x02c211 thru 0x0325b1 (0x63a1 bytes) 000114 1761238671.540744 write 0x01af53 thru 0x0212fe (0x63ac bytes) 000115 1761238671.557609 mapread 0x029c94 thru 0x0325b1 (0x891e bytes) 000116 1761238671.563797 mapread 0x00eb22 thru 0x0162bc (0x779b bytes) 000117 1761238671.579079 trunc from 0x000225 to 0x0015c5 (0x13a1 bytes) 000118 1761238671.588681 fallocate 0x037a85 thru 0x0393bd (0x1939 bytes) 000119 1761238671.597184 mapread 0x0031ca thru 0x003e17 (0xc4e bytes) 000121 1761238671.606232 read 0x0273ad thru 0x035f08 (0xeb5c bytes) 000122 1761238671.631971 mapread 0x027e35 thru 0x033199 (0xb365 bytes) 000123 1761238671.635505 trunc from 0x0071b6 to 0x00c0dd (0x4f28 bytes) 000124 1761238671.643395 trunc from 0x0393be to 0x025b09 (0x138b4 bytes) 000125 1761238671.669365 mapread 0x016e21 thru 0x01c71a (0x58fa bytes) 000126 1761238671.670317 read 0x00a7ce thru 0x014308 (0x9b3b bytes) 000127 1761238671.682232 punch from 0x01d1d2 to 0x020be7 (0x3a16 bytes) 000129 1761238671.698372 trunc from 0x025b0a to 0x039dee (0x142e5 bytes) 000130 1761238671.719352 read 0x012248 thru 0x01ada9 (0x8b62 bytes) 000131 1761238671.722297 write 0x03b7a5 thru 0x03ffff (0x485b bytes) 000132 1761238671.724311 trunc from 0x01c8c6 to 0x02841e (0xbb59 bytes) 000133 1761238671.731841 punch from 0x00afe1 to 0x00b2c4 (0x2e4 bytes) 000134 1761238671.736572 trunc from 0x040000 to 0x00d3c6 (0x32c39 bytes) 000135 1761238671.756209 write 0x01e86a thru 0x020374 (0x1b0b bytes) 000137 1761238671.757709 fallocate 0x03d4e2 thru 0x03ffff (0x2b1e bytes) 000138 1761238671.767944 fallocate 0x021a05 thru 0x022c9d (0x1299 bytes) 000139 1761238671.772860 read 0x03e2a0 thru 0x03ffff (0x1d60 bytes) 000140 1761238671.802573 fallocate 0x0034cf thru 0x003d43 (0x875 bytes) 000141 1761238671.811653 mapread 0x0017fb thru 0x007e1a (0x6620 bytes) 000142 1761238671.813848 trunc from 0x01a467 to 0x029616 (0xf1b0 bytes) 000143 1761238671.841964 MIRROR_EXTEND to 4 mirrors 000144 1761238672.539341 trunc from 0x040000 to 0x03cd2c (0x32d3 bytes) 000145 1761238672.599927 trunc from 0x023a56 to 0x031694 (0xdc3f bytes) 000147 1761238672.611735 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000148 1761238672.898535 read 0x038618 thru 0x03cd2c (0x4715 bytes) 000149 1761238672.916361 read 0x01631c thru 0x01940b (0x30f0 bytes) 000150 1761238672.933748 mapwrite 0x00fb2f thru 0x01b78c (0xbc5e bytes) 000151 1761238672.983677 mapread 0x02827e thru 0x032800 (0xa583 bytes) 000152 1761238672.986464 read 0x00f030 thru 0x00f7b7 (0x788 bytes) 000153 1761238672.986984 write 0x004ff9 thru 0x009511 (0x4519 bytes) 000154 1761238673.027704 MIRROR_EXTEND to 4 mirrors 000155 1761238674.185383 trunc from 0x03cd2d to 0x00f3eb (0x2d941 bytes) 000156 1761238674.295462 mapwrite 0x027f0a thru 0x036839 (0xe930 bytes) 000157 1761238674.404515 mapread 0x030958 thru 0x0353df (0x4a88 bytes) 000158 1761238674.405497 trunc from 0x03272f to 0x036839 (0x410b bytes) 000159 1761238674.411573 mapwrite 0x017f04 thru 0x026cce (0xedcb bytes) 000160 1761238674.497450 trunc from 0x03683a to 0x004de0 (0x31a59 bytes) 000161 1761238674.535193 trunc from 0x0011d5 to 0x004de0 (0x3c0c bytes) 000162 1761238674.545206 punch from 0x000e45 to 0x004de0 (0x3f9c bytes) 000164 1761238674.556281 punch from 0x0000a2 to 0x004de0 (0x4d3f bytes) 000165 1761238674.574612 MIRROR_EXTEND to 5 mirrors 000167 1761238675.533704 mapwrite 0x01f005 thru 0x0286d0 (0x96cc bytes) 000168 1761238675.614421 trunc from 0x02220e to 0x0286d0 (0x64c3 bytes) 000169 1761238675.621768 trunc from 0x00dc2d to 0x01b26d (0xd641 bytes) 000170 1761238675.628765 trunc from 0x0286d1 to 0x039a16 (0x11346 bytes) 000171 1761238675.655859 MIRROR_EXTEND to 6 mirrors 000172 1761238676.615028 trunc from 0x00acca to 0x00cb86 (0x1ebd bytes) 000173 1761238676.667604 trunc from 0x035527 to 0x039a16 (0x44f0 bytes) 000174 1761238676.672373 trunc from 0x01ceed to 0x0278c3 (0xa9d7 bytes) 000175 1761238676.676634 trunc from 0x003cb0 to 0x0122c1 (0xe612 bytes) 000176 1761238676.682794 write 0x034322 thru 0x03b427 (0x7106 bytes) 000177 1761238676.700265 MIRROR_RESYNC 6 mirrors 000178 1761238677.809891 MIRROR_EXTEND to 7 mirrors 000179 1761238678.494815 mapwrite 0x018bb0 thru 0x01ca15 (0x3e66 bytes) 000180 1761238678.641054 trunc from 0x03b428 to 0x02a649 (0x10dde bytes) 000181 1761238678.680423 fallocate 0x015948 thru 0x020dbe (0xb477 bytes) 000182 1761238678.686926 fallocate 0x0152d1 thru 0x021fc0 (0xccf0 bytes) 000183 1761238678.697653 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000184 1761238679.060801 read 0x016f0f thru 0x021596 (0xa688 bytes) 000185 1761238679.071655 fallocate 0x01ed10 thru 0x01fb0f (0xe00 bytes) 000186 1761238679.077868 mapwrite 0x012373 thru 0x01f417 (0xd0a5 bytes) 000187 1761238679.126567 mapwrite 0x0023bc thru 0x002724 (0x369 bytes) 000188 1761238679.153698 read 0x0073fb thru 0x00b1ea (0x3df0 bytes) 000189 1761238679.154242 trunc from 0x02a64a to 0x028947 (0x1d02 bytes) 000190 1761238679.196176 read 0x005e89 thru 0x0097c4 (0x393c bytes) 000191 1761238679.196799 write 0x02c84e thru 0x02f12e (0x28e1 bytes) 000192 1761238679.197752 trunc from 0x02f12f to 0x033fad (0x4e7f bytes) 000193 1761238679.230708 write 0x03f969 thru 0x03ffff (0x697 bytes) 000195 1761238679.232407 punch from 0x034b22 to 0x03d9c8 (0x8ea7 bytes) 000196 1761238679.237447 fallocate 0x010bfc thru 0x015877 (0x4c7c bytes) 000197 1761238679.245002 fallocate 0x0194b5 thru 0x02669e (0xd1ea bytes) 000198 1761238679.251207 punch from 0x0306ef to 0x03813a (0x7a4c bytes) 000199 1761238679.256351 mapread 0x02d0e9 thru 0x03baed (0xea05 bytes) 000200 1761238679.272277 punch from 0x018239 to 0x0208bd (0x8685 bytes) 000201 1761238679.278357 punch from 0x002af6 to 0x00a6df (0x7bea bytes) 000202 1761238679.291410 punch from 0x03fdf6 to 0x03ffff (0x20a bytes) 000203 1761238679.307677 fallocate 0x03199f thru 0x03ffff (0xe661 bytes) 000204 1761238679.314265 fallocate 0x019ce7 thru 0x023538 (0x9852 bytes) 000205 1761238679.322173 read 0x03ffc5 thru 0x03ffff (0x3b bytes) 000206 1761238679.336701 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000207 1761238679.598357 mapwrite 0x01ed2f thru 0x027505 (0x87d7 bytes) 000208 1761238679.655709 fallocate 0x01943c thru 0x024c87 (0xb84c bytes) 000209 1761238679.669938 trunc from 0x02cb39 to 0x0328ce (0x5d96 bytes) 000210 1761238679.696402 MIRROR_RESYNC 5 mirrors 000211 1761238681.752817 trunc from 0x02a81a to 0x02df9a (0x3781 bytes) 000212 1761238681.814647 punch from 0x01452a to 0x01b828 (0x72ff bytes) 000213 1761238681.821573 punch from 0x02b1fa to 0x02b3a7 (0x1ae bytes) 000214 1761238681.831005 write 0x03b690 thru 0x03ffff (0x4970 bytes) 000215 1761238681.838726 mapread 0x0038a5 thru 0x008f3f (0x569b bytes) 000216 1761238681.863473 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000217 1761238682.108020 mapwrite 0x02fbc4 thru 0x0362e6 (0x6723 bytes) 000219 1761238682.147922 read 0x014683 thru 0x016832 (0x21b0 bytes) 000220 1761238682.153550 trunc from 0x03306a to 0x03c389 (0x9320 bytes) 000221 1761238682.158812 read 0x023732 thru 0x02f5c5 (0xbe94 bytes) 000222 1761238682.167506 punch from 0x0379a5 to 0x03ffff (0x865b bytes) 000224 1761238682.174860 trunc from 0x02625f to 0x02a40f (0x41b1 bytes) 000225 1761238682.186846 write 0x034869 thru 0x037807 (0x2f9f bytes) 000226 1761238682.198890 mapread 0x012be7 thru 0x01c4a7 (0x98c1 bytes) 000227 1761238682.204831 MIRROR_RESYNC 4 mirrors 000228 1761238683.517805 read 0x018f2c thru 0x01e0d3 (0x51a8 bytes) 000229 1761238683.526208 trunc from 0x02548d to 0x028c9d (0x3811 bytes) 000230 1761238683.561707 mapread 0x0350bf thru 0x03ffff (0xaf41 bytes) 000231 1761238683.580323 mapread 0x019d36 thru 0x01c71c (0x29e7 bytes) 000232 1761238683.589834 trunc from 0x040000 to 0x0245f0 (0x1ba0f bytes) 000233 1761238683.607571 mapread 0x0111f2 thru 0x01dcef (0xcafe bytes) 000234 1761238683.615870 mapread 0x00b057 thru 0x0109ae (0x5958 bytes) 000236 1761238683.617556 write 0x03c626 thru 0x03ffff (0x39da bytes) 000237 1761238683.619617 write 0x03dbd9 thru 0x03ffff (0x2427 bytes) 000238 1761238683.620529 write 0x029969 thru 0x03451e (0xabb6 bytes) 000242 1761238683.639574 trunc from 0x040000 to 0x03b90a (0x46f5 bytes) 000243 1761238683.667842 write 0x0329a6 thru 0x0363ea (0x3a45 bytes) 000244 1761238683.682003 write 0x02afd3 thru 0x039051 (0xe07f bytes) 000245 1761238683.686130 trunc from 0x03b90b to 0x01260b (0x292ff bytes) 000246 1761238683.701800 mapwrite 0x038ee7 thru 0x03a971 (0x1a8b bytes) 000247 1761238683.754711 write 0x000f44 thru 0x007d50 (0x6e0d bytes) 000248 1761238683.767654 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000249 1761238683.970261 trunc from 0x03763d to 0x0381f0 (0xbb4 bytes) 000250 1761238683.975902 trunc from 0x00b879 to 0x0127b8 (0x6f40 bytes) 000251 1761238683.980359 mapwrite 0x021ee3 thru 0x023ebe (0x1fdc bytes) 000252 1761238684.005719 read 0x01ebf9 thru 0x02abde (0xbfe6 bytes) 000253 1761238684.006504 write 0x029ef7 thru 0x02ae2a (0xf34 bytes) 000254 1761238684.012378 mapread 0x00a97a thru 0x01325c (0x88e3 bytes) 000255 1761238684.036847 fallocate 0x014ac0 thru 0x01f3d2 (0xa913 bytes) 000256 1761238684.041658 mapread 0x00c50c thru 0x00f920 (0x3415 bytes) 000257 1761238684.045554 MIRROR_RESYNC 3 mirrors 000258 1761238685.001728 trunc from 0x009294 to 0x017d40 (0xeaad bytes) 000260 1761238685.055081 mapread 0x02090f thru 0x027749 (0x6e3b bytes) 000261 1761238685.073259 trunc from 0x03a972 to 0x01d4f2 (0x1d47f bytes) 000262 1761238685.130333 read 0x016ad9 thru 0x01d4f2 (0x6a1a bytes) 000263 1761238685.137716 write 0x004327 thru 0x010a53 (0xc72d bytes) 000264 1761238685.146660 punch from 0x00a3fa to 0x01289f (0x84a6 bytes) 000265 1761238685.161665 read 0x002b97 thru 0x002e6a (0x2d4 bytes) 000266 1761238685.162298 write 0x0172e8 thru 0x01827c (0xf95 bytes) 000267 1761238685.163306 trunc from 0x01d4f3 to 0x033607 (0x16115 bytes) 000268 1761238685.184470 fallocate 0x00f75d thru 0x00f813 (0xb7 bytes) 000270 1761238685.192508 trunc from 0x0156ae to 0x020037 (0xa98a bytes) 000271 1761238685.210340 punch from 0x00c784 to 0x015086 (0x8903 bytes) 000272 1761238685.215337 write 0x01c3e5 thru 0x020a9c (0x46b8 bytes) 000273 1761238685.227747 read 0x016ff0 thru 0x017754 (0x765 bytes) 000274 1761238685.229194 trunc from 0x033608 to 0x0387fb (0x51f4 bytes) 000275 1761238685.241733 fallocate 0x03a933 thru 0x03ffff (0x56cd bytes) 000276 1761238685.253074 mapread 0x03659b thru 0x0387fb (0x2261 bytes) 000277 1761238685.267971 trunc from 0x0387fc to 0x03d2d5 (0x4ada bytes) 000278 1761238685.289647 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000279 1761238685.527538 trunc from 0x03d2d6 to 0x022c0b (0x1a6ca bytes) 000280 1761238685.549826 trunc from 0x019add to 0x022c0b (0x912f bytes) 000281 1761238685.555824 punch from 0x020b1e to 0x022c0b (0x20ee bytes) 000282 1761238685.559983 read 0x00171c thru 0x00e2c0 (0xcba5 bytes) 000283 1761238685.570574 read 0x006156 thru 0x01277a (0xc625 bytes) 000284 1761238685.581893 write 0x0224f5 thru 0x02e947 (0xc453 bytes) 000285 1761238685.583652 fallocate 0x036584 thru 0x03cdb5 (0x6832 bytes) 000288 1761238685.594456 punch from 0x01fedd to 0x02e0a1 (0xe1c5 bytes) 000289 1761238685.618319 write 0x03b25e thru 0x03ffff (0x4da2 bytes) 000290 1761238685.619577 trunc from 0x01c384 to 0x0232d6 (0x6f53 bytes) 000292 1761238685.628632 trunc from 0x0161bd to 0x016acb (0x90f bytes) 000293 1761238685.635039 fallocate 0x03a1de thru 0x03d6e2 (0x3505 bytes) 000294 1761238685.640736 write 0x00391e thru 0x00d106 (0x97e9 bytes) 000295 1761238685.641978 trunc from 0x01e55e to 0x025f85 (0x7a28 bytes) 000296 1761238685.646763 read 0x000350 thru 0x009a6f (0x9720 bytes) 000297 1761238685.649160 mapwrite 0x02d84c thru 0x038876 (0xb02b bytes) 000298 1761238685.678447 mapread 0x022c75 thru 0x0249f9 (0x1d85 bytes) 000299 1761238685.679400 punch from 0x009283 to 0x00d67b (0x43f9 bytes) 000300 1761238685.686708 trunc from 0x040000 to 0x035e9d (0xa162 bytes) 000301 1761238685.705316 mapread 0x0215c8 thru 0x022895 (0x12ce bytes) 000302 1761238685.706304 trunc from 0x002655 to 0x00a6fa (0x80a6 bytes) 000303 1761238685.713433 trunc from 0x035e9e to 0x03b31d (0x5480 bytes) 000304 1761238685.733435 trunc from 0x0226b6 to 0x024b9f (0x24ea bytes) 000305 1761238685.742639 write 0x02877c thru 0x034c89 (0xc50e bytes) 000306 1761238685.744157 trunc from 0x02d9a4 to 0x02f07a (0x16d7 bytes) 000308 1761238685.778147 punch from 0x001d04 to 0x00683e (0x4b3b bytes) 000309 1761238685.784974 trunc from 0x03b31e to 0x02f518 (0xbe05 bytes) 000310 1761238685.825327 read 0x0170b2 thru 0x01aecd (0x3e1c bytes) 000311 1761238685.838122 write 0x03a8da thru 0x03c29a (0x19c1 bytes) 000312 1761238685.839931 trunc from 0x03c29b to 0x01c2b1 (0x1ffe9 bytes) 000313 1761238685.878614 mapwrite 0x01ed9a thru 0x028be6 (0x9e4d bytes) 000314 1761238685.948897 write 0x015974 thru 0x02150d (0xbb9a bytes) 000315 1761238685.950410 trunc from 0x028be7 to 0x0337c9 (0xabe3 bytes) 000316 1761238685.986288 mapread 0x0097a8 thru 0x01885e (0xf0b7 bytes) 000317 1761238685.990701 mapwrite 0x019937 thru 0x028ce4 (0xf3ae bytes) 000318 1761238686.046952 trunc from 0x0337ca to 0x00f204 (0x245c5 bytes) 000319 1761238686.096877 write 0x035f83 thru 0x03d8a4 (0x7922 bytes) 000320 1761238686.100258 write 0x01cd62 thru 0x01f90f (0x2bae bytes) 000321 1761238686.122210 trunc from 0x02513c to 0x02e70c (0x95d1 bytes) 000322 1761238686.141072 MIRROR_RESYNC 2 mirrors 000323 1761238686.928960 trunc from 0x03d8a5 to 0x02fcf5 (0xdbaf bytes) 000324 1761238686.977682 trunc from 0x0242ba to 0x02825d (0x3fa4 bytes) 000326 1761238686.990133 punch from 0x01445a to 0x01a038 (0x5bdf bytes) 000327 1761238687.004338 MIRROR_EXTEND to 3 mirrors 000328 1761238688.073824 write 0x026785 thru 0x02fd20 (0x959c bytes) 000329 1761238688.125636 read 0x002078 thru 0x0068c8 (0x4851 bytes) 000330 1761238688.143779 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000331 1761238688.392046 mapread 0x01386f thru 0x0218e7 (0xe079 bytes) 000332 1761238688.427932 mapread 0x02bff7 thru 0x02d3a6 (0x13b0 bytes) 000333 1761238688.453176 mapwrite 0x033671 thru 0x036098 (0x2a28 bytes) 000334 1761238688.519707 write 0x03ccb4 thru 0x03fe7b (0x31c8 bytes) 000335 1761238688.520831 trunc from 0x03fe7c to 0x00c014 (0x33e67 bytes) 000336 1761238688.559820 mapread 0x00b406 thru 0x00c014 (0xc0f bytes) 000337 1761238688.570137 trunc from 0x00c015 to 0x01304e (0x703a bytes) 000338 1761238688.591269 mapread 0x00e560 thru 0x01304e (0x4aef bytes) 000339 1761238688.601394 mapread 0x005ac0 thru 0x008a84 (0x2fc5 bytes) 000340 1761238688.602266 punch from 0x003fc0 to 0x00cf23 (0x8f64 bytes) 000341 1761238688.611415 write 0x01922e thru 0x024a7e (0xb851 bytes) 000342 1761238688.616002 trunc from 0x011f85 to 0x012c31 (0xcad bytes) 000343 1761238688.627277 mapwrite 0x02c120 thru 0x02f5eb (0x34cc bytes) 000345 1761238688.722314 read 0x01c4b0 thru 0x0244c9 (0x801a bytes) 000346 1761238688.722851 punch from 0x0144dd to 0x01b345 (0x6e69 bytes) 000347 1761238688.729345 mapread 0x01bbfc thru 0x02a84b (0xec50 bytes) 000348 1761238688.748045 trunc from 0x02f5ec to 0x011757 (0x1de94 bytes) 000349 1761238688.781983 trunc from 0x009697 to 0x011757 (0x80c1 bytes) 000351 1761238688.800933 punch from 0x006e72 to 0x011757 (0xa8e6 bytes) 000352 1761238688.812672 mapread 0x00ef17 thru 0x011757 (0x2841 bytes) 000353 1761238688.822700 trunc from 0x011758 to 0x008967 (0x8df0 bytes) 000355 1761238688.841844 trunc from 0x003c0d to 0x00893f (0x4d33 bytes) 000356 1761238688.848809 punch from 0x005cee to 0x008967 (0x2c7a bytes) 000357 1761238688.876778 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000360 1761238689.026738 trunc from 0x008968 to 0x018b57 (0x101f0 bytes) 000361 1761238689.050970 mapread 0x004ded thru 0x00dd77 (0x8f8b bytes) 000363 1761238689.071421 read 0x007994 thru 0x013986 (0xbff3 bytes) 000364 1761238689.081846 punch from 0x0150f2 to 0x0187c2 (0x36d1 bytes) 000365 1761238689.088869 write 0x0383fe thru 0x03ffff (0x7c02 bytes) 000366 1761238689.090531 trunc from 0x01aaea to 0x0260ab (0xb5c2 bytes) 000368 1761238689.098332 mapwrite 0x030ba7 thru 0x0396a2 (0x8afc bytes) 000369 1761238689.129244 trunc from 0x040000 to 0x03c57a (0x3a85 bytes) 000370 1761238689.151442 write 0x021687 thru 0x026737 (0x50b1 bytes) 000371 1761238689.163937 punch from 0x0190ad to 0x01e0d7 (0x502b bytes) 000372 1761238689.169674 mapread 0x0392b1 thru 0x03c57a (0x32ca bytes) 000373 1761238689.170535 fallocate 0x0337d5 thru 0x03c44b (0x8c77 bytes) 000375 1761238689.175973 mapread 0x029d88 thru 0x035bf2 (0xbe6b bytes) 000376 1761238689.176911 trunc from 0x03c57b to 0x019840 (0x22d3a bytes) 000377 1761238689.196645 trunc from 0x019841 to 0x028e16 (0xf5d6 bytes) 000379 1761238689.224280 read 0x00f474 thru 0x0194a6 (0xa033 bytes) 000380 1761238689.244726 trunc from 0x028e17 to 0x0365cb (0xd7b5 bytes) 000381 1761238689.273324 mapread 0x028361 thru 0x0326ba (0xa35a bytes) 000382 1761238689.284268 punch from 0x0288cf to 0x0365cb (0xdcfd bytes) 000383 1761238689.293425 trunc from 0x0365cc to 0x035dd6 (0x7f5 bytes) 000384 1761238689.318235 fallocate 0x024ad7 thru 0x02fae5 (0xb00f bytes) 000386 1761238689.326188 mapread 0x00ed34 thru 0x010d8b (0x2058 bytes) 000387 1761238689.326856 trunc from 0x035dd7 to 0x0085e5 (0x2d7f1 bytes) 000388 1761238689.349195 mapwrite 0x032614 thru 0x035552 (0x2f3f bytes) 000389 1761238689.409704 fallocate 0x0224cb thru 0x02a6ca (0x8200 bytes) 000392 1761238689.415399 fallocate 0x0340ee thru 0x03fb52 (0xba65 bytes) 000394 1761238689.424594 punch from 0x0319b4 to 0x032847 (0xe94 bytes) 000395 1761238689.434627 MIRROR_EXTEND to 2 mirrors 000396 1761238690.206017 trunc from 0x005512 to 0x013f4e (0xea3d bytes) 000397 1761238690.268590 mapwrite 0x01d6f8 thru 0x024e00 (0x7709 bytes) 000398 1761238690.307993 fallocate 0x03e9a9 thru 0x03ffff (0x1657 bytes) 000400 1761238690.314731 trunc from 0x040000 to 0x01f2c8 (0x20d37 bytes) 000401 1761238690.394981 MIRROR_EXTEND to 3 mirrors 000402 1761238691.151140 read 0x004fb9 thru 0x0062ef (0x1337 bytes) 000403 1761238691.156830 trunc from 0x01f2c9 to 0x01750b (0x7dbd bytes) 000405 1761238691.227338 trunc from 0x01750c to 0x00efe9 (0x8522 bytes) 000406 1761238691.246625 trunc from 0x00efea to 0x0172ae (0x82c5 bytes) 000407 1761238691.263413 mapread 0x00cfcb thru 0x013ed4 (0x6f0a bytes) 000408 1761238691.292387 punch from 0x006512 to 0x00d7ba (0x72a9 bytes) 000409 1761238691.305809 mapwrite 0x0253d6 thru 0x033ff9 (0xec24 bytes) 000410 1761238691.387289 fallocate 0x036a3f thru 0x039402 (0x29c4 bytes) 000411 1761238691.405612 MIRROR_RESYNC 3 mirrors 000412 1761238692.126976 punch from 0x018e72 to 0x0207a9 (0x7938 bytes) 000414 1761238692.168789 trunc from 0x003f56 to 0x00a002 (0x60ad bytes) 000415 1761238692.174810 punch from 0x027cf8 to 0x033ff9 (0xc302 bytes) 000416 1761238692.180911 trunc from 0x033ffa to 0x02d28c (0x6d6d bytes) 000417 1761238692.225331 trunc from 0x001d7f to 0x00452b (0x27ad bytes) 000418 1761238692.243403 MIRROR_RESYNC 3 mirrors 000419 1761238693.099389 trunc from 0x02d28d to 0x025cc2 (0x75ca bytes) 000421 1761238693.170759 mapread 0x01aaa0 thru 0x024672 (0x9bd3 bytes) 000422 1761238693.210779 fallocate 0x004125 thru 0x00f0a9 (0xaf85 bytes) 000423 1761238693.217610 write 0x035bb7 thru 0x038ce0 (0x312a bytes) 000424 1761238693.218621 mapread 0x031639 thru 0x03627c (0x4c44 bytes) 000426 1761238693.243440 mapwrite 0x02b701 thru 0x034a62 (0x9362 bytes) 000427 1761238693.266970 read 0x036bfb thru 0x03722e (0x634 bytes) 000428 1761238693.267622 fallocate 0x01ae36 thru 0x01b5b1 (0x77c bytes) 000429 1761238693.272793 write 0x000574 thru 0x00fb73 (0xf600 bytes) 000430 1761238693.288651 trunc from 0x037449 to 0x038ce0 (0x1898 bytes) 000431 1761238693.300612 mapwrite 0x0235a7 thru 0x02c3a5 (0x8dff bytes) 000432 1761238693.364783 trunc from 0x01d4c9 to 0x021b71 (0x46a9 bytes) 000434 1761238693.370699 trunc from 0x038ce1 to 0x01585b (0x23485 bytes) 000435 1761238693.400278 write 0x00bde6 thru 0x012b63 (0x6d7e bytes) 000436 1761238693.406282 trunc from 0x01585c to 0x01fb54 (0xa2f9 bytes) 000437 1761238693.432254 trunc from 0x0022d1 to 0x009e2e (0x7b5e bytes) 000438 1761238693.449361 MIRROR_RESYNC 3 mirrors 000439 1761238694.327509 mapwrite 0x0265ab thru 0x02eb4f (0x85a5 bytes) 000441 1761238694.489297 MIRROR_EXTEND to 4 mirrors 000442 1761238695.013622 fallocate 0x003aaf thru 0x00530e (0x1860 bytes) 000443 1761238695.050561 fallocate 0x0332b8 thru 0x03f6f8 (0xc441 bytes) 000444 1761238695.056793 read 0x013328 thru 0x0218ab (0xe584 bytes) 000445 1761238695.066989 mapwrite 0x0160dd thru 0x01cfd8 (0x6efc bytes) 000446 1761238695.082909 punch from 0x01a01e to 0x025ff1 (0xbfd4 bytes) 000448 1761238695.089991 mapwrite 0x00ab27 thru 0x019960 (0xee3a bytes) 000449 1761238695.132793 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000450 1761238695.412003 mapread 0x03c485 thru 0x03f6f8 (0x3274 bytes) 000451 1761238695.446048 write 0x016ab0 thru 0x01d24e (0x679f bytes) 000452 1761238695.477500 write 0x017df7 thru 0x025590 (0xd79a bytes) 000453 1761238695.486863 punch from 0x00f14e to 0x012e97 (0x3d4a bytes) 000454 1761238695.498924 fallocate 0x035ac1 thru 0x03a918 (0x4e58 bytes) 000455 1761238695.514950 mapread 0x026b9a thru 0x02e9f9 (0x7e60 bytes) 000456 1761238695.556274 trunc from 0x03f6f9 to 0x01d245 (0x224b3 bytes) 000457 1761238695.617422 mapwrite 0x037e51 thru 0x03ffff (0x81af bytes) 000458 1761238695.696521 MIRROR_RESYNC 3 mirrors 000459 1761238696.749494 punch from 0x028b1e to 0x02c0b1 (0x3594 bytes) 000460 1761238696.829729 punch from 0x001a5a to 0x00e222 (0xc7c9 bytes) 000461 1761238696.839116 write 0x012bc9 thru 0x01f4d4 (0xc90c bytes) 000462 1761238696.868270 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000463 1761238697.142443 fallocate 0x022c8b thru 0x0321dc (0xf552 bytes) 000464 1761238697.149188 trunc from 0x040000 to 0x0315af (0xea50 bytes) 000465 1761238697.166607 trunc from 0x0315b0 to 0x030cbd (0x8f2 bytes) 000466 1761238697.190246 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000467 1761238697.350610 trunc from 0x030cbe to 0x00c015 (0x24ca8 bytes) 000468 1761238697.370703 write 0x015b07 thru 0x01b1f7 (0x56f1 bytes) 000469 1761238697.371810 read 0x00f871 thru 0x01b1f7 (0xb987 bytes) 000470 1761238697.378132 read 0x003c7c thru 0x00e684 (0xaa09 bytes) 000471 1761238697.393006 read 0x007b81 thru 0x00cd8c (0x520c bytes) 000472 1761238697.393580 read 0x0129c3 thru 0x0195d7 (0x6c15 bytes) 000473 1761238697.394145 punch from 0x0082e1 to 0x011935 (0x9655 bytes) 000476 1761238697.401367 trunc from 0x00ee38 to 0x00f7da (0x9a3 bytes) 000477 1761238697.410050 punch from 0x00b481 to 0x0189a0 (0xd520 bytes) 000478 1761238697.423623 fallocate 0x02d559 thru 0x038d0f (0xb7b7 bytes) 000479 1761238697.431046 write 0x035654 thru 0x03ffff (0xa9ac bytes) 000480 1761238697.432459 mapread 0x016e88 thru 0x01e60c (0x7785 bytes) 000481 1761238697.442687 trunc from 0x02a103 to 0x02a617 (0x515 bytes) 000482 1761238697.448514 punch from 0x02b5ea to 0x02ec83 (0x369a bytes) 000483 1761238697.453078 write 0x005d85 thru 0x009615 (0x3891 bytes) 000484 1761238697.454052 mapwrite 0x03e6eb thru 0x03ffff (0x1915 bytes) 000485 1761238697.468756 read 0x021509 thru 0x0256d7 (0x41cf bytes) 000486 1761238697.469337 punch from 0x039f45 to 0x03ffff (0x60bb bytes) 000487 1761238697.476081 punch from 0x03d20f to 0x03ffff (0x2df1 bytes) 000488 1761238697.485612 mapwrite 0x01d3a8 thru 0x01f704 (0x235d bytes) 000489 1761238697.499142 write 0x009153 thru 0x009e21 (0xccf bytes) 000491 1761238697.500545 write 0x028b04 thru 0x029577 (0xa74 bytes) 000492 1761238697.501241 read 0x03768a thru 0x03ffff (0x8976 bytes) 000493 1761238697.516229 punch from 0x00f3b1 to 0x01555f (0x61af bytes) 000494 1761238697.520742 trunc from 0x02ed7d to 0x03c0cb (0xd34f bytes) 000496 1761238697.529305 mapwrite 0x02998b thru 0x034bff (0xb275 bytes) 000497 1761238697.577473 fallocate 0x0055ee thru 0x014f9a (0xf9ad bytes) 000498 1761238697.583719 mapwrite 0x001a96 thru 0x00b8ed (0x9e58 bytes) 000500 1761238697.612569 trunc from 0x040000 to 0x004d29 (0x3b2d6 bytes) 000503 1761238697.634169 punch from 0x0029c1 to 0x004d29 (0x2369 bytes) 000504 1761238697.642190 mapread 0x004588 thru 0x004d29 (0x7a2 bytes) 000505 1761238697.649328 write 0x034a36 thru 0x03722e (0x27f9 bytes) 000507 1761238697.651370 trunc from 0x00fb5d to 0x0130dd (0x3581 bytes) 000509 1761238697.657766 write 0x035bea thru 0x03ffff (0xa416 bytes) 000510 1761238697.659075 mapwrite 0x00e6ee thru 0x017373 (0x8c86 bytes) 000511 1761238697.748243 trunc from 0x01fc0c to 0x029650 (0x9a45 bytes) 000513 1761238697.759372 mapread 0x03c216 thru 0x03faa5 (0x3890 bytes) 000514 1761238697.760304 mapwrite 0x03b99c thru 0x03ffff (0x4664 bytes) 000515 1761238697.783693 MIRROR_EXTEND to 2 mirrors 000516 1761238698.718383 mapread 0x026007 thru 0x026b97 (0xb91 bytes) 000517 1761238698.726228 trunc from 0x040000 to 0x002510 (0x3daef bytes) 000518 1761238698.805689 fallocate 0x03c409 thru 0x03ffff (0x3bf7 bytes) 000519 1761238698.830364 mapwrite 0x02310d thru 0x026ac7 (0x39bb bytes) 000521 1761238698.880846 trunc from 0x019d9a to 0x027953 (0xdbba bytes) 000522 1761238698.892266 trunc from 0x040000 to 0x010056 (0x2ffa9 bytes) 000523 1761238698.926656 trunc from 0x004610 to 0x00d344 (0x8d35 bytes) 000524 1761238698.932287 trunc from 0x010057 to 0x02682e (0x167d8 bytes) 000525 1761238698.961267 write 0x03a53d thru 0x03ffff (0x5ac3 bytes) 000526 1761238698.962624 trunc from 0x03a0fa to 0x03e879 (0x4780 bytes) 000527 1761238698.977277 mapread 0x01535c thru 0x019b86 (0x482b bytes) 000528 1761238698.985058 mapwrite 0x0321b5 thru 0x0397ed (0x7639 bytes) 000529 1761238699.009330 mapwrite 0x01cd99 thru 0x01d1f4 (0x45c bytes) 000530 1761238699.026166 fallocate 0x0367f3 thru 0x03ffff (0x980d bytes) 000531 1761238699.033053 trunc from 0x040000 to 0x00a738 (0x358c7 bytes) 000532 1761238699.050958 trunc from 0x008921 to 0x00a738 (0x1e18 bytes) 000533 1761238699.056038 trunc from 0x0061e1 to 0x00a738 (0x4558 bytes) 000534 1761238699.060290 punch from 0x003d3b to 0x00a738 (0x69fe bytes) 000535 1761238699.064950 trunc from 0x008e70 to 0x00a738 (0x18c9 bytes) 000536 1761238699.074041 MIRROR_EXTEND to 3 mirrors 000537 1761238699.658844 trunc from 0x00a739 to 0x03de19 (0x336e1 bytes) 000538 1761238699.728351 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000540 1761238699.936439 mapread 0x019f20 thru 0x02237e (0x845f bytes) 000541 1761238699.961362 fallocate 0x032123 thru 0x03ffff (0xdedd bytes) 000542 1761238699.971120 trunc from 0x040000 to 0x0045c1 (0x3ba3e bytes) 000543 1761238699.995907 mapread 0x001908 thru 0x0045c1 (0x2cba bytes) 000544 1761238700.005602 trunc from 0x0045c2 to 0x0286f7 (0x24136 bytes) 000545 1761238700.042161 mapwrite 0x03e374 thru 0x03ffff (0x1c8c bytes) 000546 1761238700.114571 trunc from 0x0140f7 to 0x0190e4 (0x4fee bytes) 000547 1761238700.122649 mapread 0x0036af thru 0x00ca3c (0x938e bytes) 000548 1761238700.134262 write 0x03b7a6 thru 0x03ffff (0x485a bytes) 000549 1761238700.135348 mapwrite 0x03f65a thru 0x03ffff (0x9a6 bytes) 000550 1761238700.161003 fallocate 0x02cc7c thru 0x03685c (0x9be1 bytes) 000551 1761238700.169445 fallocate 0x005ca0 thru 0x013dd4 (0xe135 bytes) 000552 1761238700.175986 trunc from 0x007b48 to 0x00b79b (0x3c54 bytes) 000553 1761238700.190071 MIRROR_EXTEND to 3 mirrors 000554 1761238700.867601 read 0x00e3fa thru 0x011bc4 (0x37cb bytes) 000555 1761238700.872307 trunc from 0x040000 to 0x01f669 (0x20996 bytes) 000556 1761238701.008133 write 0x0039a8 thru 0x004927 (0xf80 bytes) 000557 1761238701.026248 MIRROR_RESYNC 3 mirrors 000558 1761238701.782234 write 0x017c68 thru 0x01c94a (0x4ce3 bytes) 000559 1761238701.855881 read 0x006c00 thru 0x013692 (0xca93 bytes) 000560 1761238701.877234 write 0x0376c8 thru 0x03aa85 (0x33be bytes) 000562 1761238701.878895 mapread 0x039199 thru 0x03aa85 (0x18ed bytes) 000563 1761238701.891869 punch from 0x021b08 to 0x02f579 (0xda72 bytes) 000564 1761238701.911761 MIRROR_RESYNC 3 mirrors 000565 1761238703.319020 trunc from 0x03aa86 to 0x013338 (0x2774d bytes) 000566 1761238703.401803 punch from 0x0039e8 to 0x003cd8 (0x2f1 bytes) 000567 1761238703.413022 punch from 0x00d1ec to 0x013338 (0x614d bytes) 000568 1761238703.430910 mapwrite 0x00c73d thru 0x00e826 (0x20ea bytes) 000569 1761238703.505352 trunc from 0x004837 to 0x0116e2 (0xceac bytes) 000570 1761238703.513636 trunc from 0x013339 to 0x01afcf (0x7c97 bytes) 000571 1761238703.534370 trunc from 0x01afd0 to 0x03be4d (0x20e7e bytes) 000572 1761238703.560612 mapwrite 0x012101 thru 0x0205ef (0xe4ef bytes) 000573 1761238703.636607 fallocate 0x01a584 thru 0x01acb4 (0x731 bytes) 000574 1761238703.649881 mapwrite 0x014713 thru 0x01918a (0x4a78 bytes) 000575 1761238703.702824 MIRROR_EXTEND to 4 mirrors 000576 1761238704.691118 mapread 0x0198e1 thru 0x028cec (0xf40c bytes) 000577 1761238704.734751 fallocate 0x015006 thru 0x023c9a (0xec95 bytes) 000578 1761238704.781396 punch from 0x010bf1 to 0x018870 (0x7c80 bytes) 000579 1761238704.787714 write 0x0123ee thru 0x0199e4 (0x75f7 bytes) 000580 1761238704.805249 write 0x006849 thru 0x00b8d4 (0x508c bytes) 000581 1761238704.818611 read 0x01e452 thru 0x028b0d (0xa6bc bytes) 000582 1761238704.827038 trunc from 0x02d60e to 0x02fda1 (0x2794 bytes) 000583 1761238704.830958 punch from 0x00cdc6 to 0x00f093 (0x22ce bytes) 000584 1761238704.836359 fallocate 0x03b5eb thru 0x03ffff (0x4a15 bytes) 000585 1761238704.843768 mapread 0x0021d0 thru 0x0071c9 (0x4ffa bytes) 000586 1761238704.844690 punch from 0x0064d1 to 0x0098c4 (0x33f4 bytes) 000587 1761238704.876382 punch from 0x00bee3 to 0x019989 (0xdaa7 bytes) 000588 1761238704.882905 trunc from 0x03be4e to 0x01143d (0x2aa10 bytes) 000590 1761238704.923045 write 0x00146f thru 0x00de80 (0xca12 bytes) 000591 1761238704.932671 trunc from 0x00ae22 to 0x01143d (0x661c bytes) 000592 1761238704.961685 mapwrite 0x039a79 thru 0x03ffff (0x6587 bytes) 000593 1761238705.017728 trunc from 0x040000 to 0x02001c (0x1ffe3 bytes) 000594 1761238705.032223 trunc from 0x00e057 to 0x01d74e (0xf6f8 bytes) 000595 1761238705.038263 write 0x0349a4 thru 0x03ee5f (0xa4bc bytes) 000596 1761238705.039909 fallocate 0x025ada thru 0x02ba0e (0x5f35 bytes) 000597 1761238705.047946 punch from 0x035a6b to 0x036cf5 (0x128b bytes) 000598 1761238705.062618 trunc from 0x00ff87 to 0x01c2ef (0xc369 bytes) 000599 1761238705.066815 read 0x0026be thru 0x0117b6 (0xf0f9 bytes) 000600 1761238705.078935 punch from 0x034161 to 0x0389a1 (0x4841 bytes) 000601 1761238705.084928 mapread 0x01cd88 thru 0x01e51b (0x1794 bytes) 000602 1761238705.085889 fallocate 0x0279ca thru 0x036d21 (0xf358 bytes) 000603 1761238705.091560 trunc from 0x03ee60 to 0x0002db (0x3eb84 bytes) 000604 1761238705.111752 trunc from 0x0002dc to 0x014bbd (0x148e2 bytes) 000605 1761238705.134179 trunc from 0x014bbe to 0x008e69 (0xbd54 bytes) 000606 1761238705.151925 punch from 0x001fc5 to 0x008e69 (0x6ea5 bytes) 000607 1761238705.157887 read 0x005dff thru 0x008e69 (0x306b bytes) 000608 1761238705.167381 write 0x00299a thru 0x00dc2a (0xb291 bytes) 000609 1761238705.173827 MIRROR_EXTEND to 5 mirrors 000610 1761238705.911575 punch from 0x002381 to 0x00dc2a (0xb8aa bytes) 000611 1761238705.958580 read 0x0096a0 thru 0x00dc2a (0x458b bytes) 000613 1761238705.969295 mapwrite 0x02c0fd thru 0x036bcd (0xaad1 bytes) 000614 1761238706.069360 fallocate 0x021a52 thru 0x030036 (0xe5e5 bytes) 000615 1761238706.075888 read 0x009bfe thru 0x00cfde (0x33e1 bytes) 000616 1761238706.076447 mapread 0x02f293 thru 0x0353d7 (0x6145 bytes) 000617 1761238706.082483 MIRROR_EXTEND to 6 mirrors 000618 1761238706.782025 fallocate 0x00bd0f thru 0x013f7d (0x826f bytes) 000619 1761238706.825002 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000620 1761238707.087909 write 0x0247c7 thru 0x02e3ae (0x9be8 bytes) 000621 1761238707.102567 trunc from 0x035251 to 0x036bcd (0x197d bytes) 000622 1761238707.107964 mapread 0x02cafc thru 0x02ee84 (0x2389 bytes) 000623 1761238707.118600 MIRROR_EXTEND to 6 mirrors 000624 1761238707.706293 punch from 0x01c73c to 0x023581 (0x6e46 bytes) 000625 1761238707.745772 mapwrite 0x03d1c2 thru 0x03ffff (0x2e3e bytes) 000626 1761238707.826322 punch from 0x0060e7 to 0x009168 (0x3082 bytes) 000627 1761238707.831382 mapread 0x0310fd thru 0x03f6f0 (0xe5f4 bytes) 000628 1761238707.841795 punch from 0x023109 to 0x0279c3 (0x48bb bytes) 000629 1761238707.846000 trunc from 0x008333 to 0x0122cc (0x9f9a bytes) 000630 1761238707.853854 write 0x009785 thru 0x011fba (0x8836 bytes) 000631 1761238707.866252 mapwrite 0x01ecf0 thru 0x027a40 (0x8d51 bytes) 000632 1761238707.920143 mapread 0x01cbeb thru 0x023918 (0x6d2e bytes) 000633 1761238707.921139 mapwrite 0x00e2cb thru 0x01ac02 (0xc938 bytes) 000634 1761238707.948734 mapwrite 0x015af0 thru 0x023c49 (0xe15a bytes) 000635 1761238707.981086 mapwrite 0x02fecd thru 0x0379e9 (0x7b1d bytes) 000636 1761238708.007356 trunc from 0x040000 to 0x039a83 (0x657c bytes) 000637 1761238708.025295 fallocate 0x023487 thru 0x032516 (0xf090 bytes) 000639 1761238708.032465 mapwrite 0x01a2f5 thru 0x023845 (0x9551 bytes) 000640 1761238708.053916 trunc from 0x027730 to 0x02a12f (0x2a00 bytes) 000641 1761238708.062378 mapwrite 0x00e895 thru 0x019c0d (0xb379 bytes) 000642 1761238708.081367 trunc from 0x039a84 to 0x02a297 (0xf7ec bytes) 000643 1761238708.098957 mapwrite 0x030c11 thru 0x03a420 (0x9810 bytes) 000644 1761238708.154481 fallocate 0x011536 thru 0x01ab55 (0x9620 bytes) 000645 1761238708.160782 read 0x015cee thru 0x0221c5 (0xc4d8 bytes) 000646 1761238708.161538 mapwrite 0x03ba16 thru 0x03ffff (0x45ea bytes) 000647 1761238708.222743 trunc from 0x0387ec to 0x03ffff (0x7814 bytes) 000648 1761238708.228990 punch from 0x00fea9 to 0x01f594 (0xf6ec bytes) 000649 1761238708.239881 write 0x00e18d thru 0x014ba6 (0x6a1a bytes) 000650 1761238708.253165 mapread 0x007c82 thru 0x014e41 (0xd1c0 bytes) 000652 1761238708.254622 mapread 0x028e95 thru 0x0340be (0xb22a bytes) 000653 1761238708.263691 mapread 0x01d910 thru 0x02cf07 (0xf5f8 bytes) 000654 1761238708.277512 mapwrite 0x0342da thru 0x03ffff (0xbd26 bytes) 000655 1761238708.308881 mapwrite 0x020cb9 thru 0x02df06 (0xd24e bytes) 000656 1761238708.327153 mapread 0x00de5e thru 0x01c0d8 (0xe27b bytes) 000657 1761238708.333790 trunc from 0x040000 to 0x03c5cf (0x3a30 bytes) 000658 1761238708.356371 write 0x01a3a2 thru 0x01ba31 (0x1690 bytes) 000660 1761238708.358816 punch from 0x00f048 to 0x00f918 (0x8d1 bytes) 000661 1761238708.366543 trunc from 0x02ca1b to 0x03b046 (0xe62c bytes) 000662 1761238708.384277 mapread 0x028b82 thru 0x036ce6 (0xe165 bytes) 000663 1761238708.407822 fallocate 0x020d78 thru 0x02813b (0x73c4 bytes) 000664 1761238708.419184 trunc from 0x03c5d0 to 0x014090 (0x2853f bytes) 000665 1761238708.454555 write 0x020bea thru 0x025475 (0x488c bytes) 000666 1761238708.456595 mapread 0x020c27 thru 0x025475 (0x484f bytes) 000667 1761238708.457582 mapread 0x01a4bd thru 0x01ea1e (0x4562 bytes) 000668 1761238708.478027 read 0x00396e thru 0x00c826 (0x8eb9 bytes) 000669 1761238708.480263 trunc from 0x01d226 to 0x025475 (0x8250 bytes) 000670 1761238708.509455 trunc from 0x0096bc to 0x015448 (0xbd8d bytes) 000671 1761238708.521293 read 0x017b9f thru 0x019607 (0x1a69 bytes) 000673 1761238708.525441 trunc from 0x01622d to 0x022281 (0xc055 bytes) 000674 1761238708.536915 punch from 0x016580 to 0x024b47 (0xe5c8 bytes) 000675 1761238708.546859 mapwrite 0x02532c thru 0x02fbaf (0xa884 bytes) 000677 1761238708.600046 mapwrite 0x01abf9 thru 0x027801 (0xcc09 bytes) 000678 1761238708.625002 MIRROR_EXTEND to 7 mirrors 000679 1761238709.444230 write 0x022132 thru 0x022d57 (0xc26 bytes) 000680 1761238709.525205 mapread 0x00cf04 thru 0x01281a (0x5917 bytes) 000681 1761238709.537653 trunc from 0x008853 to 0x00bc82 (0x3430 bytes) 000682 1761238709.541920 read 0x02ecd5 thru 0x02fbaf (0xedb bytes) 000683 1761238709.552118 read 0x00ddb4 thru 0x01bb5a (0xdda7 bytes) 000684 1761238709.580333 MIRROR_RESYNC 7 mirrors 000685 1761238710.889966 punch from 0x015741 to 0x01ae91 (0x5751 bytes) 000686 1761238710.950138 read 0x006ff8 thru 0x00d0ac (0x60b5 bytes) 000687 1761238710.963231 read 0x00760c thru 0x014868 (0xd25d bytes) 000688 1761238710.987968 mapwrite 0x00cc24 thru 0x00fd29 (0x3106 bytes) 000689 1761238711.008550 punch from 0x01a1a8 to 0x020757 (0x65b0 bytes) 000690 1761238711.015424 mapread 0x00be0d thru 0x019578 (0xd76c bytes) 000691 1761238711.016576 trunc from 0x02dfdb to 0x02fbaf (0x1bd5 bytes) 000692 1761238711.027059 mapwrite 0x02ae10 thru 0x031b3e (0x6d2f bytes) 000693 1761238711.116896 punch from 0x0256a3 to 0x02d254 (0x7bb2 bytes) 000694 1761238711.122273 trunc from 0x01b924 to 0x02b8fc (0xffd9 bytes) 000695 1761238711.130866 trunc from 0x02de60 to 0x031b3e (0x3cdf bytes) 000696 1761238711.139284 MIRROR_EXTEND to 8 mirrors 000697 1761238711.958387 fallocate 0x025886 thru 0x02a5aa (0x4d25 bytes) 000698 1761238712.000956 mapread 0x01448e thru 0x021140 (0xccb3 bytes) 000700 1761238712.051792 write 0x011b04 thru 0x015b25 (0x4022 bytes) 000701 1761238712.061070 mapwrite 0x002411 thru 0x009d74 (0x7964 bytes) 000702 1761238712.101824 read 0x0132c0 thru 0x021a08 (0xe749 bytes) 000703 1761238712.109712 mapread 0x029f3f thru 0x02fea8 (0x5f6a bytes) 000704 1761238712.138026 trunc from 0x025e8c to 0x02e51e (0x8693 bytes) 000705 1761238712.152364 mapwrite 0x0354b9 thru 0x03ffff (0xab47 bytes) 000706 1761238712.256629 read 0x039121 thru 0x03beba (0x2d9a bytes) 000707 1761238712.257290 fallocate 0x01043f thru 0x014386 (0x3f48 bytes) 000708 1761238712.263303 punch from 0x011c0a to 0x011f52 (0x349 bytes) 000709 1761238712.268631 mapread 0x01e9dd thru 0x025c5a (0x727e bytes) 000710 1761238712.280640 trunc from 0x02303e to 0x027127 (0x40ea bytes) 000712 1761238712.286666 mapwrite 0x032b9a thru 0x0384c6 (0x592d bytes) 000713 1761238712.311137 trunc from 0x040000 to 0x00b31d (0x34ce2 bytes) 000714 1761238712.335188 trunc from 0x002a14 to 0x00611e (0x370b bytes) 000715 1761238712.344755 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000716 1761238712.648633 mapread 0x0038d4 thru 0x003e4f (0x57c bytes) 000717 1761238712.655630 read 0x00762f thru 0x00b31d (0x3cef bytes) 000718 1761238712.668928 fallocate 0x020543 thru 0x02b03f (0xaafd bytes) 000719 1761238712.673686 mapread 0x008228 thru 0x00b31d (0x30f6 bytes) 000720 1761238712.674568 write 0x01a1ba thru 0x027339 (0xd180 bytes) 000721 1761238712.682215 fallocate 0x02e1c9 thru 0x03376a (0x55a2 bytes) 000723 1761238712.690432 write 0x037032 thru 0x03e036 (0x7005 bytes) 000724 1761238712.695042 fallocate 0x03f6cc thru 0x03ffff (0x934 bytes) 000725 1761238712.704815 trunc from 0x006680 to 0x00f5f3 (0x8f74 bytes) 000726 1761238712.719857 trunc from 0x002eec to 0x006424 (0x3539 bytes) 000727 1761238712.732342 read 0x01a39c thru 0x01a61f (0x284 bytes) 000728 1761238712.733012 trunc from 0x040000 to 0x03178c (0xe873 bytes) 000729 1761238712.762011 punch from 0x01dd77 to 0x02bac2 (0xdd4c bytes) 000730 1761238712.787572 fallocate 0x01e309 thru 0x0231a1 (0x4e99 bytes) 000731 1761238712.791700 read 0x028dbb thru 0x02bceb (0x2f31 bytes) 000732 1761238712.804148 trunc from 0x03178d to 0x018106 (0x19686 bytes) 000733 1761238712.829988 punch from 0x004f0c to 0x00ba61 (0x6b56 bytes) 000734 1761238712.836156 mapread 0x00a0c5 thru 0x00f626 (0x5562 bytes) 000735 1761238712.868676 fallocate 0x00c001 thru 0x01554e (0x954e bytes) 000736 1761238712.880212 mapwrite 0x02137e thru 0x02c722 (0xb3a5 bytes) 000737 1761238712.997902 punch from 0x02b9b7 to 0x02c722 (0xd6c bytes) 000738 1761238713.004663 mapread 0x01415c thru 0x0207b3 (0xc658 bytes) 000739 1761238713.054208 mapread 0x017883 thru 0x0202f7 (0x8a75 bytes) 000740 1761238713.062451 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000741 1761238713.287954 mapread 0x01f2a5 thru 0x0246d9 (0x5435 bytes) 000742 1761238713.317195 write 0x0146f1 thru 0x01b84a (0x715a bytes) 000743 1761238713.330847 read 0x00724d thru 0x00c713 (0x54c7 bytes) 000744 1761238713.346014 fallocate 0x0125c0 thru 0x013360 (0xda1 bytes) 000746 1761238713.358767 write 0x01dd09 thru 0x01eb63 (0xe5b bytes) 000747 1761238713.359840 trunc from 0x00e29f to 0x019f6a (0xbccc bytes) 000748 1761238713.378270 mapread 0x02790d thru 0x02c722 (0x4e16 bytes) 000750 1761238713.396873 fallocate 0x0341cc thru 0x03d865 (0x969a bytes) 000752 1761238713.426875 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000753 1761238713.661895 punch from 0x014e45 to 0x0157ae (0x96a bytes) 000754 1761238713.681173 MIRROR_EXTEND to 6 mirrors 000755 1761238714.505082 trunc from 0x03d866 to 0x02773e (0x16127 bytes) 000756 1761238714.572600 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000757 1761238714.784441 mapwrite 0x03f390 thru 0x03ffff (0xc70 bytes) 000758 1761238714.826252 mapread 0x014e71 thru 0x015f1e (0x10ae bytes) 000759 1761238714.841150 write 0x021be3 thru 0x023951 (0x1d6f bytes) 000760 1761238714.849719 read 0x012f97 thru 0x021116 (0xe180 bytes) 000761 1761238714.855319 MIRROR_RESYNC 5 mirrors 000762 1761238716.751750 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000763 1761238717.144477 fallocate 0x024780 thru 0x02f57d (0xadfe bytes) 000764 1761238717.206909 read 0x017614 thru 0x01db7d (0x656a bytes) 000765 1761238717.216308 read 0x024cb8 thru 0x02ccd5 (0x801e bytes) 000766 1761238717.231144 write 0x021de1 thru 0x02e81d (0xca3d bytes) 000767 1761238717.247570 trunc from 0x03903f to 0x03ffff (0x6fc1 bytes) 000768 1761238717.275235 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000769 1761238717.570151 trunc from 0x02f204 to 0x03b025 (0xbe22 bytes) 000770 1761238717.574810 punch from 0x01c4a0 to 0x026814 (0xa375 bytes) 000771 1761238717.581014 punch from 0x02b676 to 0x02e437 (0x2dc2 bytes) 000772 1761238717.587942 mapwrite 0x03710c thru 0x03ffff (0x8ef4 bytes) 000773 1761238717.640052 MIRROR_EXTEND to 4 mirrors 000774 1761238718.640385 trunc from 0x00d3b5 to 0x01ac63 (0xd8af bytes) 000775 1761238718.694909 read 0x00d6e9 thru 0x01b01a (0xd932 bytes) 000776 1761238718.704163 mapread 0x03588a thru 0x03aec1 (0x5638 bytes) 000777 1761238718.733722 trunc from 0x007120 to 0x01508a (0xdf6b bytes) 000778 1761238718.740239 punch from 0x017f4b to 0x01fd53 (0x7e09 bytes) 000779 1761238718.746422 mapwrite 0x01b996 thru 0x023484 (0x7aef bytes) 000780 1761238718.769638 mapwrite 0x037cb5 thru 0x03ffff (0x834b bytes) 000782 1761238718.794895 punch from 0x02ca8a to 0x037d47 (0xb2be bytes) 000783 1761238718.806400 mapread 0x024453 thru 0x033a63 (0xf611 bytes) 000784 1761238718.838886 trunc from 0x040000 to 0x01cb11 (0x234ee bytes) 000787 1761238718.879068 fallocate 0x0116a2 thru 0x01265a (0xfb9 bytes) 000788 1761238718.885267 read 0x00bf71 thru 0x00e23a (0x22ca bytes) 000789 1761238718.911087 MIRROR_RESYNC 4 mirrors 000790 1761238720.000084 mapwrite 0x02ab01 thru 0x02eb10 (0x4010 bytes) 000791 1761238720.077068 write 0x013b8f thru 0x014be8 (0x105a bytes) 000792 1761238720.093031 trunc from 0x00773a to 0x00b775 (0x403c bytes) 000793 1761238720.101983 write 0x0136b2 thru 0x0221be (0xeb0d bytes) 000795 1761238720.109607 read 0x019bc6 thru 0x01f178 (0x55b3 bytes) 000796 1761238720.110603 mapread 0x002790 thru 0x00afca (0x883b bytes) 000797 1761238720.125207 mapwrite 0x026ef2 thru 0x035734 (0xe843 bytes) 000801 1761238720.178278 read 0x0086ca thru 0x00a383 (0x1cba bytes) 000802 1761238720.178866 read 0x02568a thru 0x029b79 (0x44f0 bytes) 000803 1761238720.180157 trunc from 0x035735 to 0x036344 (0xc10 bytes) 000804 1761238720.204175 mapwrite 0x00db89 thru 0x015682 (0x7afa bytes) 000805 1761238720.224401 punch from 0x004c45 to 0x0101ba (0xb576 bytes) 000806 1761238720.230388 trunc from 0x036345 to 0x018224 (0x1e120 bytes) 000807 1761238720.255222 trunc from 0x018225 to 0x0231c2 (0xaf9e bytes) 000810 1761238720.276945 punch from 0x004d0c to 0x00a2fd (0x55f2 bytes) 000811 1761238720.284226 read 0x0065cb thru 0x00db00 (0x7536 bytes) 000812 1761238720.295031 read 0x00ecd6 thru 0x01298a (0x3cb5 bytes) 000813 1761238720.295721 read 0x00ee75 thru 0x0117db (0x2967 bytes) 000814 1761238720.296288 read 0x019cef thru 0x023198 (0x94aa bytes) 000815 1761238720.301558 MIRROR_EXTEND to 5 mirrors 000816 1761238721.032004 punch from 0x012f4a to 0x016950 (0x3a07 bytes) 000817 1761238721.071510 mapread 0x022c1d thru 0x0231c2 (0x5a6 bytes) 000818 1761238721.088016 MIRROR_RESYNC 5 mirrors 000819 1761238722.467172 MIRROR_EXTEND to 6 mirrors 000820 1761238723.297692 trunc from 0x0231c3 to 0x013330 (0xfe92 bytes) 000821 1761238723.358756 trunc from 0x013331 to 0x03f1c3 (0x2be93 bytes) 000822 1761238723.391928 trunc from 0x0274a6 to 0x02d6aa (0x6205 bytes) 000823 1761238723.398982 write 0x0066c3 thru 0x0157fc (0xf13a bytes) 000824 1761238723.421842 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000825 1761238723.712882 punch from 0x00ec58 to 0x015482 (0x682b bytes) 000826 1761238723.719523 write 0x013287 thru 0x0196f3 (0x646d bytes) 000827 1761238723.732050 punch from 0x00d1de to 0x00d7d7 (0x5fa bytes) 000829 1761238723.742887 mapread 0x02ca8a thru 0x02f282 (0x27f9 bytes) 000830 1761238723.756573 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000831 1761238724.071455 punch from 0x033e30 to 0x03afd0 (0x71a1 bytes) 000832 1761238724.091225 mapwrite 0x01767c thru 0x017f4a (0x8cf bytes) 000833 1761238724.133598 punch from 0x00ea19 to 0x0141c5 (0x57ad bytes) 000834 1761238724.147184 mapread 0x0366e2 thru 0x03f1c3 (0x8ae2 bytes) 000836 1761238724.165383 trunc from 0x03f1c4 to 0x03e7bb (0xa08 bytes) 000837 1761238724.185539 fallocate 0x03136a thru 0x03387e (0x2515 bytes) 000838 1761238724.190399 write 0x01ebd1 thru 0x020ab1 (0x1ee1 bytes) 000839 1761238724.197145 read 0x0116d8 thru 0x017824 (0x614d bytes) 000840 1761238724.207969 mapread 0x00013e thru 0x00bef0 (0xbdb3 bytes) 000841 1761238724.221607 mapread 0x03e177 thru 0x03e7bb (0x645 bytes) 000842 1761238724.223500 fallocate 0x015e50 thru 0x01ad47 (0x4ef8 bytes) 000845 1761238724.229545 trunc from 0x03e7bc to 0x03b5d3 (0x31e8 bytes) 000846 1761238724.256566 fallocate 0x0139fd thru 0x0186f2 (0x4cf6 bytes) 000847 1761238724.263403 mapread 0x01c637 thru 0x02be20 (0xf7ea bytes) 000848 1761238724.274172 mapwrite 0x01d33b thru 0x02b74e (0xe414 bytes) 000849 1761238724.309646 MIRROR_EXTEND to 5 mirrors 000850 1761238725.048981 fallocate 0x0299fb thru 0x02b904 (0x1f0a bytes) 000851 1761238725.096299 mapwrite 0x02c1ca thru 0x033b3c (0x7973 bytes) 000852 1761238725.138468 mapread 0x02256e thru 0x02f141 (0xcbd4 bytes) 000853 1761238725.151063 trunc from 0x0274eb to 0x0302ad (0x8dc3 bytes) 000854 1761238725.157669 mapread 0x01e455 thru 0x0249a6 (0x6552 bytes) 000855 1761238725.158656 trunc from 0x03b5d4 to 0x02c6b4 (0xef1f bytes) 000856 1761238725.210306 punch from 0x012785 to 0x0172a9 (0x4b25 bytes) 000858 1761238725.219170 write 0x018302 thru 0x018b39 (0x838 bytes) 000859 1761238725.228132 trunc from 0x01b8ce to 0x023a3c (0x816f bytes) 000860 1761238725.232227 mapread 0x025629 thru 0x02c6b4 (0x708c bytes) 000861 1761238725.241423 trunc from 0x01fc72 to 0x023f0e (0x429d bytes) 000862 1761238725.246438 mapread 0x021e8a thru 0x024c09 (0x2d80 bytes) 000863 1761238725.256058 write 0x0212c2 thru 0x02c554 (0xb293 bytes) 000864 1761238725.257547 mapread 0x027def thru 0x02c6b4 (0x48c6 bytes) 000865 1761238725.262958 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000866 1761238725.563016 read 0x01637f thru 0x01f10a (0x8d8c bytes) 000867 1761238725.573184 read 0x0198d1 thru 0x029784 (0xfeb4 bytes) 000868 1761238725.582872 read 0x006aa3 thru 0x009174 (0x26d2 bytes) 000869 1761238725.583506 read 0x026ef1 thru 0x02c6b4 (0x57c4 bytes) 000870 1761238725.584032 mapwrite 0x00a9e8 thru 0x00ef39 (0x4552 bytes) 000871 1761238725.613432 fallocate 0x02114d thru 0x021ec2 (0xd76 bytes) 000872 1761238725.635832 MIRROR_EXTEND to 5 mirrors 000873 1761238726.475480 fallocate 0x038aa4 thru 0x03ffff (0x755c bytes) 000874 1761238726.534731 write 0x02836e thru 0x03570d (0xd3a0 bytes) 000875 1761238726.545980 fallocate 0x0281b0 thru 0x03714a (0xef9b bytes) 000876 1761238726.552163 write 0x00006e thru 0x00682c (0x67bf bytes) 000877 1761238726.560503 mapread 0x00b45b thru 0x0135e5 (0x818b bytes) 000878 1761238726.570050 read 0x022e5c thru 0x0244fc (0x16a1 bytes) 000879 1761238726.576387 write 0x02a8ed thru 0x0321b4 (0x78c8 bytes) 000880 1761238726.577580 fallocate 0x02a77e thru 0x039b91 (0xf414 bytes) 000881 1761238726.590724 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000882 1761238726.850534 trunc from 0x01a889 to 0x01e115 (0x388d bytes) 000883 1761238726.856609 read 0x01cdb6 thru 0x025b0c (0x8d57 bytes) 000885 1761238726.871463 MIRROR_RESYNC 4 mirrors 000886 1761238727.925460 read 0x02ed60 thru 0x03298b (0x3c2c bytes) 000887 1761238727.939850 trunc from 0x02fb8f to 0x03714a (0x75bc bytes) 000888 1761238727.992648 MIRROR_RESYNC 4 mirrors 000889 1761238729.002649 punch from 0x004ca7 to 0x0097bb (0x4b15 bytes) 000890 1761238729.044452 trunc from 0x03714b to 0x01378b (0x239bf bytes) 000891 1761238729.068181 mapwrite 0x00985c thru 0x011b90 (0x8335 bytes) 000892 1761238729.110784 punch from 0x012711 to 0x01378b (0x107b bytes) 000893 1761238729.115862 read 0x00515b thru 0x01378b (0xe631 bytes) 000895 1761238729.124763 fallocate 0x028c89 thru 0x035ddf (0xd157 bytes) 000896 1761238729.142210 MIRROR_RESYNC 4 mirrors 000897 1761238729.882914 mapwrite 0x028550 thru 0x02f084 (0x6b35 bytes) 000898 1761238730.003856 write 0x00c277 thru 0x014d80 (0x8b0a bytes) 000899 1761238730.018516 mapread 0x01ba82 thru 0x024a02 (0x8f81 bytes) 000900 1761238730.038082 punch from 0x0254b0 to 0x02f084 (0x9bd5 bytes) 000901 1761238730.049531 trunc from 0x019f46 to 0x024619 (0xa6d4 bytes) 000902 1761238730.062770 punch from 0x00bc87 to 0x015e9a (0xa214 bytes) 000903 1761238730.083843 mapread 0x007c11 thru 0x00e7a2 (0x6b92 bytes) 000904 1761238730.110609 MIRROR_EXTEND to 5 mirrors 000905 1761238730.854827 trunc from 0x019209 to 0x023226 (0xa01e bytes) 000907 1761238730.927234 MIRROR_EXTEND to 6 mirrors 000909 1761238731.709728 mapwrite 0x037fb0 thru 0x03ffff (0x8050 bytes) 000910 1761238731.781476 mapwrite 0x0211c9 thru 0x02b145 (0x9f7d bytes) 000911 1761238731.843827 MIRROR_EXTEND to 7 mirrors 000912 1761238732.688081 mapwrite 0x008145 thru 0x00fda7 (0x7c63 bytes) 000913 1761238732.794031 trunc from 0x040000 to 0x02d365 (0x12c9a bytes) 000914 1761238732.816179 trunc from 0x016eda to 0x02207f (0xb1a6 bytes) 000915 1761238732.821265 read 0x0128ac thru 0x01b7ab (0x8f00 bytes) 000916 1761238732.835770 mapread 0x021507 thru 0x02283f (0x1339 bytes) 000917 1761238732.848197 mapwrite 0x00ba8e thru 0x01939d (0xd910 bytes) 000918 1761238732.889790 trunc from 0x02d366 to 0x006396 (0x26fcf bytes) 000919 1761238732.912743 MIRROR_EXTEND to 8 mirrors 000921 1761238733.644718 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000922 1761238733.989666 read 0x0054d8 thru 0x006396 (0xebf bytes) 000923 1761238734.009145 read 0x005f7c thru 0x006396 (0x41b bytes) 000924 1761238734.024152 trunc from 0x0009c6 to 0x006396 (0x59d1 bytes) 000925 1761238734.063824 fallocate 0x031462 thru 0x038b5f (0x76fe bytes) 000926 1761238734.076304 MIRROR_EXTEND to 8 mirrors 000927 1761238734.694584 punch from 0x005156 to 0x0062d3 (0x117e bytes) 000929 1761238734.740594 fallocate 0x01c4a5 thru 0x026179 (0x9cd5 bytes) 000930 1761238734.747318 read 0x005099 thru 0x006396 (0x12fe bytes) 000931 1761238734.753203 trunc from 0x006397 to 0x03e764 (0x383ce bytes) 000932 1761238734.786333 write 0x02791b thru 0x027da0 (0x486 bytes) 000933 1761238734.801017 fallocate 0x02b899 thru 0x02c022 (0x78a bytes) 000935 1761238734.808523 read 0x026ade thru 0x029acd (0x2ff0 bytes) 000936 1761238734.824809 fallocate 0x01fc5f thru 0x022972 (0x2d14 bytes) 000937 1761238734.831740 read 0x001183 thru 0x00a60c (0x948a bytes) 000938 1761238734.832808 trunc from 0x028fc1 to 0x0316f9 (0x8739 bytes) 000939 1761238734.838876 trunc from 0x02962f to 0x037b46 (0xe518 bytes) 000940 1761238734.843929 write 0x03a488 thru 0x03ffff (0x5b78 bytes) 000941 1761238734.845029 mapread 0x010f53 thru 0x013566 (0x2614 bytes) 000942 1761238734.845818 fallocate 0x0107f6 thru 0x01c540 (0xbd4b bytes) 000943 1761238734.850272 write 0x0386f3 thru 0x03ffff (0x790d bytes) 000944 1761238734.862660 MIRROR_EXTEND to 9 mirrors 000945 1761238735.529447 mapwrite 0x02ceea thru 0x0304df (0x35f6 bytes) 000946 1761238735.643855 write 0x0029bd thru 0x00ec38 (0xc27c bytes) 000947 1761238735.668785 mapwrite 0x03547c thru 0x03ffff (0xab84 bytes) 000948 1761238735.710662 punch from 0x007133 to 0x00a17d (0x304b bytes) 000949 1761238735.715485 mapread 0x025d70 thru 0x028f76 (0x3207 bytes) 000950 1761238735.730222 mapwrite 0x02b0a3 thru 0x032751 (0x76af bytes) 000951 1761238735.752235 read 0x01c997 thru 0x028cc7 (0xc331 bytes) 000952 1761238735.763628 MIRROR_EXTEND to 10 mirrors 000955 1761238736.595921 trunc from 0x040000 to 0x02a278 (0x15d87 bytes) 000956 1761238736.660951 fallocate 0x0099c3 thru 0x01125a (0x7898 bytes) 000957 1761238736.672315 trunc from 0x02a279 to 0x03859f (0xe327 bytes) 000958 1761238736.705360 read 0x022139 thru 0x025590 (0x3458 bytes) 000959 1761238736.716363 read 0x00bfdf thru 0x01151e (0x5540 bytes) 000960 1761238736.732596 read 0x0166f9 thru 0x022c33 (0xc53b bytes) 000961 1761238736.733276 punch from 0x029424 to 0x0305ba (0x7197 bytes) 000962 1761238736.738805 fallocate 0x02b83a thru 0x03a5ed (0xedb4 bytes) 000963 1761238736.744924 write 0x01be1f thru 0x023342 (0x7524 bytes) 000964 1761238736.746183 trunc from 0x0385a0 to 0x030341 (0x825e bytes) 000965 1761238736.782626 trunc from 0x030342 to 0x037c3e (0x78fd bytes) 000966 1761238736.802708 mapwrite 0x03e9dc thru 0x03ffff (0x1624 bytes) 000967 1761238736.852042 read 0x01bebc thru 0x021703 (0x5848 bytes) 000968 1761238736.854606 punch from 0x003f5d to 0x00ede2 (0xae86 bytes) 000969 1761238736.862497 trunc from 0x02230e to 0x02916b (0x6e5e bytes) 000970 1761238736.871691 mapread 0x00cb4d thru 0x01ac4a (0xe0fe bytes) 000972 1761238736.880262 trunc from 0x040000 to 0x01872c (0x278d3 bytes) 000973 1761238736.899074 read 0x00dcd3 thru 0x01872c (0xaa5a bytes) 000974 1761238736.899836 punch from 0x0075c6 to 0x00e256 (0x6c91 bytes) 000975 1761238736.914079 read 0x00a921 thru 0x01872c (0xde0c bytes) 000977 1761238736.922726 write 0x02f9f5 thru 0x03c08e (0xc69a bytes) 000978 1761238736.924485 trunc from 0x03c08f to 0x02d79b (0xe8f3 bytes) 000980 1761238736.956602 trunc from 0x02d79c to 0x017353 (0x16448 bytes) 000982 1761238736.974698 trunc from 0x017354 to 0x0108ed (0x6a66 bytes) 000985 1761238737.005838 read 0x005d95 thru 0x0108ed (0xab59 bytes) 000986 1761238737.006512 trunc from 0x0108ee to 0x012ba3 (0x22b6 bytes) 000987 1761238737.033503 punch from 0x000fd1 to 0x00176a (0x79a bytes) 000988 1761238737.044687 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28:0x0]: Invalid argument (22) 000989 1761238737.306159 write 0x00a537 thru 0x00c548 (0x2012 bytes) 000990 1761238737.321185 mapread 0x011d6c thru 0x012ba3 (0xe38 bytes) 000991 1761238737.335816 read 0x005081 thru 0x00f188 (0xa108 bytes) 000992 1761238737.347004 write 0x03ad66 thru 0x03ffff (0x529a bytes) 000993 1761238737.348511 write 0x03f151 thru 0x03ffff (0xeaf bytes) 000994 1761238737.349801 mapwrite 0x033277 thru 0x03c667 (0x93f1 bytes) 000995 1761238737.384569 read 0x03610c thru 0x03ffff (0x9ef4 bytes) 000996 1761238737.385758 mapwrite 0x033723 thru 0x03ce14 (0x96f2 bytes) 000997 1761238737.414799 fallocate 0x03fadc thru 0x03ffff (0x524 bytes) 000998 1761238737.429980 read 0x03c0d6 thru 0x03e69f (0x25ca bytes) 000999 1761238737.444969 MIRROR_EXTEND to 10 mirrors 001000 1761238738.279585 punch from 0x023eac to 0x02cf27 (0x907c bytes) All operations completed A-OK! PASS 70a (81s) == sanity-flr test 71: check mirror extend parent fid ==== 12:59:02 (1761238742) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x66:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 200a: stress test ===================== 12:59:10 (1761238750) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3163904 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5392704 .. done Extending file size to 3699296 .. Extending file size to 5850816 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6299360 .. Extending file size to 8862880 .. Extending file size to 7082336 .. Extending file size to 8956640 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8931552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8932544 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9128000 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 59178 59180 59181 59184 59185 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:00:23 (1761238823) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5125696 .. Extending file size to 5113728 .. failed Extending file size to 7551040 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7405664 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8238688 .. Extending file size to 7817248 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7914080 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8728768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8420000 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8453184 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8769216 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9389344 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8818272 .. Waiting 61333 61334 61336 61337 61340 done 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) 192.168.203.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:01:35 (1761238895) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1269208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22132 3584888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13924 3593096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36056 7177984 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2096 1269208 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22132 3584888 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 13924 3593096 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36056 7177984 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.071584 s, 29.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212917 s, 9.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:01:49 (1761238909) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0730676 s, 28.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 13:01:55 (1761238915) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x72:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:02:01 (1761238921) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.05438 s, 38.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:02:06 (1761238926) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0854955 s, 24.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:02:13 (1761238933) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0732361 s, 28.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:02:42 (1761238962) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103141 s, 10.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x253:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x255:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x253:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x255:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x254:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:03:41 (1761239021) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137225 s, 15.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:04:29 (1761239069) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg332-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg332-server: Pool lustre.test_204f created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0871357 s, 24.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg332-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:05:30 (1761239130) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:05:36 (1761239136) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:05:43 (1761239143) oleg332-server: Pool lustre.test_206 created Adding targets to pool oleg332-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg332-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg332-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg332-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:06:06 (1761239166) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513941 s, 20.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279469 s, 37.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbb4f685c Mirror 3: 0xbb4f685c PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:06:14 (1761239174) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:06:16 (1761239176) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:06:19 (1761239179) Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0393979 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2150000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x16a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.203.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg332-client.virtnet /mnt/lustre2 (opts:) PASS 209a (26s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:06:45 (1761239205) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00776238 s, 1.1 MB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:06:50 (1761239210) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442265 s, 23.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x95 lmm_fid: [0x200000405:0x95:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x95 lmm_fid: [0x200000405:0x95:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611285 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-23 13:06:51.000000000 -0400 Modify: 2025-10-23 13:06:51.000000000 -0400 Change: 2025-10-23 13:06:52.000000000 -0400 Birth: 2025-10-23 13:06:51.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:06:54 (1761239214) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183302 s, 57.2 MB/s oleg332-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg332-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg332-server: O/280000401/d24/d8: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:07:13 (1761239233) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.17587 s, 233 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0204288 s, 201 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 23 13:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 23 13:07 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test complete, duration 1473 sec =========== 13:07:19 (1761239239) === sanity-flr: start cleanup 13:07:20 (1761239240) === === sanity-flr: finish cleanup 13:07:26 (1761239246) ===